toasters December 2013

toasters@lists.teaparty.net

23 participants
8 discussions

Consistency Point Type M
by Alexander Griesser 18 Jan '14

18 Jan '14

Good morning, does anyone know what consistency point type M is and what's causing it? I was getting them yesterday on one of my 3240s, 8.1.3P1, it has a Flash Cache Card installed if that's relevant for this problem. I was alerted by OnCommand Core about high CPU usage on this filer. *> sysstat -x 1 CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 94% 0 0 0 1122 282622 5880 24406 398932 0 0 5s 99% 99% Mf 45% 0 0 1122 0 0 271544 0 93% 0 0 0 1225 286280 5973 21928 366135 0 0 5s 99% 96% Ms 44% 0 0 1225 0 0 275276 0 87% 0 0 0 1422 316299 6638 24652 243511 0 0 5s 99% 85% Mn 31% 5 0 1417 0 0 303576 0 93% 0 0 0 1018 228505 4797 27948 376695 0 0 5s 99% 87% Mf 41% 0 0 1018 0 0 219500 0 94% 0 0 0 1271 280539 5900 24036 267711 0 0 5s 99% 76% Mn 27% 0 0 1271 0 0 269419 0 91% 0 0 0 1177 271879 5717 28680 312704 0 0 5s 99% 77% M 38% 0 0 1177 0 0 261226 0 94% 0 0 0 1186 264471 5563 29846 352275 0 0 5s 99% 82% Ms 38% 0 0 1186 0 0 254111 0 92% 0 0 0 1369 297761 6271 22447 233291 0 0 5s 99% 79% Mn 30% 5 0 1364 0 0 285898 0 92% 0 0 0 1123 255828 5368 23300 367620 0 0 5s 99% 83% Mf 40% 0 0 1123 0 0 246068 0 91% 0 0 0 1201 272685 5722 26273 292545 0 0 5s 99% 77% Mn 37% 0 0 1201 0 0 261885 0 92% 0 0 0 1355 293194 6145 27210 273654 0 0 5s 99% 76% Mn 33% 83 0 1272 0 0 282260 0 94% 0 0 0 1011 234096 4915 24933 352155 0 0 5s 99% 81% Mn 37% 0 0 1011 0 0 224053 0 90% 0 0 0 1211 275554 5789 36624 285552 0 0 5s 98% 80% Mn 33% 5 0 1206 0 0 264635 0 89% 0 0 0 1325 294916 6197 21227 309395 0 0 5s 99% 78% M 30% 0 0 1325 0 0 283356 0 90% 0 0 0 1134 258937 5429 26731 379776 0 0 5s 98% 84% Mf 42% 0 0 1134 0 0 248736 0 91% 0 0 0 1271 283024 5960 29953 241203 0 0 5s 99% 65% Mn 28% 0 0 1271 0 0 272128 0 90% 0 0 0 1357 287545 6030 29769 329507 0 0 5s 99% 84% Mf 31% 0 0 1357 0 0 275694 0 92% 0 0 0 1265 271400 5695 18026 349410 0 0 9s 99% 84% Ms 39% 6 0 1259 0 0 260907 0 90% 0 0 0 1228 285091 5995 27944 256361 0 0 9s 99% 72% Mn 30% 0 0 1228 0 0 273977 0 *> sysstat -M 1 ANY1+ ANY2+ ANY3+ ANY4+ AVG CPU0 CPU1 CPU2 CPU3 Network Protocol Cluster Storage Raid Target Kahuna WAFL_Ex(Kahu) WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CP 97% 86% 65% 39% 74% 73% 76% 75% 71% 31% 0% 0% 18% 25% 18% 17% 121%( 68%) 12% 0% 0% 41% 5% 7% 8717 50% 94% 79% 54% 25% 66% 68% 68% 69% 59% 38% 0% 0% 16% 20% 20% 12% 104%( 64%) 2% 0% 0% 37% 5% 10% 8336 67% 98% 89% 72% 44% 78% 80% 81% 80% 70% 31% 0% 0% 17% 30% 17% 12% 127%( 73%) 15% 0% 0% 51% 5% 6% 8099 67% 96% 83% 65% 41% 75% 77% 77% 79% 67% 30% 0% 0% 17% 28% 16% 11% 120%( 71%) 14% 0% 0% 46% 4% 14% 7461 49% 94% 78% 52% 24% 65% 65% 66% 65% 62% 37% 0% 0% 15% 15% 20% 13% 117%( 65%) 0% 0% 0% 27% 6% 8% 9042 33% 96% 84% 67% 42% 74% 76% 78% 78% 64% 31% 0% 0% 17% 29% 16% 9% 112%( 66%) 13% 0% 0% 58% 4% 6% 7642 83% 97% 84% 61% 39% 72% 74% 72% 74% 68% 39% 0% 0% 16% 25% 16% 14% 116%( 70%) 13% 0% 0% 40% 4% 6% 7811 34% 94% 78% 55% 28% 67% 69% 70% 70% 60% 36% 0% 0% 16% 19% 20% 12% 113%( 63%) 2% 0% 0% 33% 5% 12% 9018 51% 96% 85% 66% 40% 73% 76% 77% 78% 62% 34% 0% 0% 17% 28% 17% 7% 123%( 72%) 14% 0% 0% 44% 5% 6% 7926 78% 96% 84% 64% 37% 72% 73% 74% 75% 67% 34% 0% 0% 16% 25% 17% 14% 118%( 68%) 14% 0% 0% 41% 5% 6% 7682 69% 95% 83% 63% 38% 72% 72% 73% 73% 69% 34% 0% 0% 15% 21% 16% 10% 130%( 73%) 8% 0% 0% 41% 5% 6% 7491 87% 96% 84% 66% 40% 74% 75% 77% 77% 68% 38% 0% 0% 15% 24% 18% 13% 126%( 70%) 11% 0% 0% 38% 5% 7% 8309 61% 97% 85% 66% 42% 74% 75% 75% 77% 70% 35% 0% 0% 17% 27% 17% 13% 122%( 71%) 10% 0% 0% 46% 5% 6% 7431 63% 95% 85% 65% 42% 73% 75% 75% 76% 69% 38% 0% 0% 14% 24% 15% 6% 133%( 77%) 13% 0% 0% 40% 4% 5% 6856 96% 95% 86% 70% 47% 76% 79% 77% 80% 69% 36% 0% 0% 15% 29% 14% 9% 127%( 73%) 16% 0% 0% 49% 4% 5% 6005 100% 98% 88% 72% 49% 80% 79% 79% 81% 80% 34% 0% 0% 15% 28% 14% 17% 132%( 75%) 16% 0% 0% 50% 4% 11% 5859 82% 96% 86% 67% 41% 75% 77% 77% 78% 68% 38% 0% 0% 15% 26% 16% 9% 126%( 74%) 14% 0% 0% 43% 5% 6% 6800 65% 98% 89% 73% 49% 79% 79% 80% 80% 77% 35% 0% 0% 16% 28% 16% 15% 132%( 75%) 15% 0% 0% 50% 4% 6% 7030 54% 95% 84% 68% 45% 75% 75% 77% 77% 71% 33% 0% 0% 14% 26% 14% 11% 135%( 75%) 15% 0% 0% 42% 4% 6% 6613 62% The only thing I was able to find so far ist hat it has something to do with low mbufs, but that didn't really help me. At the time I took this sysstat, wafl scan status showed some bitmap rearrangements, but they're always running, sometimes more, sometimes less. No Snapmirror, Snapvault, Deduplication or anything like this running on this filer - it just has aggregate snapshots turned on, but no volume snapshots. Any idea? Thanks, Alexander Griesser System-Administrator ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-320 Telefax: +43-5-0556-500 E-Mail: ag(a)anexia.at<mailto:ag@anexia.at> Web: http://www.anexia.at<http://www.anexia.at/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

7 14

OnTap cluster mode questions
by John Stoffel 02 Jan '14

02 Jan '14

Guys, We're looking to spec out a big Netapp cluster, and since 7-mode is end of life in terms of new features, and it looks like 8.2.1RC1 is now out, what do people think of it? One question I have is whether I should get a bunch of 3250s in cluster pairs, on the order of four or six heads all clustered together, or whether it makes more sense to get a 6250 pair (or pairs?) instead. We're looking at roughtly 1 Petabyte of storage. Probably 50% can be SATA, the rest SAS, but possibly <20% would need to be FC LUNs for Oracle DBs doing some ERP stuff. Lots of IOPs will happen there. Another thought is to get SSDs and setup FlexPools. Which is nice since you can prioritze volumes to get that speedup. But which costs like crazy from what I hear. I'm hesitant of putting all my eggs into one big basket. After talking with a FE, it looks like there are all sorts of goodies in 8.2.1 and up, such that we can setup multiple Virtual Filers (now known as SVMs I think) on the cluster and do transparent volume moves across heads/aggregates, etc. And moving a SVM from one cluster head to another or to another set of nodes in the cluster. My question of course, is how well dows this work in practise? Esp for FCP luns. Being able to live move volumes between aggregates looks to be a real help, since now I won't be stuck withe large, but not endless aggregates. Basically, I'm looking for what people think of how well this all works in real life and what the gotchas are. And of course, what about tape backups over direct attached SAN FC tape drives from an SVM? How is that performance as well? Thanks, John

12 17

long running snapdelta cmd
by rrhodes＠firstenergycorp.com 26 Dec '13

26 Dec '13

I have a script that performs backups for an application. Part of this script is a snapvault of volumes with just Luns. After the snapvault finishes, I run a "snap delta" on the snapvault target vol to determine the time span between the most recent and oldest snapshots/backups. This is check to make sure snapvaults are working. Sometimes after a lot of changes on the lun, and thus a large snapvault operation, the snap delta can run for hours!!! All filers are: IBM N2857 (NetApp FAS2240) Data ONTAP Release 8.1.2P1 7-Mode For example, here is the snapstatus of a snapvault after it completed. It took 17m and transferred 6.4gb. The qtree contains about 5 luns totaling to about 450GB. Source: xxxx:/vol/v_asdf_db/q_asdf_db Destination: yyyy:/vol/v_asdf_db/q_asdf_db Status: Idle Progress: - State: Snapvaulted Lag: 00:20:10 Mirror Timestamp: Tue Dec 24 16:39:33 EST 2013 Base Snapshot: yyyy(1896301030)_v_asdf_db-base.0 Current Transfer Type: - Current Transfer Error: - Contents: Replica Last Transfer Type: Update Last Transfer Size: 6450652 KB Last Transfer Duration: 00:16:59 Last Transfer From: xxxx:/vol/v_asdf_db/q_asdf_db After the above snapvault, the script ran a "snap delta" on vol xxxx:/vol/v_asdf_db (snapvault target). The cmd took +2hr to run!!! (from my script - note the start and end time stamps in form yyyymmddhhmmss) ====> CheckSnapAge - Start 20131224171935 ==> cmd = rsh yyyy -n snap delta v_asdf_db v_asdf_db.31 v_asdf_db.31 Active File System 989985036 9d 17:51 4233243.016 ====> CheckSnapAge - End 20131224193203 In this case, the snapvault target spans 9 days of backups. To determine this took +2 HOURS! The snapvault target isn't getting hit very hard at all, so resources don't appear to be the limitation. I've looked for other ways to determine The time span of the snaps, but haven't come up with any. {The dates on a "snap list" don't have the year!) Q) Why can a "snap delta" run a very long time (but not always)? Thanks for any thoughts! Rick ----------------------------------------- The information contained in this message is intended only for the personal and confidential use of the recipient(s) named above. If the reader of this message is not the intended recipient or an agent responsible for delivering it to the intended recipient, you are hereby notified that you have received this document in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify us immediately, and delete the original message.

1 0

Backup Software Managed Snapshots
by Ray Van Dolson 13 Dec '13

13 Dec '13

Greetings; We're using CommVault and have begun making broader use of their NetApp FlexClone integrated "SnapProtect" to do backups of our VMware environments (as opposed to CommVault using VMware snapshots for taking backups). We also have our NetApps taking two daily snapshots of most VM volumes for a quicker recovery option in some cases. CommVault is recommending that we let it manage all snapshot creation and removal duties and not to mix what they do with NetApp's snapshot scheduling. So far, I'm not understanding why the two can't co-exist. They use different naming conventions on their snapshots, CommVault's snapshots are FlexClone snapshots rather than regular snapshots, and their method offers much less flexibility. What do those of you who have their backup software talking to your filers for snapshot-creation do? Thanks, Ray

2 1

FlashPool with SyncMiror
by Alon Zeltser 11 Dec '13

11 Dec '13

Hi All Does anyone know if FlashPool is supported with SyncMirror (MetroCluster) ? If so do I need to mirror also the SSD raid group with matching plex ? Or can I have one plex with hybrid and other plex without SSD ? Is anyone seen any documentation for such a configuration ? Thanks Alonz

3 3

FC 144 tech refresh and performance doubts
by Milazzo Giacomo 10 Dec '13

10 Dec '13

Hi all, a customer of mine will have to perform a tech refresh on his 144 FC 15k disks. Now the "specialist", considering ONLY budget aspects offer this: A FAS3240 having 3 144 GB DS14MK4 shelves will be replaced by ONE DS4246 24x600 SAS A FAS3240 having 12 144 FB DS14MK4 shelves will be replaced by TWI DS4243 24x600 SAS Now, also considering the SAS multiplex and the direct access to the block customer, and me too, are very concerned about a degradation of performances. It was a lot of data disk against few ones. Suggestions? Opinions? Dott. Giacomo Milazzo [cid:image001.jpg@01C75422.0D316270] Technical Account Manager Sinergy SpA Filiale di Roma * 00198. viale Regina Margherita, 269 ' (+39) 3406001045 0644245272 7 (+39) 0226922048 * Giacomo.Milazzo(a)Sinergy.it

2 1

free_space_realloc and measuring potentical benifit of free space optimization
by Jordan Slingerland 09 Dec '13

09 Dec '13

I have a unit that is experiencing some write bottle necks and back to back consistency points at times. I never see any of the disks busier than 60% even during the periods where I notice lots of CpfromCps. The unit has 2 aggregates of 50 disk with a raid size of 18. 2 raid groups are full and the last only has 14 disks. Looking at a statit during normal operation The smaller raid group is not any busier than the larger rgs. (I'm going to try to get a statit while the back to back cps are occurring and see if this holds true) During the periods of back to back cps I am seeing about 2500 writes ops and 7k read ops. This is a IBM rebranded N6240 with flexscale cards. (FAS3240) So, the question is, I am thinking of enabling free_space_realloc on the aggregate and possibly read_realloc on the volumes. (sis is not in use on this unit) Is there a way to measure the potential benefit of free_space_realloc, or rather how fragmented the aggregate is. I would not have expected it to be very fragmented at 60% used, but I would aslo expect to get maybe closer to 5k write ops before seeing back to back copies. Questions, comments, warnings, random thoughts? -JMS For the fourth consecutive year, Independent Health is the highest ranked health insurance plan in the New York/New Jersey region by J.D. Power and Associates 2013 Member Health Plan StudySM. For the sixth consecutive year, Independent Health was named one of the best companies to work for in New York State. Go to www.independenthealth.com to view careers and available positions. We are an Equal Opportunity Employer. CONFIDENTIALITY NOTICE: This e-mail and its attachments (collectively referred to as "e-mail") may contain confidential information that is privileged and protected from disclosure by Federal and State confidentiality laws, rules or regulations. This e-mail is intended for the designated addressee only. If you are not the designated addressee, you are notified that any disclosure, copying or distribution of this e-mail may be unlawful and may subject you to legal consequences. If you have received this e-mail in error, please contact me immediately by telephone at (716) 631 - 3001 and delete the e-mail from your computer immediately. Thank you for your attention.

3 2

FlexCache monitoring
by Touretsky, Gregory 05 Dec '13

05 Dec '13

Hi, Is anybody on this list using FlexCache for NFS access over WAN? What kind of performance metrics would you be tracking to detect potential performance issues? Are you leveraging things like "flexcache hist" for any kind of troubleshooting? Thanks, Gregory --------------------------------------------------------------------- Intel Israel (74) Limited This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

2 2

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters December 2013