toasters November 2013

toasters@lists.teaparty.net

26 participants
11 discussions

Consistency Point Type M
by Alexander Griesser 18 Jan '14

18 Jan '14

Good morning, does anyone know what consistency point type M is and what's causing it? I was getting them yesterday on one of my 3240s, 8.1.3P1, it has a Flash Cache Card installed if that's relevant for this problem. I was alerted by OnCommand Core about high CPU usage on this filer. *> sysstat -x 1 CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 94% 0 0 0 1122 282622 5880 24406 398932 0 0 5s 99% 99% Mf 45% 0 0 1122 0 0 271544 0 93% 0 0 0 1225 286280 5973 21928 366135 0 0 5s 99% 96% Ms 44% 0 0 1225 0 0 275276 0 87% 0 0 0 1422 316299 6638 24652 243511 0 0 5s 99% 85% Mn 31% 5 0 1417 0 0 303576 0 93% 0 0 0 1018 228505 4797 27948 376695 0 0 5s 99% 87% Mf 41% 0 0 1018 0 0 219500 0 94% 0 0 0 1271 280539 5900 24036 267711 0 0 5s 99% 76% Mn 27% 0 0 1271 0 0 269419 0 91% 0 0 0 1177 271879 5717 28680 312704 0 0 5s 99% 77% M 38% 0 0 1177 0 0 261226 0 94% 0 0 0 1186 264471 5563 29846 352275 0 0 5s 99% 82% Ms 38% 0 0 1186 0 0 254111 0 92% 0 0 0 1369 297761 6271 22447 233291 0 0 5s 99% 79% Mn 30% 5 0 1364 0 0 285898 0 92% 0 0 0 1123 255828 5368 23300 367620 0 0 5s 99% 83% Mf 40% 0 0 1123 0 0 246068 0 91% 0 0 0 1201 272685 5722 26273 292545 0 0 5s 99% 77% Mn 37% 0 0 1201 0 0 261885 0 92% 0 0 0 1355 293194 6145 27210 273654 0 0 5s 99% 76% Mn 33% 83 0 1272 0 0 282260 0 94% 0 0 0 1011 234096 4915 24933 352155 0 0 5s 99% 81% Mn 37% 0 0 1011 0 0 224053 0 90% 0 0 0 1211 275554 5789 36624 285552 0 0 5s 98% 80% Mn 33% 5 0 1206 0 0 264635 0 89% 0 0 0 1325 294916 6197 21227 309395 0 0 5s 99% 78% M 30% 0 0 1325 0 0 283356 0 90% 0 0 0 1134 258937 5429 26731 379776 0 0 5s 98% 84% Mf 42% 0 0 1134 0 0 248736 0 91% 0 0 0 1271 283024 5960 29953 241203 0 0 5s 99% 65% Mn 28% 0 0 1271 0 0 272128 0 90% 0 0 0 1357 287545 6030 29769 329507 0 0 5s 99% 84% Mf 31% 0 0 1357 0 0 275694 0 92% 0 0 0 1265 271400 5695 18026 349410 0 0 9s 99% 84% Ms 39% 6 0 1259 0 0 260907 0 90% 0 0 0 1228 285091 5995 27944 256361 0 0 9s 99% 72% Mn 30% 0 0 1228 0 0 273977 0 *> sysstat -M 1 ANY1+ ANY2+ ANY3+ ANY4+ AVG CPU0 CPU1 CPU2 CPU3 Network Protocol Cluster Storage Raid Target Kahuna WAFL_Ex(Kahu) WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CP 97% 86% 65% 39% 74% 73% 76% 75% 71% 31% 0% 0% 18% 25% 18% 17% 121%( 68%) 12% 0% 0% 41% 5% 7% 8717 50% 94% 79% 54% 25% 66% 68% 68% 69% 59% 38% 0% 0% 16% 20% 20% 12% 104%( 64%) 2% 0% 0% 37% 5% 10% 8336 67% 98% 89% 72% 44% 78% 80% 81% 80% 70% 31% 0% 0% 17% 30% 17% 12% 127%( 73%) 15% 0% 0% 51% 5% 6% 8099 67% 96% 83% 65% 41% 75% 77% 77% 79% 67% 30% 0% 0% 17% 28% 16% 11% 120%( 71%) 14% 0% 0% 46% 4% 14% 7461 49% 94% 78% 52% 24% 65% 65% 66% 65% 62% 37% 0% 0% 15% 15% 20% 13% 117%( 65%) 0% 0% 0% 27% 6% 8% 9042 33% 96% 84% 67% 42% 74% 76% 78% 78% 64% 31% 0% 0% 17% 29% 16% 9% 112%( 66%) 13% 0% 0% 58% 4% 6% 7642 83% 97% 84% 61% 39% 72% 74% 72% 74% 68% 39% 0% 0% 16% 25% 16% 14% 116%( 70%) 13% 0% 0% 40% 4% 6% 7811 34% 94% 78% 55% 28% 67% 69% 70% 70% 60% 36% 0% 0% 16% 19% 20% 12% 113%( 63%) 2% 0% 0% 33% 5% 12% 9018 51% 96% 85% 66% 40% 73% 76% 77% 78% 62% 34% 0% 0% 17% 28% 17% 7% 123%( 72%) 14% 0% 0% 44% 5% 6% 7926 78% 96% 84% 64% 37% 72% 73% 74% 75% 67% 34% 0% 0% 16% 25% 17% 14% 118%( 68%) 14% 0% 0% 41% 5% 6% 7682 69% 95% 83% 63% 38% 72% 72% 73% 73% 69% 34% 0% 0% 15% 21% 16% 10% 130%( 73%) 8% 0% 0% 41% 5% 6% 7491 87% 96% 84% 66% 40% 74% 75% 77% 77% 68% 38% 0% 0% 15% 24% 18% 13% 126%( 70%) 11% 0% 0% 38% 5% 7% 8309 61% 97% 85% 66% 42% 74% 75% 75% 77% 70% 35% 0% 0% 17% 27% 17% 13% 122%( 71%) 10% 0% 0% 46% 5% 6% 7431 63% 95% 85% 65% 42% 73% 75% 75% 76% 69% 38% 0% 0% 14% 24% 15% 6% 133%( 77%) 13% 0% 0% 40% 4% 5% 6856 96% 95% 86% 70% 47% 76% 79% 77% 80% 69% 36% 0% 0% 15% 29% 14% 9% 127%( 73%) 16% 0% 0% 49% 4% 5% 6005 100% 98% 88% 72% 49% 80% 79% 79% 81% 80% 34% 0% 0% 15% 28% 14% 17% 132%( 75%) 16% 0% 0% 50% 4% 11% 5859 82% 96% 86% 67% 41% 75% 77% 77% 78% 68% 38% 0% 0% 15% 26% 16% 9% 126%( 74%) 14% 0% 0% 43% 5% 6% 6800 65% 98% 89% 73% 49% 79% 79% 80% 80% 77% 35% 0% 0% 16% 28% 16% 15% 132%( 75%) 15% 0% 0% 50% 4% 6% 7030 54% 95% 84% 68% 45% 75% 75% 77% 77% 71% 33% 0% 0% 14% 26% 14% 11% 135%( 75%) 15% 0% 0% 42% 4% 6% 6613 62% The only thing I was able to find so far ist hat it has something to do with low mbufs, but that didn't really help me. At the time I took this sysstat, wafl scan status showed some bitmap rearrangements, but they're always running, sometimes more, sometimes less. No Snapmirror, Snapvault, Deduplication or anything like this running on this filer - it just has aggregate snapshots turned on, but no volume snapshots. Any idea? Thanks, Alexander Griesser System-Administrator ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-320 Telefax: +43-5-0556-500 E-Mail: ag(a)anexia.at<mailto:ag@anexia.at> Web: http://www.anexia.at<http://www.anexia.at/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

7 14

[OT] General Purpose Storage Discussion
by Ray Van Dolson 29 Nov '13

29 Nov '13

This list is great -- but obviously is NetApp centric. I'm wondering if any of you can recommend a mailing list or forum, with a similar level of participation and expertise, that's a bit more "general purpose" but still storage-centric. Something similar to the SAGE/LISA mailing lists for Systems Administration. Perhaps SNIA offers such discussion groups to its members? Thanks, Ray

7 15

Definitive supported snapvault doc
by Jeremy Page 22 Nov '13

22 Nov '13

Is there a document out there that can tell me what the supported snapvault configurations are? Like is 8 > 7 in 7 mode etc? I am getting different answers depending on who I ask. Please be advised that this email may contain confidential information. If you are not the intended recipient, please notify us by email by replying to the sender and delete this message. The sender disclaims that the content of this email constitutes an offer to enter into, or the acceptance of, any agreement; provided that the foregoing does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment.

6 7

question about quotas for 7mode DFM 4.x
by Klise, Steve 22 Nov '13

22 Nov '13

7 mode; 8.2 something.. User wants to create a user that has rights to manager Quotas.. What rights do they need? Also, in the custom alerting, can we CC’ say they help desk, or email? I am not sure this is possible.. Hopefully the user can use system manager to manage the quotas, but might need console rights.. I believe the version of DFM is 4.2, so a bit on the old side.. Any help is appreciated.

1 0

Netapp FAS610 8.1.3P1 stalling
by Martin 21 Nov '13

21 Nov '13

One of our 6210 Filers running 8.1.3P1 stalled I/O for approx 1 minute (11:08:27 -11:09:24) We saw this as latency of 5 seconds on the VMware hosts attached to the Filer via NFS and eventually "All Paths Down" messages on the ESX hosts. I also saw warnings in the messages file: Tue Oct 22 11:09:24 BST [TOASTER1: NwkThd_01:warning]: NFS response to client x.x.x.x for volume 0x5834a5c(vol004) was slow, op was v3 write, 69 > 60 (in seconds) >From looking in DFM Filer Summary view I see a "Z" shape in the graph for most of the counters on the Filer e.g. CPU, Network Throughput, All Protocol Ops. The counters dip low then rapidly increase and tail off again. (see attached JPG) <http://network-appliance-toasters.10978.n7.nabble.com/file/n25314/Filer-Z-s…> During this time it all of the ESX hosts saw timeouts to the NFS datastores. I checked the disk_busy on the only aggregate (90 x 15K SAS 450GB) on the Filer and it only shows the disks as 30-40% busy and the disk_busy drops during the time the Filer stalled. It seems odd as the overall load on the Filer didn't increase to precipitate this. <http://network-appliance-toasters.10978.n7.nabble.com/file/n25314/Filer-Z-s…> >From previous experience of performance cases with Netapp, we're usually asked to gather a perfstat next time it happens but this isn't possible as it's unpredictable and short lived. The thing that concerns me is from previous experience this is usually a precursor to the Filer stalling for much longer periods causing much more impact. In the past we've found the only option is to upgrade the Filer head. I would appreciate any pointers on how to identify the root cause of this. -- View this message in context: http://network-appliance-toasters.10978.n7.nabble.com/Netapp-FAS610-8-1-3P1… Sent from the Network Appliance - Toasters mailing list archive at Nabble.com.

3 9

Cluster DATA vs 7-mode: admin prospective
by Iluhes 21 Nov '13

21 Nov '13

Netapp is pushing hard to start using Cluster Data ONTAP. From what I see, from admin/operator level is not as elegant and friendly as "Classic". I just cannot see going to it, it goes against almost everything I like about administrating netapp. Do people feel the same way out there is the field? Netapp is telling us that 7-mode will go away. Is that for real? Do you "like" Cluster OS?

14 24

CPU utilzation
by Alon Zeltser 08 Nov '13

08 Nov '13

Hi I have a FAS3240 running 8.0.5 with 256g flashcache The filer has single aggregate with 92 X 15k disks is using for vmware (nfs over 10g) I have very good performance and latency no complains here The filer is running with average of 13,000 iops and I'm considering adding another 7000 iops to this controller Trying to decide if it's a good idea or not (planning to add 2 new disk shelves) Disk utilization is low 40% average Analyzing perfstat I see I have average CPU utilization of 80-85% (see logs below) First question do you think 80% disk utilization without any other process (no dedupe, no compression , no scrub , no snapmirror, no backups) only user activity, is 80% CPU is normal for this platform with this load ? SPM say the controller should be around 40% CPU with this load Another question I can't figure out Looking at FlashCache statistics I see excellent hit rate over 80% and 24,500 reads replaced per second How can it be that over 24,000 iops replaced per second when sysstat show much less total iops Is the ext_cache_obj show only the maximum or average for the perfstat duration ? ext_cache_obj:ec0:usage:77% ext_cache_obj:ec0:accesses_total:424571082367 ext_cache_obj:ec0:accesses:45930/s ext_cache_obj:ec0:disk_reads_replaced:24502/s ext_cache_obj:ec0:hit:39725/s ext_cache_obj:ec0:hit_percent:86% The last question do you think I should add the 7000 iops to this controller or get a new system ? Thanks ------- ext_cache_obj ---------- TIME: Tue Nov 05 09:54:47 GMT 2013 TIME_DELTA: 60:34 (3634s) ext_cache_obj:ec0:type:IOMEM-FLASH ext_cache_obj:ec0:uptime:1302310171 ext_cache_obj:ec0:blocks:67108864 ext_cache_obj:ec0:size:256 ext_cache_obj:ec0:media:Flash ext_cache_obj:ec0:associativity:8 ext_cache_obj:ec0:sets:16777216 ext_cache_obj:ec0:usage:77% ext_cache_obj:ec0:accesses_total:424571082367 ext_cache_obj:ec0:accesses:45930/s ext_cache_obj:ec0:disk_reads_replaced:24502/s ext_cache_obj:ec0:hit:39725/s ext_cache_obj:ec0:hit_flushq:0/s ext_cache_obj:ec0:hit_once:0/s ext_cache_obj:ec0:hit_age:0/s ext_cache_obj:ec0:hit_normal_lev0:34737/s ext_cache_obj:ec0:hit_metadata_file:3833/s ext_cache_obj:ec0:hit_directory:0/s ext_cache_obj:ec0:hit_redirect:0/s ext_cache_obj:ec0:hit_indirect:1238/s ext_cache_obj:ec0:hit_type.l0.old/new inomap file:11 ext_cache_obj:ec0:hit_type.l0.normal file:124863682 ext_cache_obj:ec0:hit_type.l0.directory:1190 ext_cache_obj:ec0:hit_type.l0.active bitmap:7498789 ext_cache_obj:ec0:hit_type.l0.snap summary:6262825 ext_cache_obj:ec0:hit_type.l0.block types:21600 ext_cache_obj:ec0:hit_type.l0.spacemap:211740 ext_cache_obj:ec0:hit_type.l0.metafile:258 ext_cache_obj:ec0:hit_type.l0.private inode file:6 ext_cache_obj:ec0:hit_type.l0.public inode file:7159 ext_cache_obj:ec0:hit_type.l0.block refcount:853578 ext_cache_obj:ec0:hit_type.l0.private inode map:6 ext_cache_obj:ec0:hit_type.l1.normal file:2477006 ext_cache_obj:ec0:hit_type.l1.directory:11 ext_cache_obj:ec0:hit_type.l1.special file:1 ext_cache_obj:ec0:hit_type.l1.active bitmap:1617 ext_cache_obj:ec0:hit_type.l1.snap summary:36 ext_cache_obj:ec0:hit_type.l1.block types:24413 ext_cache_obj:ec0:hit_type.l1.vvol container:203275 ext_cache_obj:ec0:hit_type.l1.private inode file:2 ext_cache_obj:ec0:hit_type.l1.public inode file:47 ext_cache_obj:ec0:hit_type.l1.block refcount:278778 ext_cache_obj:ec0:hit_type.l1.delete log:1170345 ext_cache_obj:ec0:hit_type.l1.purge file:359 ext_cache_obj:ec0:hit_type.l2.normal file:290697 ext_cache_obj:ec0:hit_type.l2.active bitmap:6 ext_cache_obj:ec0:hit_type.l2.block types:183 ext_cache_obj:ec0:hit_type.l2.vvol container:44698 ext_cache_obj:ec0:hit_type.l2.delete log:2577 ext_cache_obj:ec0:hit_type.l3.normal file:2567 ext_cache_obj:ec0:hit_type.l3.vvol container:711 ext_cache_obj:ec0:hit_type.l3.delete log:40 ext_cache_obj:ec0:hit_partial:101/s ext_cache_obj:ec0:total_metadata_hits:5072/s ext_cache_obj:ec0:total_metadata_misses:784/s ext_cache_obj:ec0:miss:6204/s ext_cache_obj:ec0:miss_flushq:3851/s ext_cache_obj:ec0:miss_once:2808/s ext_cache_obj:ec0:miss_age:0/s ext_cache_obj:ec0:miss_normal_lev0:5420/s ext_cache_obj:ec0:miss_metadata_file:249/s ext_cache_obj:ec0:miss_directory:0/s ext_cache_obj:ec0:miss_indirect:535/s ext_cache_obj:ec0:miss_ssd:0/s ext_cache_obj:ec0:miss_type.l0.old/new inomap file:1 ext_cache_obj:ec0:miss_type.l0.normal file:18874217 ext_cache_obj:ec0:miss_type.l0.directory:244 ext_cache_obj:ec0:miss_type.l0.active bitmap:1415914 ext_cache_obj:ec0:miss_type.l0.snap summary:79062 ext_cache_obj:ec0:miss_type.l0.block types:34523 ext_cache_obj:ec0:miss_type.l0.spacemap:8153 ext_cache_obj:ec0:miss_type.l0.metafile:1 ext_cache_obj:ec0:miss_type.l0.private inode file:1 ext_cache_obj:ec0:miss_type.l0.public inode file:5431 ext_cache_obj:ec0:miss_type.l0.block refcount:165473 ext_cache_obj:ec0:miss_type.l0.private inode map:1 ext_cache_obj:ec0:miss_type.l1.normal file:325314 ext_cache_obj:ec0:miss_type.l1.special file:12 ext_cache_obj:ec0:miss_type.l1.active bitmap:1432 ext_cache_obj:ec0:miss_type.l1.block types:928 ext_cache_obj:ec0:miss_type.l1.vvol container:1518334 ext_cache_obj:ec0:miss_type.l1.block refcount:1460 ext_cache_obj:ec0:miss_type.l1.delete log:76923 ext_cache_obj:ec0:miss_type.l1.purge file:19 ext_cache_obj:ec0:miss_type.l2.normal file:13912 ext_cache_obj:ec0:miss_type.l2.special file:1 ext_cache_obj:ec0:miss_type.l2.active bitmap:3 ext_cache_obj:ec0:miss_type.l2.block types:11 ext_cache_obj:ec0:miss_type.l2.vvol container:3838 ext_cache_obj:ec0:miss_type.l2.delete log:102 ext_cache_obj:ec0:miss_type.l3.normal file:154 ext_cache_obj:ec0:miss_type.l3.vvol container:27 ext_cache_obj:ec0:miss_type.l3.delete log:2 ext_cache_obj:ec0:lookup_reject:1506/s ext_cache_obj:ec0:lookup_reject_normal_l0:0/s ext_cache_obj:ec0:lookup_reject_io:0/s ext_cache_obj:ec0:lookup_reject_lock:0/s ext_cache_obj:ec0:lookup_generation_mismatch:3007 ext_cache_obj:ec0:lookup_checksum_mismatch:2 ext_cache_obj:ec0:lookup_context_mismatch:0 ext_cache_obj:ec0:lookup_disk_ios_issued:3009 ext_cache_obj:ec0:lookup_chains:28388/s ext_cache_obj:ec0:lookup_chain_cnt:50724/s ext_cache_obj:ec0:lookup_lwr_mismatch:0/s ext_cache_obj:ec0:hit_percent:86% ext_cache_obj:ec0:inserts:3227/s ext_cache_obj:ec0:inserts_mbuf:20/s ext_cache_obj:ec0:inserts_mbuf_zerocopy:0/s ext_cache_obj:ec0:inserts_flushq:0/s ext_cache_obj:ec0:inserts_once:0/s ext_cache_obj:ec0:inserts_age:20/s ext_cache_obj:ec0:inserts_normal_lev0:1472/s ext_cache_obj:ec0:inserts_metadata_file:812/s ext_cache_obj:ec0:inserts_directory:0/s ext_cache_obj:ec0:inserts_indirect:955/s ext_cache_obj:ec0:insert_rejects_misc:0/s ext_cache_obj:ec0:insert_rejects_compressed:0/s ext_cache_obj:ec0:insert_rejects_ssd:0/s ext_cache_obj:ec0:insert_rejects_present:36702/s ext_cache_obj:ec0:insert_rejects_flushq:16717/s ext_cache_obj:ec0:insert_rejects_normal_lev0:0/s ext_cache_obj:ec0:insert_rejects_throttle:144/s ext_cache_obj:ec0:insert_rejects_throttle_io:0/s ext_cache_obj:ec0:insert_rejects_throttle_refill:144/s ext_cache_obj:ec0:insert_rejects_throttle_mem:0/s ext_cache_obj:ec0:insert_rejects_cache_reuse:0/s ext_cache_obj:ec0:insert_rejects_vbn_invalid:3227/s ext_cache_obj:ec0:insert_rejects_pvbn_aliased:0/s ext_cache_obj:ec0:insert_rejects_mbuf_nobuf:0/s ext_cache_obj:ec0:insert_rejects_mbuf_low:2189/s ext_cache_obj:ec0:insert_rejects_throttle_rate:0/s ext_cache_obj:ec0:insert_rejects_erase_stripe_state:0/s ext_cache_obj:ec0:insert_rejects_erase_queue:0/s ext_cache_obj:ec0:insert_rejects_mbuf_off:0/s ext_cache_obj:ec0:insert_rejects_no_wip:987/s ext_cache_obj:ec0:insert_rejects_vol_not_mounted:0/s ext_cache_obj:ec0:insert_rejects_delete_log:0/s ext_cache_obj:ec0:insert_rejects_not_valid:104/s ext_cache_obj:ec0:insert_rejects_errored:0/s ext_cache_obj:ec0:insert_rejects_no_context:0/s ext_cache_obj:ec0:reuse_percent:1230% ext_cache_obj:ec0:evicts:36/s ext_cache_obj:ec0:evicts_ref:16/s ext_cache_obj:ec0:readio_solitary:17462/s ext_cache_obj:ec0:readio_chains:24501/s ext_cache_obj:ec0:readio_blocks:39369/s ext_cache_obj:ec0:readio_in_flight:0 ext_cache_obj:ec0:readio_max_in_flight:511 ext_cache_obj:ec0:readio_general_errors:0 ext_cache_obj:ec0:readio_transient_errors:0 ext_cache_obj:ec0:readio_avg_chainlength:1.61 ext_cache_obj:ec0:readio_avg_latency:1.70ms ext_cache_obj:ec0:readio_latency.0:20693251 ext_cache_obj:ec0:readio_latency.1:39478840 ext_cache_obj:ec0:readio_latency.2:11080740 ext_cache_obj:ec0:readio_latency.3:7053418 ext_cache_obj:ec0:readio_latency.4:3915379 ext_cache_obj:ec0:readio_latency.5:2101373 ext_cache_obj:ec0:readio_latency.6:1295253 ext_cache_obj:ec0:readio_latency.7:877946 ext_cache_obj:ec0:readio_latency.<16:2131721 ext_cache_obj:ec0:readio_latency.<24:245311 ext_cache_obj:ec0:readio_latency.<32:38177 ext_cache_obj:ec0:readio_latency.<40:18450 ext_cache_obj:ec0:readio_latency.<48:8463 ext_cache_obj:ec0:readio_latency.<56:4540 ext_cache_obj:ec0:readio_latency.<64:2248 ext_cache_obj:ec0:readio_latency.<128:3371 ext_cache_obj:ec0:readio_latency.<192:19 ext_cache_obj:ec0:readio_chain_lengths.1:63394306 ext_cache_obj:ec0:readio_chain_lengths.2:5874105 ext_cache_obj:ec0:readio_chain_lengths.3:16774665 ext_cache_obj:ec0:readio_chain_lengths.4:1088869 ext_cache_obj:ec0:readio_chain_lengths.5:306521 ext_cache_obj:ec0:readio_chain_lengths.6:871098 ext_cache_obj:ec0:readio_chain_lengths.7:157207 ext_cache_obj:ec0:readio_chain_lengths.<16:434548 ext_cache_obj:ec0:readio_chain_lengths.<24:36558 ext_cache_obj:ec0:readio_chain_lengths.<32:5244 ext_cache_obj:ec0:readio_chain_lengths.<40:3359 ext_cache_obj:ec0:readio_chain_lengths.<48:936 ext_cache_obj:ec0:readio_chain_lengths.<56:632 ext_cache_obj:ec0:readio_chain_lengths.<64:389 ext_cache_obj:ec0:readio_chain_lengths.<128:53 ext_cache_obj:ec0:writeio_solitary:0/s ext_cache_obj:ec0:writeio_chains:50/s ext_cache_obj:ec0:writeio_blocks:3227/s ext_cache_obj:ec0:writeio_in_flight:0 ext_cache_obj:ec0:writeio_max_in_flight:107 ext_cache_obj:ec0:writeio_general_errors:0 ext_cache_obj:ec0:writeio_transient_errors:0 ext_cache_obj:ec0:writeio_avg_chainlength:64.00 ext_cache_obj:ec0:writeio_avg_latency:4.62ms ext_cache_obj:ec0:writeio_latency.0:358 ext_cache_obj:ec0:writeio_latency.1:23690 ext_cache_obj:ec0:writeio_latency.2:41774 ext_cache_obj:ec0:writeio_latency.3:27620 ext_cache_obj:ec0:writeio_latency.4:17766 ext_cache_obj:ec0:writeio_latency.5:14558 ext_cache_obj:ec0:writeio_latency.6:12651 ext_cache_obj:ec0:writeio_latency.7:12580 ext_cache_obj:ec0:writeio_latency.<16:29129 ext_cache_obj:ec0:writeio_latency.<24:2504 ext_cache_obj:ec0:writeio_latency.<32:337 ext_cache_obj:ec0:writeio_latency.<40:83 ext_cache_obj:ec0:writeio_latency.<48:27 ext_cache_obj:ec0:writeio_latency.<56:5 ext_cache_obj:ec0:writeio_latency.<64:2 ext_cache_obj:ec0:writeio_latency.<128:4 ext_cache_obj:ec0:writeio_chain_lengths.<128:183088 ext_cache_obj:ec0:blocks_ref0:41301539 ext_cache_obj:ec0:blocks_ref1:18120071 ext_cache_obj:ec0:blocks_ref2:2121072 ext_cache_obj:ec0:blocks_ref3:1084969 ext_cache_obj:ec0:blocks_ref4:599609 ext_cache_obj:ec0:blocks_ref5:417174 ext_cache_obj:ec0:blocks_ref6:286331 ext_cache_obj:ec0:blocks_ref7:3178099 ext_cache_obj:ec0:blocks_ref0_arrivals:27596243622 ext_cache_obj:ec0:blocks_ref1_arrivals:27622050947 ext_cache_obj:ec0:blocks_ref2_arrivals:10423310000 ext_cache_obj:ec0:blocks_ref3_arrivals:6427750307 ext_cache_obj:ec0:blocks_ref4_arrivals:4495536161 ext_cache_obj:ec0:blocks_ref5_arrivals:3557830045 ext_cache_obj:ec0:blocks_ref6_arrivals:2957867966 ext_cache_obj:ec0:blocks_ref7_arrivals:117062905990 ext_cache_obj:ec0:lru_ticks:3563396220 ext_cache_obj:ec0:invalidates:3526/s ext_cache_obj:ec0:erase_stripe_invalidates:6/s ext_cache_obj:ec0:erase_stripe_blocks_invalidates:3226/s ext_cache_obj:ec0:cache_utilization:77% ext_cache_obj:ec0:pcs_invalidates_flash:0/s ext_cache_obj:ec0:pcs_wraps:0 ext_cache_obj:ec0:pcs_wrap_time:0 ext_cache_obj:ec0:pcs_confidence:100 ext_cache_obj:ec0:pcs_total_ops_insert:0/s ext_cache_obj:ec0:pcs_total_ops_lookup:0/s ext_cache_obj:ec0:pcs_total_ops_invalidate:0/s ext_cache_obj:ec0:pcs_total_ops_misc:0/s ext_cache_obj:ec0:pcs_sampled_ops_insert:0/s ext_cache_obj:ec0:pcs_sampled_ops_lookup:0/s ext_cache_obj:ec0:pcs_sampled_ops_invalidate:0/s ext_cache_obj:ec0:pcs_sampled_ops_misc:0/s ext_cache_obj:ec0:pcs_sampling_rate:0 ---------- sysstat -x 1 ----------- CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 96% 16951 0 0 16951 22579 180997 108622 57263 0 0 4s 67% 100% :f 50% 0 0 0 0 0 0 0 89% 20267 0 0 20267 45941 229706 111661 93433 0 0 5s 51% 100% :f 44% 0 0 0 0 0 0 0 90% 17307 0 0 17307 56792 223833 118109 105672 0 0 3s 48% 100% :f 44% 0 0 0 0 0 0 0 90% 20658 0 0 20658 34301 267718 41860 22109 0 0 7s 50% 38% : 39% 0 0 0 0 0 0 0 77% 12840 0 0 12843 44520 142228 49804 0 0 0 8s 58% 0% - 47% 3 0 0 0 0 0 0 95% 16191 0 0 16191 33156 180821 149909 121486 0 0 5s 53% 58% Dn 46% 0 0 0 0 0 0 0 97% 15198 0 0 15198 23309 199482 100311 43518 0 0 4s 58% 100% :s 42% 0 0 0 0 0 0 0 91% 15363 0 0 15363 22212 216210 82631 62381 0 0 5s 62% 100% :f 46% 0 0 0 0 0 0 0 87% 17662 0 0 17662 36828 209514 118430 79394 0 0 6s 53% 100% :f 49% 0 0 0 0 0 0 0 80% 14307 0 0 14310 47874 166153 74677 54134 0 0 6s 55% 100% :f 49% 3 0 0 0 0 0 0 80% 13905 0 0 13905 42593 202792 49344 49444 0 0 5s 53% 57% : 48% 0 0 0 0 0 0 0 88% 16720 0 0 16720 59662 244810 41227 0 0 0 3s 48% 0% - 44% 0 0 0 0 0 0 0 93% 14717 0 0 14717 17933 223608 155425 102506 0 0 7s 55% 46% Dn 46% 0 0 0 0 0 0 0 96% 13937 0 0 13937 17215 164887 112227 64931 0 0 7s 54% 100% :s 43% 0 0 0 0 0 0 0 92% 13494 0 0 13497 28609 195722 71916 54271 0 0 5s 64% 100% :f 46% 3 0 0 0 0 0 0 93% 22023 0 0 22023 28545 254401 73108 96616 0 0 4s 48% 100% :f 41% 0 0 0 0 0 0 0 88% 23112 0 0 23112 24806 213939 64348 73510 0 0 5s 53% 100% :f 44% 0 0 0 0 0 0 0 84% 17240 0 0 17240 30511 203088 45549 22405 0 0 3s 54% 36% : 41% 0 0 0 0 0 0 0 90% 20676 0 0 20676 61721 269589 35822 24 0 0 4s 51% 0% - 36% 0 0 0 0 0 0 0 84% 19089 0 0 19100 42507 229811 34014 0 0 0 5s 54% 0% - 38% 11 0 0 0 0 0 0 CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 92% 17690 0 0 17690 32740 195063 106798 63482 0 0 5s 51% 33% Dn 40% 0 0 0 0 0 0 0 96% 13450 0 0 13450 18298 175936 97234 80224 0 0 4s 60% 100% :s 39% 0 0 0 0 0 0 0 94% 14813 0 0 14813 13928 204597 69566 61750 0 0 5s 57% 100% :f 39% 0 0 0 0 0 0 0 89% 18505 0 0 18509 23968 247285 104228 69749 0 0 6s 54% 100% :f 47% 4 0 0 0 0 0 0 91% 21963 0 0 21967 60710 223419 78861 66270 0 0 3s 54% 100% :f 42% 4 0 0 0 0 0 0 89% 19919 0 0 19919 33959 215890 79309 59648 0 0 5s 53% 100% :f 45% 0 0 0 0 0 0 0 90% 19707 0 0 19707 27499 247462 35972 33722 0 0 6s 50% 59% : 35% 0 0 0 0 0 0 0 86% 19456 0 0 19456 43880 233384 37526 0 0 0 4s 55% 0% - 38% 0 0 0 0 0 0 0 94% 20130 0 0 20130 63247 229293 78845 48829 0 0 2s 54% 31% Dn 35% 0 0 0 0 0 0 0 96% 10225 0 0 10228 45026 101768 193293 199449 0 0 3s 68% 100% :s 47% 3 0 0 0 0 0 0 95% 13359 0 0 13359 39462 191730 99143 52233 0 0 4s 61% 100% :f 46% 0 0 0 0 0 0 0 87% 15657 0 0 15657 21113 257858 95879 61368 0 0 5s 53% 100% :f 41% 0 0 0 0 0 0 0 89% 19590 0 0 19590 51499 252323 57820 65116 0 0 6s 51% 100% :v 39% 0 0 0 0 0 0 0 77% 14530 0 0 14530 31816 178572 28376 32 0 0 6s 53% 7% : 39% 0 0 0 0 0 0 0 95% 14670 0 0 14673 36120 218937 136512 118201 0 0 5s 55% 54% Dn 45% 3 0 0 0 0 0 0 97% 15592 0 0 15592 24149 170596 136341 76393 0 0 6s 56% 100% :s 41% 0 0 0 0 0 0 0 95% 15562 0 0 15562 46759 158751 158750 58356 0 0 7s 72% 100% :f 59% 0 0 0 0 0 0 0 84% 15471 0 0 15471 15530 229961 145671 73636 0 0 2s 55% 100% :f 47% 0 0 0 0 0 0 0 85% 17753 0 0 17753 53878 214190 53348 46938 0 0 2s 50% 76% : 43% 0 0 0 0 0 0 0 84% 15532 0 0 15535 33703 231932 52108 0 0 0 10s 54% 0% - 48% 3 0 0 0 0 0 0 Summary Statistics ( 3595 samples 0.1 secs/sample) CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out Min 31% 3582 0 0 3582 9751 7799 8404 0 0 0 0s 42% 0% * 12% 0 0 0 0 0 0 0 Avg 87% 13331 0 0 13340 64420 184870 114575 102713 0 0 5s 56% 73% * 43% 8 0 0 0 0 0 0 Max 99% 25894 0 0 25894 215371 661784 322220 330317 0 0 21s 94% 100% * 90% 547 0 0 0 0 0 0 --------- Sysstat -M --------- ANY1+ ANY2+ ANY3+ ANY4+ AVG CPU0 CPU1 CPU2 CPU3 Network Protocol Cluster Storage Raid Target Kahuna WAFL_Ex(Kahu) WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CP 98% 88% 74% 56% 79% 75% 78% 79% 86% 46% 0% 0% 15% 28% 0% 40% 109%( 52%) 10% 0% 0% 65% 4% 1% 13358 81% 98% 88% 73% 55% 79% 75% 77% 80% 86% 46% 0% 0% 13% 27% 0% 41% 108%( 52%) 11% 0% 0% 65% 4% 2% 13152 78% 95% 81% 64% 46% 72% 67% 68% 73% 80% 43% 0% 0% 11% 21% 0% 38% 102%( 50%) 7% 0% 0% 61% 5% 1% 12848 73% 92% 79% 64% 47% 71% 68% 69% 73% 75% 44% 0% 0% 11% 23% 0% 32% 104%( 51%) 8% 0% 0% 57% 5% 1% 13178 64% 90% 78% 64% 48% 71% 68% 69% 72% 73% 44% 0% 0% 11% 24% 0% 30% 101%( 50%) 8% 0% 0% 58% 4% 1% 12550 70% 93% 80% 66% 49% 73% 70% 71% 75% 75% 47% 0% 0% 12% 24% 0% 31% 106%( 53%) 8% 0% 0% 58% 5% 2% 13090 64% 87% 71% 55% 40% 64% 60% 62% 67% 68% 40% 0% 0% 9% 18% 0% 27% 97%( 49%) 6% 0% 0% 51% 5% 1% 12890 67% 91% 77% 61% 45% 69% 65% 67% 71% 74% 43% 0% 0% 11% 22% 0% 32% 100%( 50%) 8% 0% 0% 55% 4% 1% 12697 73% 89% 73% 57% 42% 66% 62% 63% 69% 69% 41% 0% 0% 10% 20% 0% 28% 99%( 50%) 7% 0% 0% 52% 5% 1% 13111 66% 90% 76% 60% 44% 68% 66% 66% 69% 71% 43% 0% 0% 11% 22% 0% 29% 99%( 50%) 8% 0% 0% 55% 4% 2% 12771 72% 87% 71% 55% 40% 64% 60% 62% 66% 67% 41% 0% 0% 10% 18% 0% 27% 96%( 49%) 6% 0% 0% 51% 5% 1% 12675 66% 86% 71% 56% 41% 64% 61% 62% 67% 68% 41% 0% 0% 10% 19% 0% 28% 97%( 48%) 7% 0% 0% 52% 5% 1% 12678 62% 92% 77% 62% 45% 70% 66% 68% 72% 74% 44% 0% 0% 11% 22% 0% 31% 103%( 52%) 8% 0% 0% 55% 5% 1% 13402 75% 94% 81% 67% 49% 74% 69% 72% 76% 77% 45% 0% 0% 12% 26% 0% 33% 104%( 52%) 10% 0% 0% 58% 4% 2% 13289 76% 89% 75% 60% 44% 67% 64% 65% 71% 70% 43% 0% 0% 10% 21% 0% 29% 99%( 50%) 8% 0% 0% 54% 5% 1% 12985 66% 85% 68% 52% 37% 62% 58% 60% 64% 64% 40% 0% 0% 9% 16% 0% 24% 96%( 49%) 6% 0% 0% 49% 5% 1% 12931 64% 90% 75% 61% 45% 69% 64% 66% 72% 72% 42% 0% 0% 10% 21% 0% 30% 103%( 51%) 8% 0% 0% 55% 5% 1% 13163 66% 90% 75% 58% 42% 67% 64% 65% 70% 70% 44% 0% 0% 10% 19% 0% 27% 104%( 52%) 6% 0% 0% 53% 5% 2% 13929 71% 99% 94% 84% 68% 87% 83% 86% 90% 87% 51% 0% 0% 15% 37% 0% 33% 123%( 61%) 14% 0% 0% 69% 4% 1% 14002 81% -- Summary Statistics ( 58 samples 6.0 secs/sample) ANY1+ ANY2+ ANY3+ ANY4+ AVG CPU0 CPU1 CPU2 CPU3 Network Protocol Cluster Storage Raid Target Kahuna WAFL_Ex(Kahu) WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CP Min 85% 68% 52% 37% 62% 58% 60% 64% 64% 40% 0% 0% 9% 16% 0% 24% 95%( 46%) 6% 0% 0% 49% 3% 1% 10226 58% Avg 95% 84% 70% 54% 76% 73% 75% 78% 80% 46% 0% 0% 13% 28% 0% 35% 106%( 51%) 9% 0% 0% 61% 4% 1% 13234 73% Max 100% 96% 87% 70% 89% 84% 93% 90% 91% 56% 0% 0% 26% 49% 0% 49% 123%( 61%) 18% 0% 0% 72% 5% 2% 16583 95% ---------- STATIT ----------- NetApp Release 8.0.5 7-Mode: Sun Jan 20 21:53:54 PST 2013 <1O> Start time: Tue Nov 5 11:25:18 IST 2013 CPU Statistics 60.182572 time (seconds) 100 % 173.781160 system time 289 % 2.716133 rupt time 5 % (1632432 rupts x 2 usec/rupt) 171.065027 non-rupt system time 284 % 66.949128 idle time 111 % 41.250883 time in CP 69 % 100 % 1.790287 rupt time in CP 4 % (1107164 rupts x 2 usec/rupt) Multiprocessor Statistics (per second) cpu0 cpu1 cpu2 cpu3 total sk switches 58254.32 59427.55 57820.04 45838.67 221340.59 hard switches 29026.92 30923.44 35440.13 16979.75 112370.24 domain switches 1174.79 2076.38 238.46 316.05 3805.69 CP rupts 14284.20 1370.85 1370.83 1370.88 18396.75 nonCP rupts 6840.54 629.14 629.14 629.10 8727.91 IPI rupts 0.00 0.00 0.00 0.00 0.00 grab kahuna 0.00 0.98 0.91 0.00 1.89 grab w_xcleaner 0.00 0.18 0.81 268.87 269.86 grab kahuna usec 0.00 488.56 910.73 0.00 1399.29 grab w_xcleaner usec 0.00 123.31 347.31 156723.71 157194.33 CP rupt usec 26808.77 651.73 1779.62 507.44 29747.60 nonCP rupt usec 14115.33 288.34 768.00 212.25 15383.96 idle 318889.16 304923.34 259322.25 229299.07 1112433.85 kahuna 0.47 0.00 0.00 332371.27 332371.75 storage 84402.34 12364.19 16805.60 0.00 113572.15 exempt 199610.16 176903.95 212622.70 13.92 589150.76 raid 815.39 200269.12 22897.81 0.00 223982.33 target 15.04 17.15 16.67 0.00 48.88 netcache 0.00 0.00 0.20 0.00 0.20 netcache2 0.00 0.00 0.00 0.00 0.00 cifs 3.66 3.72 2.77 0.00 10.19 wafl_exempt 254690.06 231801.77 136534.46 437596.00 1060622.33 wafl_xcleaner 33024.31 19066.62 23468.94 0.00 75559.88 sm_exempt 19.36 22.46 21.42 0.00 63.27 cluster 0.00 0.00 0.00 0.00 0.00 protocol 26.95 25.87 26.54 0.00 79.38 nwk_exclusive 975.88 511.33 1150.90 0.00 2638.12 nwk_exempt 261.45 359.64 366.10 0.00 987.23 nwk_legacy 1585.44 1830.31 1686.75 0.00 5102.52 nwk_ctx1 360.39 24181.75 303022.64 0.00 327564.81 nwk_ctx2 59706.95 22852.83 16500.72 0.00 99060.52 nwk_ctx3 0.00 0.00 0.00 0.00 0.00 nwk_ctx4 0.00 0.00 0.00 0.00 0.00 hostOS 4688.73 3925.69 3005.77 0.00 11620.19 56.181715 seconds with one or more CPUs active ( 93%) 48.289199 seconds with 2 or more CPUs active ( 80%) 38.846369 seconds with 3 or more CPUs active ( 65%) 7.892516 seconds with one CPU active ( 13%) 9.442829 seconds with 2 CPUs active ( 16%) 10.092179 seconds with 3 CPUs active ( 17%) 28.754190 seconds with all CPUs active ( 48%) Domain Utilization of Shared Domains (per second) 0.00 idle 516212.17 kahuna 0.00 storage 0.00 exempt 0.00 raid 0.00 target 0.00 netcache 0.00 netcache2 0.00 cifs 0.00 wafl_exempt 0.00 wafl_xcleaner 0.00 sm_exempt 0.00 cluster 0.00 protocol 392429.57 nwk_exclusive 0.00 nwk_exempt 0.00 nwk_legacy 0.00 nwk_ctx1 0.00 nwk_ctx2 0.00 nwk_ctx3 0.00 nwk_ctx4 0.00 hostOS Miscellaneous Statistics (per second) 112370.24 hard context switches 13517.83 NFS operations 0.00 CIFS operations 0.00 HTTP operations 44559.59 network KB received 184368.21 network KB transmitted 94098.07 disk KB read 78261.13 disk KB written 40535.46 NVRAM KB written 0.00 nolog KB written 45831.13 WAFL bufs given to clients 0.80 checksum cache hits ( 0%) 45787.34 no checksum - partial buffer 0.00 FCP operations 0.00 iSCSI operations WAFL Statistics (per second) 81.17 name cache hits ( 77%) 24.81 name cache misses ( 23%) 432838.23 buf hash hits ( 87%) 62737.71 buf hash misses ( 13%) 29606.73 inode cache hits ( 100%) 0.00 inode cache misses ( 0%) 47745.08 buf cache hits ( 55%) 39604.57 buf cache misses ( 45%) 2150.64 blocks read 5882.00 blocks read-ahead 1252.07 chains read-ahead 3674.07 dummy reads 4269.61 blocks speculative read-ahead 15858.58 blocks written 30.57 stripes written 0.00 blocks over-written 0.00 wafl_timer generated CP 0.00 snapshot generated CP 0.00 wafl_avail_bufs generated CP 0.02 dirty_blk_cnt generated CP 0.02 full NV-log generated CP 0.00 back-to-back CP 0.00 flush generated CP 0.00 sync generated CP 0.00 deferred back-to-back CP 0.00 container-indirect-pin CP 0.03 low mbufs generated CP 0.13 low datavecs generated CP 59836.91 non-restart messages 516.21 IOWAIT suspends 142731372.60 next nvlog nearly full msecs 0.00 dirty buffer susp msecs 0.00 nvlog full susp msecs 933124 buffers RAID Statistics (per second) 3433.15 xors 0.00 long dispatches [0] 0.00 long consumed [0] 0.00 long consumed hipri [0] 0.00 long low priority [0] 0.00 long high priority [0] 0.00 long monitor tics [0] 0.00 long monitor clears [0] 0.00 long dispatches [1] 0.00 long consumed [1] 0.00 long consumed hipri [1] 0.00 long low priority [1] 0.00 long high priority [1] 0.00 long monitor tics [1] 0.00 long monitor clears [1] 18 max batch 6.95 blocked mode xor 1220.85 timed mode xor 2.18 fast adjustments 2.63 slow adjustments 0 avg batch start 0 avg stripe/msec 3737.36 checksum dispatches 23508.15 checksum consumed 31.42 tetrises written 0.00 master tetrises 0.00 slave tetrises 1729.77 stripes written 1713.19 partial stripes 16.58 full stripes 16095.72 blocks written 11922.54 blocks read 126.73 1 blocks per stripe size 21 100.83 2 blocks per stripe size 21 94.76 3 blocks per stripe size 21 96.27 4 blocks per stripe size 21 96.14 5 blocks per stripe size 21 96.47 6 blocks per stripe size 21 100.00 7 blocks per stripe size 21 102.87 8 blocks per stripe size 21 100.89 9 blocks per stripe size 21 98.15 10 blocks per stripe size 21 95.36 11 blocks per stripe size 21 91.70 12 blocks per stripe size 21 86.60 13 blocks per stripe size 21 84.97 14 blocks per stripe size 21 78.96 15 blocks per stripe size 21 71.90 16 blocks per stripe size 21 60.37 17 blocks per stripe size 21 54.10 18 blocks per stripe size 21 44.07 19 blocks per stripe size 21 32.04 20 blocks per stripe size 21 16.58 21 blocks per stripe size 21 Network Interface Statistics (per second) iface side bytes packets multicasts errors collisions pkt drops e0a recv 3683.86 50.26 0.00 0.00 0.00 xmit 5048.36 19.94 0.00 0.00 0.00 e0b recv 4408.92 59.15 0.00 0.00 0.00 xmit 20.14 0.10 0.00 0.00 0.00 e2a recv 45620024.58 50126.21 0.00 0.00 0.00 xmit 188787476.51 29046.10 0.18 0.00 0.00 e2b recv 134.66 1.88 0.00 0.00 0.00 xmit 6.98 0.17 0.17 0.00 0.00 e6a recv 71.10 0.91 0.00 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00 e6b recv 59.10 0.63 0.00 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00 e6c recv 0.00 0.00 0.00 71365630.73 0.00 xmit 0.00 0.00 0.00 0.00 0.00 e6d recv 0.00 0.00 0.00 71365630.72 0.00 xmit 0.00 0.00 0.00 0.00 0.00 c0a recv 285.13 21.93 1.46 0.00 0.00 xmit 216.34 1.16 1.16 0.00 0.00 c0b recv 285.13 1.46 1.46 0.00 0.00 xmit 216.34 1.16 1.16 0.00 0.00 e0M recv 0.00 0.00 0.00 0.15 0.00 xmit 0.00 0.00 0.00 0.00 0.00 e0P recv 80.09 0.93 0.08 0.00 0.00 xmit 66.50 0.85 0.00 0.00 0.00 vh recv 0.00 0.00 0.00 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00 vif0-manage recv 8092.77 109.42 89.15 0.00 0.00 xmit 5068.49 20.04 0.00 0.00 0.00 vif1-10g recv 45620159.24 50128.08 3.74 0.70 0.00 xmit 188787483.49 29046.27 0.35 0.00 0.00 vif-174-1g recv 130.20 1.55 1.55 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00 Disk Statistics (per second) ut% is the percent of time the disk was busy. xfers is the number of data-transfer commands issued per second. xfers = ureads + writes + cpreads + greads + gwrites chain is the average number of 4K blocks per command. usecs is the average disk round-trip time per 4K block. disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs /aggr0_sas_600g/plex0/rg0: 0a.01.4 8 23.16 0.76 1.39 2859 9.29 46.62 120 13.11 19.58 181 0.00 .... . 0.00 .... . 0a.01.5 9 23.76 0.80 1.38 3394 9.80 44.28 119 13.16 19.71 199 0.00 .... . 0.00 .... . 0a.01.6 25 64.90 42.39 2.39 3282 6.23 27.14 297 16.28 9.66 441 0.00 .... . 0.00 .... . 0a.02.13 25 67.20 44.22 2.14 3537 6.06 29.12 311 16.92 8.81 470 0.00 .... . 0.00 .... . 0a.02.2 25 63.74 40.44 2.13 3620 6.25 29.57 297 17.05 9.75 393 0.00 .... . 0.00 .... . 0a.03.19 24 62.74 38.78 2.40 3366 6.28 27.77 293 17.68 9.16 365 0.00 .... . 0.00 .... . 0a.03.16 24 63.24 40.51 2.33 3237 6.10 29.41 282 16.63 9.34 403 0.00 .... . 0.00 .... . 0a.02.9 24 61.31 37.85 2.26 3474 5.95 29.06 289 17.51 8.78 460 0.00 .... . 0.00 .... . 0a.02.14 24 61.83 38.05 2.37 3185 6.16 27.20 309 17.61 8.69 474 0.00 .... . 0.00 .... . 0a.02.10 23 60.75 37.97 2.33 3423 5.97 29.50 283 16.82 8.62 452 0.00 .... . 0.00 .... . 0a.02.7 23 57.48 35.26 2.33 3412 5.85 28.95 310 16.37 8.65 440 0.00 .... . 0.00 .... . 0a.03.11 24 61.75 38.92 2.24 3490 5.93 29.31 298 16.90 8.47 453 0.00 .... . 0.00 .... . 0a.03.4 23 61.35 39.36 2.21 3562 5.78 28.82 291 16.20 9.83 365 0.00 .... . 0.00 .... . 0a.02.16 24 61.70 38.00 2.40 3473 6.11 27.69 312 17.58 8.85 491 0.00 .... . 0.00 .... . 0a.03.3 23 62.76 39.20 2.62 2912 6.08 29.18 281 17.48 8.68 386 0.00 .... . 0.00 .... . 0a.03.7 24 62.99 40.16 2.42 3037 6.16 29.02 294 16.67 9.64 374 0.00 .... . 0.00 .... . 0a.03.5 24 63.29 39.66 2.40 3344 6.15 27.10 315 17.48 8.68 428 0.00 .... . 0.00 .... . 0a.01.19 24 62.24 39.45 2.51 3041 6.11 27.71 303 16.68 9.47 511 0.00 .... . 0.00 .... . 0a.01.17 24 62.38 39.51 2.16 3351 6.02 29.16 308 16.85 8.97 564 0.00 .... . 0.00 .... . 0a.01.16 24 61.46 38.62 2.34 3270 6.18 28.09 306 16.67 8.64 555 0.00 .... . 0.00 .... . 0a.01.14 25 64.27 40.96 2.52 3063 6.16 28.16 296 17.15 9.97 454 0.00 .... . 0.00 .... . 0a.03.13 24 62.54 39.08 2.75 2821 6.15 28.75 299 17.31 9.30 443 0.00 .... . 0.00 .... . 0a.03.9 24 64.59 41.79 2.54 3015 5.93 28.06 323 16.87 8.91 428 0.00 .... . 0.00 .... . /aggr0_sas_600g/plex0/rg1: 0a.03.8 7 21.60 0.00 .... . 7.78 54.57 120 13.82 19.67 161 0.00 .... . 0.00 .... . 0a.02.21 8 21.72 0.00 .... . 7.89 53.80 124 13.82 19.64 194 0.00 .... . 0.00 .... . 0a.03.18 25 63.57 41.21 2.68 2986 6.03 28.37 332 16.33 9.35 418 0.00 .... . 0.00 .... . 0a.03.0 24 62.14 40.19 2.34 3235 6.05 28.49 307 15.90 9.74 336 0.00 .... . 0.00 .... . 0a.02.18 25 62.19 41.27 2.16 3611 5.83 28.69 318 15.09 9.18 451 0.00 .... . 0.00 .... . 0a.02.15 24 62.18 40.48 2.36 3201 5.95 27.20 327 15.75 8.67 456 0.00 .... . 0.00 .... . 0a.03.21 24 63.62 42.40 2.29 3318 5.92 27.13 336 15.30 9.85 386 0.00 .... . 0.00 .... . 0a.03.12 24 61.95 40.08 2.41 2970 5.82 27.61 334 16.05 9.34 380 0.00 .... . 0.00 .... . 0a.02.20 25 63.19 41.42 2.41 3333 5.85 28.05 312 15.92 8.87 482 0.00 .... . 0.00 .... . 0a.02.23 25 62.86 40.58 2.20 3548 6.02 27.25 324 16.27 8.83 520 0.00 .... . 0.00 .... . 0a.02.12 25 64.39 42.82 2.43 3185 5.80 27.85 315 15.77 8.92 464 0.00 .... . 0.00 .... . 0a.03.20 24 63.44 41.57 2.34 3159 5.87 27.65 303 16.00 8.72 423 0.00 .... . 0.00 .... . 0a.03.14 24 62.74 40.76 2.38 3287 6.08 28.84 310 15.90 9.07 427 0.00 .... . 0.00 .... . 0a.02.5 24 61.81 40.05 2.14 3729 6.02 28.72 311 15.75 8.73 506 0.00 .... . 0.00 .... . 0a.01.2 26 62.73 41.37 2.37 3226 6.11 27.65 335 15.24 8.99 552 0.00 .... . 0.00 .... . 0a.01.3 24 57.66 35.92 2.39 3542 5.82 26.97 338 15.92 8.93 516 0.00 .... . 0.00 .... . 0a.01.10 24 61.45 39.43 2.40 3215 6.18 28.27 315 15.84 8.81 545 0.00 .... . 0.00 .... . 0a.01.23 24 60.92 39.20 2.54 2881 5.90 26.77 344 15.82 9.01 574 0.00 .... . 0.00 .... . 0a.01.15 24 60.35 38.05 2.33 3345 6.02 28.27 305 16.28 8.92 555 0.00 .... . 0.00 .... . 0a.01.21 25 63.16 41.62 2.31 3374 5.65 29.21 306 15.89 8.90 572 0.00 .... . 0.00 .... . 0a.01.13 25 62.38 40.58 2.51 3214 5.88 27.73 338 15.92 9.22 573 0.00 .... . 0.00 .... . 0a.01.8 26 64.69 43.12 2.35 3346 5.80 27.22 330 15.77 9.34 530 0.00 .... . 0.00 .... . 0a.01.12 25 62.34 40.58 2.29 3404 5.92 28.99 316 15.85 9.06 557 0.00 .... . 0.00 .... . /aggr0_sas_600g/plex0/rg2: 0a.02.0 7 20.47 0.00 .... . 7.69 55.57 119 12.78 18.03 178 0.00 .... . 0.00 .... . 0a.03.1 7 20.59 0.00 .... . 7.81 54.77 120 12.78 17.99 163 0.00 .... . 0.00 .... . 0a.01.0 28 66.68 44.17 2.32 3498 6.41 30.22 311 16.10 9.73 506 0.00 .... . 0.00 .... . 0a.02.1 26 64.67 42.92 2.31 3459 6.26 31.12 317 15.49 9.80 449 0.00 .... . 0.00 .... . 0a.03.2 26 65.77 43.57 2.47 3255 6.08 30.55 307 16.12 9.35 422 0.00 .... . 0.00 .... . 0a.01.1 27 67.36 45.53 2.36 3291 6.23 31.81 283 15.60 10.42 418 0.00 .... . 0.00 .... . 0a.02.3 27 66.85 45.00 2.19 3569 6.05 29.12 317 15.80 9.93 425 0.00 .... . 0.00 .... . 0a.03.6 27 70.77 47.82 2.21 3569 6.38 30.96 308 16.57 9.28 410 0.00 .... . 0.00 .... . 0a.01.7 27 65.07 42.47 2.37 3412 6.25 30.77 305 16.35 9.73 489 0.00 .... . 0.00 .... . 0a.02.4 27 65.73 44.65 2.46 3134 5.93 30.35 302 15.15 10.49 414 0.00 .... . 0.00 .... . 0a.03.10 27 66.80 44.37 2.24 3635 6.41 30.98 311 16.02 9.60 410 0.00 .... . 0.00 .... . 0a.01.9 27 66.53 44.56 2.25 3698 6.11 29.33 320 15.85 10.37 502 0.00 .... . 0.00 .... . 0a.02.6 26 64.67 42.16 2.38 3264 6.15 32.52 293 16.37 9.80 452 0.00 .... . 0.00 .... . 0a.03.15 28 68.97 46.26 2.12 3974 6.21 30.80 327 16.50 9.21 487 0.00 .... . 0.00 .... . 0a.01.11 27 67.25 44.28 2.27 3737 6.38 30.34 320 16.58 9.04 607 0.00 .... . 0.00 .... . 0a.02.8 26 65.32 41.81 2.43 3292 6.28 30.20 321 17.23 8.81 517 0.00 .... . 0.00 .... . 0a.03.17 28 67.45 44.95 2.20 3792 6.18 30.23 329 16.32 9.06 494 0.00 .... . 0.00 .... . 0a.01.18 27 66.80 44.78 2.28 3647 6.18 29.31 336 15.84 9.67 557 0.00 .... . 0.00 .... . 0a.02.11 27 64.69 42.47 2.27 3677 6.26 29.47 311 15.95 10.48 463 0.00 .... . 0.00 .... . 0a.03.22 27 65.58 43.82 2.34 3382 5.97 29.82 312 15.80 9.65 477 0.00 .... . 0.00 .... . 0a.01.20 28 67.93 45.13 2.47 3315 6.45 29.24 322 16.35 9.60 588 0.00 .... . 0.00 .... . 0a.02.17 28 67.00 43.77 2.34 3529 6.48 29.63 332 16.75 9.66 516 0.00 .... . 0.00 .... . 0a.03.23 27 70.05 46.68 2.25 3592 6.33 30.59 320 17.05 9.20 509 0.00 .... . 0.00 .... . /aggr0_sas_600g/plex0/rg3: 3b.05.0 6 18.91 0.00 .... . 7.86 56.64 108 11.05 15.07 148 0.00 .... . 0.00 .... . 0a.02.19 6 19.03 0.00 .... . 7.98 55.84 110 11.05 15.07 179 0.00 .... . 0.00 .... . 0a.01.22 30 74.03 51.36 2.46 3262 6.58 35.44 279 16.08 10.13 494 0.00 .... . 0.00 .... . 3b.05.1 31 77.13 54.15 2.37 3481 6.75 37.11 262 16.23 9.43 452 0.00 .... . 0.00 .... . 0a.02.22 31 77.30 54.24 2.49 3373 6.55 35.55 284 16.52 9.43 498 0.00 .... . 0.00 .... . 3b.05.2 30 73.74 50.76 2.35 3684 6.75 36.56 275 16.23 9.47 419 0.00 .... . 0.00 .... . 3b.05.20 31 76.57 54.02 2.50 3139 6.70 35.59 259 15.85 9.89 448 0.00 .... . 0.00 .... . 3b.05.4 30 76.29 54.19 2.34 3523 6.43 37.21 236 15.67 10.22 378 0.00 .... . 0.00 .... . 3b.05.5 29 74.09 52.11 2.32 3478 6.46 36.30 247 15.52 9.83 382 0.00 .... . 0.00 .... . 3b.05.6 30 74.94 53.11 2.60 3120 6.48 34.96 273 15.35 10.44 401 0.00 .... . 0.00 .... . 3b.05.7 30 76.80 54.24 2.51 3188 6.75 34.71 280 15.82 10.31 396 0.00 .... . 0.00 .... . 3b.05.8 31 79.71 56.46 2.40 3347 6.66 35.86 278 16.58 9.35 382 0.00 .... . 0.00 .... . 3b.05.9 30 74.57 52.08 2.41 3424 6.70 35.91 262 15.80 9.59 426 0.00 .... . 0.00 .... . 3b.05.10 30 76.72 54.43 2.52 3285 6.65 35.93 253 15.64 9.97 437 0.00 .... . 0.00 .... . 3b.05.11 32 76.53 54.07 2.38 3536 6.83 33.47 301 15.64 9.76 463 0.00 .... . 0.00 .... . 3b.05.12 31 77.58 55.02 2.42 3403 6.65 35.36 262 15.92 10.24 430 0.00 .... . 0.00 .... . 3b.05.13 31 75.65 52.47 2.35 3424 6.76 35.10 270 16.42 9.69 481 0.00 .... . 0.00 .... . 3b.05.14 31 75.22 52.61 2.53 3314 6.71 37.28 251 15.90 9.64 455 0.00 .... . 0.00 .... . 3b.05.15 30 74.26 51.59 2.52 3221 6.55 36.06 266 16.12 9.73 439 0.00 .... . 0.00 .... . 3b.05.16 32 78.20 55.80 2.37 3604 6.70 36.85 267 15.70 9.54 527 0.00 .... . 0.00 .... . 3b.05.17 31 75.70 53.24 2.42 3446 6.58 36.37 261 15.89 9.55 492 0.00 .... . 0.00 .... . 3b.05.18 31 77.58 55.10 2.35 3422 6.68 35.76 272 15.80 9.65 515 0.00 .... . 0.00 .... . 3b.05.19 30 75.87 53.40 2.57 3261 6.65 36.61 271 15.82 9.57 542 0.00 .... . 0.00 .... . Aggregate statistics: Minimum 6 18.91 0.00 5.65 11.05 0.00 0.00 Mean 25 62.98 40.64 6.40 15.92 0.00 0.00 Maximum 32 79.71 56.46 9.80 17.68 0.00 0.00 Alon Zeltser Solution Architect ,E&M Computing Tel: 972-35766966 (227) | Mobile: 972-544770108 alonz(a)emet.co.il| www.emet.co.il emet_logo

3 2

Which drives and shelves support full drive encryption?
by Mike Borkowski 06 Nov '13

06 Nov '13

We are considering using Netapp's full drive encryption, I can't seem to find a list of supported drives and shelves. Anyone able to point me at the information? Thanks. Mike Borkowski Manager, Storage & Servers, IST University of Waterloo

2 2

ndmpcopy and vol copy via ssh?
by Jeff Cleverley 01 Nov '13

01 Nov '13

Greetings, Is it acceptable to use ssh from an admin host to run ndmpcopy and vol copy commands? It seems to work initially, but both seem to die after varying periods of time with no useful explanations in the log files. I found a KB about someone connecting to the filer and running ctrl-c thinking the console is idle, but that doesn't apply here unless an ssh command from a cron job or something has the same effect. ndmpd status and vol copy status verify that there are no copies running. All filers are 8.1.2P4, source is a 6080, destination is a 6290, 10G networking on both. I'm doing restore time tests of entire volumes using various methods and these 2 are part of the list. The vol copy ran for ~10 minutes out of the 466 estimated, then stopped. No errors on the command line. It does run, so all the host.equiv entries and permissions are good. I'm sure it did not do the 11TB in 10 minutes :-) The ndmpcopy command ran for ~80 minutes and silently quit. It restored 217G, almost all are in smaller directories. The larger directories didn't show up and there are no abort messages in the logs. They just quit logging. For the ndmpcopy I had a script do an ndmpcopy for each directory (9 total). It looked like multiple ones were running at the same time which seems OK. I was hoping to use parallel threads to speed things up. Any ideas on the silent failures? Thanks, Jeff -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611

5 10

Methods to offline transfer a large amount of data between two filers
by Strauss Wolfram 01 Nov '13

01 Nov '13

The scenario is as follows: Main site with a filer used as a Nearstore System, in particular as a Snapvault destination. A branch site with a smaller filer backing up its data every night with Snapvault to the main site. Both sites are connected over a rather low bandwidth WAN connection. Restoring single files from the main site's Nearstore System to the branch site's filer is no big deal. But what if we have to restore a complete volume in the Terabyte size? Over the WAN connection this would take days to complete. Here comes in the offline transfer scenario like so 1. Make a dump of the volume from the Nearstore System to a portable raid box 2. Hop in the car with the raid box and drive to the branch site (about 100 kilometers away) 3. Restore the volume from the raid box to the branch site filer The crux is how to dump a volume to the portable raid box fast enough. I haven't found a way to directly connect external storage to a filer so we definitely need to use a computer to connect the raid box to and then dump the data from the filer over the network onto this computer. We expect to more or less saturate a 1 Gbits/s link. This would allow us to carry out a restore of a 1 Terabyte volume within one working day. So far we have tried - Copy the files directly over NFS / CIFS -> too slow as expected, especially if there are many files and what if the volume contains LUNs and/or qtrees? - Using NetApp's LREP tool (offline SnapVault) -> astonishingly too slow, we only achieved about 200 Mbits/s although neither the filer nor the laptop used in the scenario hit any resource limit - NetApp's dump command to dump the volume into a file on a second volume and then transfer the dump file over NFS onto the laptop and the raid box. On the branch site copy the dump file over NFS to a second volume on the filer and then restore the main volume from the dump file. -> Especially the creation of the dump file was quite slow and the method doubles space requirements on both filers Out of curiosity, since all the above methods were too slow, I tried the following. I connected a LUN over iSCSI to the laptop used in the scenario and copied the files from the LUN onto the raid box. I easily managed to saturate the 1 Gbits/s link even with many small files residing in the LUN. So basically, the filer, the laptop and the raid box are capable of saturating 1 Gbits/s. What I'm missing is a method to dump a hole volume, including all its sub structures like qtrees and LUNs, in a decent way out of a filer. Does anybody have tried something similar and found a performing solution or did anybody succeed with one of the three above mentions methods and if so, how exactly did your scenario look like? Regards, Wolfram Strauss

3 2

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters November 2013