toasters

toasters@lists.teaparty.net

13531 discussions

question about quotas for 7mode DFM 4.x
by Klise, Steve 22 Nov '13

22 Nov '13

7 mode; 8.2 something.. User wants to create a user that has rights to manager Quotas.. What rights do they need? Also, in the custom alerting, can we CC’ say they help desk, or email? I am not sure this is possible.. Hopefully the user can use system manager to manage the quotas, but might need console rights.. I believe the version of DFM is 4.2, so a bit on the old side.. Any help is appreciated.

1 0

Netapp FAS610 8.1.3P1 stalling
by Martin 21 Nov '13

21 Nov '13

One of our 6210 Filers running 8.1.3P1 stalled I/O for approx 1 minute (11:08:27 -11:09:24) We saw this as latency of 5 seconds on the VMware hosts attached to the Filer via NFS and eventually "All Paths Down" messages on the ESX hosts. I also saw warnings in the messages file: Tue Oct 22 11:09:24 BST [TOASTER1: NwkThd_01:warning]: NFS response to client x.x.x.x for volume 0x5834a5c(vol004) was slow, op was v3 write, 69 > 60 (in seconds) >From looking in DFM Filer Summary view I see a "Z" shape in the graph for most of the counters on the Filer e.g. CPU, Network Throughput, All Protocol Ops. The counters dip low then rapidly increase and tail off again. (see attached JPG) <http://network-appliance-toasters.10978.n7.nabble.com/file/n25314/Filer-Z-s…> During this time it all of the ESX hosts saw timeouts to the NFS datastores. I checked the disk_busy on the only aggregate (90 x 15K SAS 450GB) on the Filer and it only shows the disks as 30-40% busy and the disk_busy drops during the time the Filer stalled. It seems odd as the overall load on the Filer didn't increase to precipitate this. <http://network-appliance-toasters.10978.n7.nabble.com/file/n25314/Filer-Z-s…> >From previous experience of performance cases with Netapp, we're usually asked to gather a perfstat next time it happens but this isn't possible as it's unpredictable and short lived. The thing that concerns me is from previous experience this is usually a precursor to the Filer stalling for much longer periods causing much more impact. In the past we've found the only option is to upgrade the Filer head. I would appreciate any pointers on how to identify the root cause of this. -- View this message in context: http://network-appliance-toasters.10978.n7.nabble.com/Netapp-FAS610-8-1-3P1… Sent from the Network Appliance - Toasters mailing list archive at Nabble.com.

3 9

Cluster DATA vs 7-mode: admin prospective
by Iluhes 21 Nov '13

21 Nov '13

Netapp is pushing hard to start using Cluster Data ONTAP. From what I see, from admin/operator level is not as elegant and friendly as "Classic". I just cannot see going to it, it goes against almost everything I like about administrating netapp. Do people feel the same way out there is the field? Netapp is telling us that 7-mode will go away. Is that for real? Do you "like" Cluster OS?

14 24

CPU utilzation
by Alon Zeltser 08 Nov '13

08 Nov '13

Hi I have a FAS3240 running 8.0.5 with 256g flashcache The filer has single aggregate with 92 X 15k disks is using for vmware (nfs over 10g) I have very good performance and latency no complains here The filer is running with average of 13,000 iops and I'm considering adding another 7000 iops to this controller Trying to decide if it's a good idea or not (planning to add 2 new disk shelves) Disk utilization is low 40% average Analyzing perfstat I see I have average CPU utilization of 80-85% (see logs below) First question do you think 80% disk utilization without any other process (no dedupe, no compression , no scrub , no snapmirror, no backups) only user activity, is 80% CPU is normal for this platform with this load ? SPM say the controller should be around 40% CPU with this load Another question I can't figure out Looking at FlashCache statistics I see excellent hit rate over 80% and 24,500 reads replaced per second How can it be that over 24,000 iops replaced per second when sysstat show much less total iops Is the ext_cache_obj show only the maximum or average for the perfstat duration ? ext_cache_obj:ec0:usage:77% ext_cache_obj:ec0:accesses_total:424571082367 ext_cache_obj:ec0:accesses:45930/s ext_cache_obj:ec0:disk_reads_replaced:24502/s ext_cache_obj:ec0:hit:39725/s ext_cache_obj:ec0:hit_percent:86% The last question do you think I should add the 7000 iops to this controller or get a new system ? Thanks ------- ext_cache_obj ---------- TIME: Tue Nov 05 09:54:47 GMT 2013 TIME_DELTA: 60:34 (3634s) ext_cache_obj:ec0:type:IOMEM-FLASH ext_cache_obj:ec0:uptime:1302310171 ext_cache_obj:ec0:blocks:67108864 ext_cache_obj:ec0:size:256 ext_cache_obj:ec0:media:Flash ext_cache_obj:ec0:associativity:8 ext_cache_obj:ec0:sets:16777216 ext_cache_obj:ec0:usage:77% ext_cache_obj:ec0:accesses_total:424571082367 ext_cache_obj:ec0:accesses:45930/s ext_cache_obj:ec0:disk_reads_replaced:24502/s ext_cache_obj:ec0:hit:39725/s ext_cache_obj:ec0:hit_flushq:0/s ext_cache_obj:ec0:hit_once:0/s ext_cache_obj:ec0:hit_age:0/s ext_cache_obj:ec0:hit_normal_lev0:34737/s ext_cache_obj:ec0:hit_metadata_file:3833/s ext_cache_obj:ec0:hit_directory:0/s ext_cache_obj:ec0:hit_redirect:0/s ext_cache_obj:ec0:hit_indirect:1238/s ext_cache_obj:ec0:hit_type.l0.old/new inomap file:11 ext_cache_obj:ec0:hit_type.l0.normal file:124863682 ext_cache_obj:ec0:hit_type.l0.directory:1190 ext_cache_obj:ec0:hit_type.l0.active bitmap:7498789 ext_cache_obj:ec0:hit_type.l0.snap summary:6262825 ext_cache_obj:ec0:hit_type.l0.block types:21600 ext_cache_obj:ec0:hit_type.l0.spacemap:211740 ext_cache_obj:ec0:hit_type.l0.metafile:258 ext_cache_obj:ec0:hit_type.l0.private inode file:6 ext_cache_obj:ec0:hit_type.l0.public inode file:7159 ext_cache_obj:ec0:hit_type.l0.block refcount:853578 ext_cache_obj:ec0:hit_type.l0.private inode map:6 ext_cache_obj:ec0:hit_type.l1.normal file:2477006 ext_cache_obj:ec0:hit_type.l1.directory:11 ext_cache_obj:ec0:hit_type.l1.special file:1 ext_cache_obj:ec0:hit_type.l1.active bitmap:1617 ext_cache_obj:ec0:hit_type.l1.snap summary:36 ext_cache_obj:ec0:hit_type.l1.block types:24413 ext_cache_obj:ec0:hit_type.l1.vvol container:203275 ext_cache_obj:ec0:hit_type.l1.private inode file:2 ext_cache_obj:ec0:hit_type.l1.public inode file:47 ext_cache_obj:ec0:hit_type.l1.block refcount:278778 ext_cache_obj:ec0:hit_type.l1.delete log:1170345 ext_cache_obj:ec0:hit_type.l1.purge file:359 ext_cache_obj:ec0:hit_type.l2.normal file:290697 ext_cache_obj:ec0:hit_type.l2.active bitmap:6 ext_cache_obj:ec0:hit_type.l2.block types:183 ext_cache_obj:ec0:hit_type.l2.vvol container:44698 ext_cache_obj:ec0:hit_type.l2.delete log:2577 ext_cache_obj:ec0:hit_type.l3.normal file:2567 ext_cache_obj:ec0:hit_type.l3.vvol container:711 ext_cache_obj:ec0:hit_type.l3.delete log:40 ext_cache_obj:ec0:hit_partial:101/s ext_cache_obj:ec0:total_metadata_hits:5072/s ext_cache_obj:ec0:total_metadata_misses:784/s ext_cache_obj:ec0:miss:6204/s ext_cache_obj:ec0:miss_flushq:3851/s ext_cache_obj:ec0:miss_once:2808/s ext_cache_obj:ec0:miss_age:0/s ext_cache_obj:ec0:miss_normal_lev0:5420/s ext_cache_obj:ec0:miss_metadata_file:249/s ext_cache_obj:ec0:miss_directory:0/s ext_cache_obj:ec0:miss_indirect:535/s ext_cache_obj:ec0:miss_ssd:0/s ext_cache_obj:ec0:miss_type.l0.old/new inomap file:1 ext_cache_obj:ec0:miss_type.l0.normal file:18874217 ext_cache_obj:ec0:miss_type.l0.directory:244 ext_cache_obj:ec0:miss_type.l0.active bitmap:1415914 ext_cache_obj:ec0:miss_type.l0.snap summary:79062 ext_cache_obj:ec0:miss_type.l0.block types:34523 ext_cache_obj:ec0:miss_type.l0.spacemap:8153 ext_cache_obj:ec0:miss_type.l0.metafile:1 ext_cache_obj:ec0:miss_type.l0.private inode file:1 ext_cache_obj:ec0:miss_type.l0.public inode file:5431 ext_cache_obj:ec0:miss_type.l0.block refcount:165473 ext_cache_obj:ec0:miss_type.l0.private inode map:1 ext_cache_obj:ec0:miss_type.l1.normal file:325314 ext_cache_obj:ec0:miss_type.l1.special file:12 ext_cache_obj:ec0:miss_type.l1.active bitmap:1432 ext_cache_obj:ec0:miss_type.l1.block types:928 ext_cache_obj:ec0:miss_type.l1.vvol container:1518334 ext_cache_obj:ec0:miss_type.l1.block refcount:1460 ext_cache_obj:ec0:miss_type.l1.delete log:76923 ext_cache_obj:ec0:miss_type.l1.purge file:19 ext_cache_obj:ec0:miss_type.l2.normal file:13912 ext_cache_obj:ec0:miss_type.l2.special file:1 ext_cache_obj:ec0:miss_type.l2.active bitmap:3 ext_cache_obj:ec0:miss_type.l2.block types:11 ext_cache_obj:ec0:miss_type.l2.vvol container:3838 ext_cache_obj:ec0:miss_type.l2.delete log:102 ext_cache_obj:ec0:miss_type.l3.normal file:154 ext_cache_obj:ec0:miss_type.l3.vvol container:27 ext_cache_obj:ec0:miss_type.l3.delete log:2 ext_cache_obj:ec0:lookup_reject:1506/s ext_cache_obj:ec0:lookup_reject_normal_l0:0/s ext_cache_obj:ec0:lookup_reject_io:0/s ext_cache_obj:ec0:lookup_reject_lock:0/s ext_cache_obj:ec0:lookup_generation_mismatch:3007 ext_cache_obj:ec0:lookup_checksum_mismatch:2 ext_cache_obj:ec0:lookup_context_mismatch:0 ext_cache_obj:ec0:lookup_disk_ios_issued:3009 ext_cache_obj:ec0:lookup_chains:28388/s ext_cache_obj:ec0:lookup_chain_cnt:50724/s ext_cache_obj:ec0:lookup_lwr_mismatch:0/s ext_cache_obj:ec0:hit_percent:86% ext_cache_obj:ec0:inserts:3227/s ext_cache_obj:ec0:inserts_mbuf:20/s ext_cache_obj:ec0:inserts_mbuf_zerocopy:0/s ext_cache_obj:ec0:inserts_flushq:0/s ext_cache_obj:ec0:inserts_once:0/s ext_cache_obj:ec0:inserts_age:20/s ext_cache_obj:ec0:inserts_normal_lev0:1472/s ext_cache_obj:ec0:inserts_metadata_file:812/s ext_cache_obj:ec0:inserts_directory:0/s ext_cache_obj:ec0:inserts_indirect:955/s ext_cache_obj:ec0:insert_rejects_misc:0/s ext_cache_obj:ec0:insert_rejects_compressed:0/s ext_cache_obj:ec0:insert_rejects_ssd:0/s ext_cache_obj:ec0:insert_rejects_present:36702/s ext_cache_obj:ec0:insert_rejects_flushq:16717/s ext_cache_obj:ec0:insert_rejects_normal_lev0:0/s ext_cache_obj:ec0:insert_rejects_throttle:144/s ext_cache_obj:ec0:insert_rejects_throttle_io:0/s ext_cache_obj:ec0:insert_rejects_throttle_refill:144/s ext_cache_obj:ec0:insert_rejects_throttle_mem:0/s ext_cache_obj:ec0:insert_rejects_cache_reuse:0/s ext_cache_obj:ec0:insert_rejects_vbn_invalid:3227/s ext_cache_obj:ec0:insert_rejects_pvbn_aliased:0/s ext_cache_obj:ec0:insert_rejects_mbuf_nobuf:0/s ext_cache_obj:ec0:insert_rejects_mbuf_low:2189/s ext_cache_obj:ec0:insert_rejects_throttle_rate:0/s ext_cache_obj:ec0:insert_rejects_erase_stripe_state:0/s ext_cache_obj:ec0:insert_rejects_erase_queue:0/s ext_cache_obj:ec0:insert_rejects_mbuf_off:0/s ext_cache_obj:ec0:insert_rejects_no_wip:987/s ext_cache_obj:ec0:insert_rejects_vol_not_mounted:0/s ext_cache_obj:ec0:insert_rejects_delete_log:0/s ext_cache_obj:ec0:insert_rejects_not_valid:104/s ext_cache_obj:ec0:insert_rejects_errored:0/s ext_cache_obj:ec0:insert_rejects_no_context:0/s ext_cache_obj:ec0:reuse_percent:1230% ext_cache_obj:ec0:evicts:36/s ext_cache_obj:ec0:evicts_ref:16/s ext_cache_obj:ec0:readio_solitary:17462/s ext_cache_obj:ec0:readio_chains:24501/s ext_cache_obj:ec0:readio_blocks:39369/s ext_cache_obj:ec0:readio_in_flight:0 ext_cache_obj:ec0:readio_max_in_flight:511 ext_cache_obj:ec0:readio_general_errors:0 ext_cache_obj:ec0:readio_transient_errors:0 ext_cache_obj:ec0:readio_avg_chainlength:1.61 ext_cache_obj:ec0:readio_avg_latency:1.70ms ext_cache_obj:ec0:readio_latency.0:20693251 ext_cache_obj:ec0:readio_latency.1:39478840 ext_cache_obj:ec0:readio_latency.2:11080740 ext_cache_obj:ec0:readio_latency.3:7053418 ext_cache_obj:ec0:readio_latency.4:3915379 ext_cache_obj:ec0:readio_latency.5:2101373 ext_cache_obj:ec0:readio_latency.6:1295253 ext_cache_obj:ec0:readio_latency.7:877946 ext_cache_obj:ec0:readio_latency.<16:2131721 ext_cache_obj:ec0:readio_latency.<24:245311 ext_cache_obj:ec0:readio_latency.<32:38177 ext_cache_obj:ec0:readio_latency.<40:18450 ext_cache_obj:ec0:readio_latency.<48:8463 ext_cache_obj:ec0:readio_latency.<56:4540 ext_cache_obj:ec0:readio_latency.<64:2248 ext_cache_obj:ec0:readio_latency.<128:3371 ext_cache_obj:ec0:readio_latency.<192:19 ext_cache_obj:ec0:readio_chain_lengths.1:63394306 ext_cache_obj:ec0:readio_chain_lengths.2:5874105 ext_cache_obj:ec0:readio_chain_lengths.3:16774665 ext_cache_obj:ec0:readio_chain_lengths.4:1088869 ext_cache_obj:ec0:readio_chain_lengths.5:306521 ext_cache_obj:ec0:readio_chain_lengths.6:871098 ext_cache_obj:ec0:readio_chain_lengths.7:157207 ext_cache_obj:ec0:readio_chain_lengths.<16:434548 ext_cache_obj:ec0:readio_chain_lengths.<24:36558 ext_cache_obj:ec0:readio_chain_lengths.<32:5244 ext_cache_obj:ec0:readio_chain_lengths.<40:3359 ext_cache_obj:ec0:readio_chain_lengths.<48:936 ext_cache_obj:ec0:readio_chain_lengths.<56:632 ext_cache_obj:ec0:readio_chain_lengths.<64:389 ext_cache_obj:ec0:readio_chain_lengths.<128:53 ext_cache_obj:ec0:writeio_solitary:0/s ext_cache_obj:ec0:writeio_chains:50/s ext_cache_obj:ec0:writeio_blocks:3227/s ext_cache_obj:ec0:writeio_in_flight:0 ext_cache_obj:ec0:writeio_max_in_flight:107 ext_cache_obj:ec0:writeio_general_errors:0 ext_cache_obj:ec0:writeio_transient_errors:0 ext_cache_obj:ec0:writeio_avg_chainlength:64.00 ext_cache_obj:ec0:writeio_avg_latency:4.62ms ext_cache_obj:ec0:writeio_latency.0:358 ext_cache_obj:ec0:writeio_latency.1:23690 ext_cache_obj:ec0:writeio_latency.2:41774 ext_cache_obj:ec0:writeio_latency.3:27620 ext_cache_obj:ec0:writeio_latency.4:17766 ext_cache_obj:ec0:writeio_latency.5:14558 ext_cache_obj:ec0:writeio_latency.6:12651 ext_cache_obj:ec0:writeio_latency.7:12580 ext_cache_obj:ec0:writeio_latency.<16:29129 ext_cache_obj:ec0:writeio_latency.<24:2504 ext_cache_obj:ec0:writeio_latency.<32:337 ext_cache_obj:ec0:writeio_latency.<40:83 ext_cache_obj:ec0:writeio_latency.<48:27 ext_cache_obj:ec0:writeio_latency.<56:5 ext_cache_obj:ec0:writeio_latency.<64:2 ext_cache_obj:ec0:writeio_latency.<128:4 ext_cache_obj:ec0:writeio_chain_lengths.<128:183088 ext_cache_obj:ec0:blocks_ref0:41301539 ext_cache_obj:ec0:blocks_ref1:18120071 ext_cache_obj:ec0:blocks_ref2:2121072 ext_cache_obj:ec0:blocks_ref3:1084969 ext_cache_obj:ec0:blocks_ref4:599609 ext_cache_obj:ec0:blocks_ref5:417174 ext_cache_obj:ec0:blocks_ref6:286331 ext_cache_obj:ec0:blocks_ref7:3178099 ext_cache_obj:ec0:blocks_ref0_arrivals:27596243622 ext_cache_obj:ec0:blocks_ref1_arrivals:27622050947 ext_cache_obj:ec0:blocks_ref2_arrivals:10423310000 ext_cache_obj:ec0:blocks_ref3_arrivals:6427750307 ext_cache_obj:ec0:blocks_ref4_arrivals:4495536161 ext_cache_obj:ec0:blocks_ref5_arrivals:3557830045 ext_cache_obj:ec0:blocks_ref6_arrivals:2957867966 ext_cache_obj:ec0:blocks_ref7_arrivals:117062905990 ext_cache_obj:ec0:lru_ticks:3563396220 ext_cache_obj:ec0:invalidates:3526/s ext_cache_obj:ec0:erase_stripe_invalidates:6/s ext_cache_obj:ec0:erase_stripe_blocks_invalidates:3226/s ext_cache_obj:ec0:cache_utilization:77% ext_cache_obj:ec0:pcs_invalidates_flash:0/s ext_cache_obj:ec0:pcs_wraps:0 ext_cache_obj:ec0:pcs_wrap_time:0 ext_cache_obj:ec0:pcs_confidence:100 ext_cache_obj:ec0:pcs_total_ops_insert:0/s ext_cache_obj:ec0:pcs_total_ops_lookup:0/s ext_cache_obj:ec0:pcs_total_ops_invalidate:0/s ext_cache_obj:ec0:pcs_total_ops_misc:0/s ext_cache_obj:ec0:pcs_sampled_ops_insert:0/s ext_cache_obj:ec0:pcs_sampled_ops_lookup:0/s ext_cache_obj:ec0:pcs_sampled_ops_invalidate:0/s ext_cache_obj:ec0:pcs_sampled_ops_misc:0/s ext_cache_obj:ec0:pcs_sampling_rate:0 ---------- sysstat -x 1 ----------- CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 96% 16951 0 0 16951 22579 180997 108622 57263 0 0 4s 67% 100% :f 50% 0 0 0 0 0 0 0 89% 20267 0 0 20267 45941 229706 111661 93433 0 0 5s 51% 100% :f 44% 0 0 0 0 0 0 0 90% 17307 0 0 17307 56792 223833 118109 105672 0 0 3s 48% 100% :f 44% 0 0 0 0 0 0 0 90% 20658 0 0 20658 34301 267718 41860 22109 0 0 7s 50% 38% : 39% 0 0 0 0 0 0 0 77% 12840 0 0 12843 44520 142228 49804 0 0 0 8s 58% 0% - 47% 3 0 0 0 0 0 0 95% 16191 0 0 16191 33156 180821 149909 121486 0 0 5s 53% 58% Dn 46% 0 0 0 0 0 0 0 97% 15198 0 0 15198 23309 199482 100311 43518 0 0 4s 58% 100% :s 42% 0 0 0 0 0 0 0 91% 15363 0 0 15363 22212 216210 82631 62381 0 0 5s 62% 100% :f 46% 0 0 0 0 0 0 0 87% 17662 0 0 17662 36828 209514 118430 79394 0 0 6s 53% 100% :f 49% 0 0 0 0 0 0 0 80% 14307 0 0 14310 47874 166153 74677 54134 0 0 6s 55% 100% :f 49% 3 0 0 0 0 0 0 80% 13905 0 0 13905 42593 202792 49344 49444 0 0 5s 53% 57% : 48% 0 0 0 0 0 0 0 88% 16720 0 0 16720 59662 244810 41227 0 0 0 3s 48% 0% - 44% 0 0 0 0 0 0 0 93% 14717 0 0 14717 17933 223608 155425 102506 0 0 7s 55% 46% Dn 46% 0 0 0 0 0 0 0 96% 13937 0 0 13937 17215 164887 112227 64931 0 0 7s 54% 100% :s 43% 0 0 0 0 0 0 0 92% 13494 0 0 13497 28609 195722 71916 54271 0 0 5s 64% 100% :f 46% 3 0 0 0 0 0 0 93% 22023 0 0 22023 28545 254401 73108 96616 0 0 4s 48% 100% :f 41% 0 0 0 0 0 0 0 88% 23112 0 0 23112 24806 213939 64348 73510 0 0 5s 53% 100% :f 44% 0 0 0 0 0 0 0 84% 17240 0 0 17240 30511 203088 45549 22405 0 0 3s 54% 36% : 41% 0 0 0 0 0 0 0 90% 20676 0 0 20676 61721 269589 35822 24 0 0 4s 51% 0% - 36% 0 0 0 0 0 0 0 84% 19089 0 0 19100 42507 229811 34014 0 0 0 5s 54% 0% - 38% 11 0 0 0 0 0 0 CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 92% 17690 0 0 17690 32740 195063 106798 63482 0 0 5s 51% 33% Dn 40% 0 0 0 0 0 0 0 96% 13450 0 0 13450 18298 175936 97234 80224 0 0 4s 60% 100% :s 39% 0 0 0 0 0 0 0 94% 14813 0 0 14813 13928 204597 69566 61750 0 0 5s 57% 100% :f 39% 0 0 0 0 0 0 0 89% 18505 0 0 18509 23968 247285 104228 69749 0 0 6s 54% 100% :f 47% 4 0 0 0 0 0 0 91% 21963 0 0 21967 60710 223419 78861 66270 0 0 3s 54% 100% :f 42% 4 0 0 0 0 0 0 89% 19919 0 0 19919 33959 215890 79309 59648 0 0 5s 53% 100% :f 45% 0 0 0 0 0 0 0 90% 19707 0 0 19707 27499 247462 35972 33722 0 0 6s 50% 59% : 35% 0 0 0 0 0 0 0 86% 19456 0 0 19456 43880 233384 37526 0 0 0 4s 55% 0% - 38% 0 0 0 0 0 0 0 94% 20130 0 0 20130 63247 229293 78845 48829 0 0 2s 54% 31% Dn 35% 0 0 0 0 0 0 0 96% 10225 0 0 10228 45026 101768 193293 199449 0 0 3s 68% 100% :s 47% 3 0 0 0 0 0 0 95% 13359 0 0 13359 39462 191730 99143 52233 0 0 4s 61% 100% :f 46% 0 0 0 0 0 0 0 87% 15657 0 0 15657 21113 257858 95879 61368 0 0 5s 53% 100% :f 41% 0 0 0 0 0 0 0 89% 19590 0 0 19590 51499 252323 57820 65116 0 0 6s 51% 100% :v 39% 0 0 0 0 0 0 0 77% 14530 0 0 14530 31816 178572 28376 32 0 0 6s 53% 7% : 39% 0 0 0 0 0 0 0 95% 14670 0 0 14673 36120 218937 136512 118201 0 0 5s 55% 54% Dn 45% 3 0 0 0 0 0 0 97% 15592 0 0 15592 24149 170596 136341 76393 0 0 6s 56% 100% :s 41% 0 0 0 0 0 0 0 95% 15562 0 0 15562 46759 158751 158750 58356 0 0 7s 72% 100% :f 59% 0 0 0 0 0 0 0 84% 15471 0 0 15471 15530 229961 145671 73636 0 0 2s 55% 100% :f 47% 0 0 0 0 0 0 0 85% 17753 0 0 17753 53878 214190 53348 46938 0 0 2s 50% 76% : 43% 0 0 0 0 0 0 0 84% 15532 0 0 15535 33703 231932 52108 0 0 0 10s 54% 0% - 48% 3 0 0 0 0 0 0 Summary Statistics ( 3595 samples 0.1 secs/sample) CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out Min 31% 3582 0 0 3582 9751 7799 8404 0 0 0 0s 42% 0% * 12% 0 0 0 0 0 0 0 Avg 87% 13331 0 0 13340 64420 184870 114575 102713 0 0 5s 56% 73% * 43% 8 0 0 0 0 0 0 Max 99% 25894 0 0 25894 215371 661784 322220 330317 0 0 21s 94% 100% * 90% 547 0 0 0 0 0 0 --------- Sysstat -M --------- ANY1+ ANY2+ ANY3+ ANY4+ AVG CPU0 CPU1 CPU2 CPU3 Network Protocol Cluster Storage Raid Target Kahuna WAFL_Ex(Kahu) WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CP 98% 88% 74% 56% 79% 75% 78% 79% 86% 46% 0% 0% 15% 28% 0% 40% 109%( 52%) 10% 0% 0% 65% 4% 1% 13358 81% 98% 88% 73% 55% 79% 75% 77% 80% 86% 46% 0% 0% 13% 27% 0% 41% 108%( 52%) 11% 0% 0% 65% 4% 2% 13152 78% 95% 81% 64% 46% 72% 67% 68% 73% 80% 43% 0% 0% 11% 21% 0% 38% 102%( 50%) 7% 0% 0% 61% 5% 1% 12848 73% 92% 79% 64% 47% 71% 68% 69% 73% 75% 44% 0% 0% 11% 23% 0% 32% 104%( 51%) 8% 0% 0% 57% 5% 1% 13178 64% 90% 78% 64% 48% 71% 68% 69% 72% 73% 44% 0% 0% 11% 24% 0% 30% 101%( 50%) 8% 0% 0% 58% 4% 1% 12550 70% 93% 80% 66% 49% 73% 70% 71% 75% 75% 47% 0% 0% 12% 24% 0% 31% 106%( 53%) 8% 0% 0% 58% 5% 2% 13090 64% 87% 71% 55% 40% 64% 60% 62% 67% 68% 40% 0% 0% 9% 18% 0% 27% 97%( 49%) 6% 0% 0% 51% 5% 1% 12890 67% 91% 77% 61% 45% 69% 65% 67% 71% 74% 43% 0% 0% 11% 22% 0% 32% 100%( 50%) 8% 0% 0% 55% 4% 1% 12697 73% 89% 73% 57% 42% 66% 62% 63% 69% 69% 41% 0% 0% 10% 20% 0% 28% 99%( 50%) 7% 0% 0% 52% 5% 1% 13111 66% 90% 76% 60% 44% 68% 66% 66% 69% 71% 43% 0% 0% 11% 22% 0% 29% 99%( 50%) 8% 0% 0% 55% 4% 2% 12771 72% 87% 71% 55% 40% 64% 60% 62% 66% 67% 41% 0% 0% 10% 18% 0% 27% 96%( 49%) 6% 0% 0% 51% 5% 1% 12675 66% 86% 71% 56% 41% 64% 61% 62% 67% 68% 41% 0% 0% 10% 19% 0% 28% 97%( 48%) 7% 0% 0% 52% 5% 1% 12678 62% 92% 77% 62% 45% 70% 66% 68% 72% 74% 44% 0% 0% 11% 22% 0% 31% 103%( 52%) 8% 0% 0% 55% 5% 1% 13402 75% 94% 81% 67% 49% 74% 69% 72% 76% 77% 45% 0% 0% 12% 26% 0% 33% 104%( 52%) 10% 0% 0% 58% 4% 2% 13289 76% 89% 75% 60% 44% 67% 64% 65% 71% 70% 43% 0% 0% 10% 21% 0% 29% 99%( 50%) 8% 0% 0% 54% 5% 1% 12985 66% 85% 68% 52% 37% 62% 58% 60% 64% 64% 40% 0% 0% 9% 16% 0% 24% 96%( 49%) 6% 0% 0% 49% 5% 1% 12931 64% 90% 75% 61% 45% 69% 64% 66% 72% 72% 42% 0% 0% 10% 21% 0% 30% 103%( 51%) 8% 0% 0% 55% 5% 1% 13163 66% 90% 75% 58% 42% 67% 64% 65% 70% 70% 44% 0% 0% 10% 19% 0% 27% 104%( 52%) 6% 0% 0% 53% 5% 2% 13929 71% 99% 94% 84% 68% 87% 83% 86% 90% 87% 51% 0% 0% 15% 37% 0% 33% 123%( 61%) 14% 0% 0% 69% 4% 1% 14002 81% -- Summary Statistics ( 58 samples 6.0 secs/sample) ANY1+ ANY2+ ANY3+ ANY4+ AVG CPU0 CPU1 CPU2 CPU3 Network Protocol Cluster Storage Raid Target Kahuna WAFL_Ex(Kahu) WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CP Min 85% 68% 52% 37% 62% 58% 60% 64% 64% 40% 0% 0% 9% 16% 0% 24% 95%( 46%) 6% 0% 0% 49% 3% 1% 10226 58% Avg 95% 84% 70% 54% 76% 73% 75% 78% 80% 46% 0% 0% 13% 28% 0% 35% 106%( 51%) 9% 0% 0% 61% 4% 1% 13234 73% Max 100% 96% 87% 70% 89% 84% 93% 90% 91% 56% 0% 0% 26% 49% 0% 49% 123%( 61%) 18% 0% 0% 72% 5% 2% 16583 95% ---------- STATIT ----------- NetApp Release 8.0.5 7-Mode: Sun Jan 20 21:53:54 PST 2013 <1O> Start time: Tue Nov 5 11:25:18 IST 2013 CPU Statistics 60.182572 time (seconds) 100 % 173.781160 system time 289 % 2.716133 rupt time 5 % (1632432 rupts x 2 usec/rupt) 171.065027 non-rupt system time 284 % 66.949128 idle time 111 % 41.250883 time in CP 69 % 100 % 1.790287 rupt time in CP 4 % (1107164 rupts x 2 usec/rupt) Multiprocessor Statistics (per second) cpu0 cpu1 cpu2 cpu3 total sk switches 58254.32 59427.55 57820.04 45838.67 221340.59 hard switches 29026.92 30923.44 35440.13 16979.75 112370.24 domain switches 1174.79 2076.38 238.46 316.05 3805.69 CP rupts 14284.20 1370.85 1370.83 1370.88 18396.75 nonCP rupts 6840.54 629.14 629.14 629.10 8727.91 IPI rupts 0.00 0.00 0.00 0.00 0.00 grab kahuna 0.00 0.98 0.91 0.00 1.89 grab w_xcleaner 0.00 0.18 0.81 268.87 269.86 grab kahuna usec 0.00 488.56 910.73 0.00 1399.29 grab w_xcleaner usec 0.00 123.31 347.31 156723.71 157194.33 CP rupt usec 26808.77 651.73 1779.62 507.44 29747.60 nonCP rupt usec 14115.33 288.34 768.00 212.25 15383.96 idle 318889.16 304923.34 259322.25 229299.07 1112433.85 kahuna 0.47 0.00 0.00 332371.27 332371.75 storage 84402.34 12364.19 16805.60 0.00 113572.15 exempt 199610.16 176903.95 212622.70 13.92 589150.76 raid 815.39 200269.12 22897.81 0.00 223982.33 target 15.04 17.15 16.67 0.00 48.88 netcache 0.00 0.00 0.20 0.00 0.20 netcache2 0.00 0.00 0.00 0.00 0.00 cifs 3.66 3.72 2.77 0.00 10.19 wafl_exempt 254690.06 231801.77 136534.46 437596.00 1060622.33 wafl_xcleaner 33024.31 19066.62 23468.94 0.00 75559.88 sm_exempt 19.36 22.46 21.42 0.00 63.27 cluster 0.00 0.00 0.00 0.00 0.00 protocol 26.95 25.87 26.54 0.00 79.38 nwk_exclusive 975.88 511.33 1150.90 0.00 2638.12 nwk_exempt 261.45 359.64 366.10 0.00 987.23 nwk_legacy 1585.44 1830.31 1686.75 0.00 5102.52 nwk_ctx1 360.39 24181.75 303022.64 0.00 327564.81 nwk_ctx2 59706.95 22852.83 16500.72 0.00 99060.52 nwk_ctx3 0.00 0.00 0.00 0.00 0.00 nwk_ctx4 0.00 0.00 0.00 0.00 0.00 hostOS 4688.73 3925.69 3005.77 0.00 11620.19 56.181715 seconds with one or more CPUs active ( 93%) 48.289199 seconds with 2 or more CPUs active ( 80%) 38.846369 seconds with 3 or more CPUs active ( 65%) 7.892516 seconds with one CPU active ( 13%) 9.442829 seconds with 2 CPUs active ( 16%) 10.092179 seconds with 3 CPUs active ( 17%) 28.754190 seconds with all CPUs active ( 48%) Domain Utilization of Shared Domains (per second) 0.00 idle 516212.17 kahuna 0.00 storage 0.00 exempt 0.00 raid 0.00 target 0.00 netcache 0.00 netcache2 0.00 cifs 0.00 wafl_exempt 0.00 wafl_xcleaner 0.00 sm_exempt 0.00 cluster 0.00 protocol 392429.57 nwk_exclusive 0.00 nwk_exempt 0.00 nwk_legacy 0.00 nwk_ctx1 0.00 nwk_ctx2 0.00 nwk_ctx3 0.00 nwk_ctx4 0.00 hostOS Miscellaneous Statistics (per second) 112370.24 hard context switches 13517.83 NFS operations 0.00 CIFS operations 0.00 HTTP operations 44559.59 network KB received 184368.21 network KB transmitted 94098.07 disk KB read 78261.13 disk KB written 40535.46 NVRAM KB written 0.00 nolog KB written 45831.13 WAFL bufs given to clients 0.80 checksum cache hits ( 0%) 45787.34 no checksum - partial buffer 0.00 FCP operations 0.00 iSCSI operations WAFL Statistics (per second) 81.17 name cache hits ( 77%) 24.81 name cache misses ( 23%) 432838.23 buf hash hits ( 87%) 62737.71 buf hash misses ( 13%) 29606.73 inode cache hits ( 100%) 0.00 inode cache misses ( 0%) 47745.08 buf cache hits ( 55%) 39604.57 buf cache misses ( 45%) 2150.64 blocks read 5882.00 blocks read-ahead 1252.07 chains read-ahead 3674.07 dummy reads 4269.61 blocks speculative read-ahead 15858.58 blocks written 30.57 stripes written 0.00 blocks over-written 0.00 wafl_timer generated CP 0.00 snapshot generated CP 0.00 wafl_avail_bufs generated CP 0.02 dirty_blk_cnt generated CP 0.02 full NV-log generated CP 0.00 back-to-back CP 0.00 flush generated CP 0.00 sync generated CP 0.00 deferred back-to-back CP 0.00 container-indirect-pin CP 0.03 low mbufs generated CP 0.13 low datavecs generated CP 59836.91 non-restart messages 516.21 IOWAIT suspends 142731372.60 next nvlog nearly full msecs 0.00 dirty buffer susp msecs 0.00 nvlog full susp msecs 933124 buffers RAID Statistics (per second) 3433.15 xors 0.00 long dispatches [0] 0.00 long consumed [0] 0.00 long consumed hipri [0] 0.00 long low priority [0] 0.00 long high priority [0] 0.00 long monitor tics [0] 0.00 long monitor clears [0] 0.00 long dispatches [1] 0.00 long consumed [1] 0.00 long consumed hipri [1] 0.00 long low priority [1] 0.00 long high priority [1] 0.00 long monitor tics [1] 0.00 long monitor clears [1] 18 max batch 6.95 blocked mode xor 1220.85 timed mode xor 2.18 fast adjustments 2.63 slow adjustments 0 avg batch start 0 avg stripe/msec 3737.36 checksum dispatches 23508.15 checksum consumed 31.42 tetrises written 0.00 master tetrises 0.00 slave tetrises 1729.77 stripes written 1713.19 partial stripes 16.58 full stripes 16095.72 blocks written 11922.54 blocks read 126.73 1 blocks per stripe size 21 100.83 2 blocks per stripe size 21 94.76 3 blocks per stripe size 21 96.27 4 blocks per stripe size 21 96.14 5 blocks per stripe size 21 96.47 6 blocks per stripe size 21 100.00 7 blocks per stripe size 21 102.87 8 blocks per stripe size 21 100.89 9 blocks per stripe size 21 98.15 10 blocks per stripe size 21 95.36 11 blocks per stripe size 21 91.70 12 blocks per stripe size 21 86.60 13 blocks per stripe size 21 84.97 14 blocks per stripe size 21 78.96 15 blocks per stripe size 21 71.90 16 blocks per stripe size 21 60.37 17 blocks per stripe size 21 54.10 18 blocks per stripe size 21 44.07 19 blocks per stripe size 21 32.04 20 blocks per stripe size 21 16.58 21 blocks per stripe size 21 Network Interface Statistics (per second) iface side bytes packets multicasts errors collisions pkt drops e0a recv 3683.86 50.26 0.00 0.00 0.00 xmit 5048.36 19.94 0.00 0.00 0.00 e0b recv 4408.92 59.15 0.00 0.00 0.00 xmit 20.14 0.10 0.00 0.00 0.00 e2a recv 45620024.58 50126.21 0.00 0.00 0.00 xmit 188787476.51 29046.10 0.18 0.00 0.00 e2b recv 134.66 1.88 0.00 0.00 0.00 xmit 6.98 0.17 0.17 0.00 0.00 e6a recv 71.10 0.91 0.00 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00 e6b recv 59.10 0.63 0.00 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00 e6c recv 0.00 0.00 0.00 71365630.73 0.00 xmit 0.00 0.00 0.00 0.00 0.00 e6d recv 0.00 0.00 0.00 71365630.72 0.00 xmit 0.00 0.00 0.00 0.00 0.00 c0a recv 285.13 21.93 1.46 0.00 0.00 xmit 216.34 1.16 1.16 0.00 0.00 c0b recv 285.13 1.46 1.46 0.00 0.00 xmit 216.34 1.16 1.16 0.00 0.00 e0M recv 0.00 0.00 0.00 0.15 0.00 xmit 0.00 0.00 0.00 0.00 0.00 e0P recv 80.09 0.93 0.08 0.00 0.00 xmit 66.50 0.85 0.00 0.00 0.00 vh recv 0.00 0.00 0.00 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00 vif0-manage recv 8092.77 109.42 89.15 0.00 0.00 xmit 5068.49 20.04 0.00 0.00 0.00 vif1-10g recv 45620159.24 50128.08 3.74 0.70 0.00 xmit 188787483.49 29046.27 0.35 0.00 0.00 vif-174-1g recv 130.20 1.55 1.55 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00 Disk Statistics (per second) ut% is the percent of time the disk was busy. xfers is the number of data-transfer commands issued per second. xfers = ureads + writes + cpreads + greads + gwrites chain is the average number of 4K blocks per command. usecs is the average disk round-trip time per 4K block. disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs /aggr0_sas_600g/plex0/rg0: 0a.01.4 8 23.16 0.76 1.39 2859 9.29 46.62 120 13.11 19.58 181 0.00 .... . 0.00 .... . 0a.01.5 9 23.76 0.80 1.38 3394 9.80 44.28 119 13.16 19.71 199 0.00 .... . 0.00 .... . 0a.01.6 25 64.90 42.39 2.39 3282 6.23 27.14 297 16.28 9.66 441 0.00 .... . 0.00 .... . 0a.02.13 25 67.20 44.22 2.14 3537 6.06 29.12 311 16.92 8.81 470 0.00 .... . 0.00 .... . 0a.02.2 25 63.74 40.44 2.13 3620 6.25 29.57 297 17.05 9.75 393 0.00 .... . 0.00 .... . 0a.03.19 24 62.74 38.78 2.40 3366 6.28 27.77 293 17.68 9.16 365 0.00 .... . 0.00 .... . 0a.03.16 24 63.24 40.51 2.33 3237 6.10 29.41 282 16.63 9.34 403 0.00 .... . 0.00 .... . 0a.02.9 24 61.31 37.85 2.26 3474 5.95 29.06 289 17.51 8.78 460 0.00 .... . 0.00 .... . 0a.02.14 24 61.83 38.05 2.37 3185 6.16 27.20 309 17.61 8.69 474 0.00 .... . 0.00 .... . 0a.02.10 23 60.75 37.97 2.33 3423 5.97 29.50 283 16.82 8.62 452 0.00 .... . 0.00 .... . 0a.02.7 23 57.48 35.26 2.33 3412 5.85 28.95 310 16.37 8.65 440 0.00 .... . 0.00 .... . 0a.03.11 24 61.75 38.92 2.24 3490 5.93 29.31 298 16.90 8.47 453 0.00 .... . 0.00 .... . 0a.03.4 23 61.35 39.36 2.21 3562 5.78 28.82 291 16.20 9.83 365 0.00 .... . 0.00 .... . 0a.02.16 24 61.70 38.00 2.40 3473 6.11 27.69 312 17.58 8.85 491 0.00 .... . 0.00 .... . 0a.03.3 23 62.76 39.20 2.62 2912 6.08 29.18 281 17.48 8.68 386 0.00 .... . 0.00 .... . 0a.03.7 24 62.99 40.16 2.42 3037 6.16 29.02 294 16.67 9.64 374 0.00 .... . 0.00 .... . 0a.03.5 24 63.29 39.66 2.40 3344 6.15 27.10 315 17.48 8.68 428 0.00 .... . 0.00 .... . 0a.01.19 24 62.24 39.45 2.51 3041 6.11 27.71 303 16.68 9.47 511 0.00 .... . 0.00 .... . 0a.01.17 24 62.38 39.51 2.16 3351 6.02 29.16 308 16.85 8.97 564 0.00 .... . 0.00 .... . 0a.01.16 24 61.46 38.62 2.34 3270 6.18 28.09 306 16.67 8.64 555 0.00 .... . 0.00 .... . 0a.01.14 25 64.27 40.96 2.52 3063 6.16 28.16 296 17.15 9.97 454 0.00 .... . 0.00 .... . 0a.03.13 24 62.54 39.08 2.75 2821 6.15 28.75 299 17.31 9.30 443 0.00 .... . 0.00 .... . 0a.03.9 24 64.59 41.79 2.54 3015 5.93 28.06 323 16.87 8.91 428 0.00 .... . 0.00 .... . /aggr0_sas_600g/plex0/rg1: 0a.03.8 7 21.60 0.00 .... . 7.78 54.57 120 13.82 19.67 161 0.00 .... . 0.00 .... . 0a.02.21 8 21.72 0.00 .... . 7.89 53.80 124 13.82 19.64 194 0.00 .... . 0.00 .... . 0a.03.18 25 63.57 41.21 2.68 2986 6.03 28.37 332 16.33 9.35 418 0.00 .... . 0.00 .... . 0a.03.0 24 62.14 40.19 2.34 3235 6.05 28.49 307 15.90 9.74 336 0.00 .... . 0.00 .... . 0a.02.18 25 62.19 41.27 2.16 3611 5.83 28.69 318 15.09 9.18 451 0.00 .... . 0.00 .... . 0a.02.15 24 62.18 40.48 2.36 3201 5.95 27.20 327 15.75 8.67 456 0.00 .... . 0.00 .... . 0a.03.21 24 63.62 42.40 2.29 3318 5.92 27.13 336 15.30 9.85 386 0.00 .... . 0.00 .... . 0a.03.12 24 61.95 40.08 2.41 2970 5.82 27.61 334 16.05 9.34 380 0.00 .... . 0.00 .... . 0a.02.20 25 63.19 41.42 2.41 3333 5.85 28.05 312 15.92 8.87 482 0.00 .... . 0.00 .... . 0a.02.23 25 62.86 40.58 2.20 3548 6.02 27.25 324 16.27 8.83 520 0.00 .... . 0.00 .... . 0a.02.12 25 64.39 42.82 2.43 3185 5.80 27.85 315 15.77 8.92 464 0.00 .... . 0.00 .... . 0a.03.20 24 63.44 41.57 2.34 3159 5.87 27.65 303 16.00 8.72 423 0.00 .... . 0.00 .... . 0a.03.14 24 62.74 40.76 2.38 3287 6.08 28.84 310 15.90 9.07 427 0.00 .... . 0.00 .... . 0a.02.5 24 61.81 40.05 2.14 3729 6.02 28.72 311 15.75 8.73 506 0.00 .... . 0.00 .... . 0a.01.2 26 62.73 41.37 2.37 3226 6.11 27.65 335 15.24 8.99 552 0.00 .... . 0.00 .... . 0a.01.3 24 57.66 35.92 2.39 3542 5.82 26.97 338 15.92 8.93 516 0.00 .... . 0.00 .... . 0a.01.10 24 61.45 39.43 2.40 3215 6.18 28.27 315 15.84 8.81 545 0.00 .... . 0.00 .... . 0a.01.23 24 60.92 39.20 2.54 2881 5.90 26.77 344 15.82 9.01 574 0.00 .... . 0.00 .... . 0a.01.15 24 60.35 38.05 2.33 3345 6.02 28.27 305 16.28 8.92 555 0.00 .... . 0.00 .... . 0a.01.21 25 63.16 41.62 2.31 3374 5.65 29.21 306 15.89 8.90 572 0.00 .... . 0.00 .... . 0a.01.13 25 62.38 40.58 2.51 3214 5.88 27.73 338 15.92 9.22 573 0.00 .... . 0.00 .... . 0a.01.8 26 64.69 43.12 2.35 3346 5.80 27.22 330 15.77 9.34 530 0.00 .... . 0.00 .... . 0a.01.12 25 62.34 40.58 2.29 3404 5.92 28.99 316 15.85 9.06 557 0.00 .... . 0.00 .... . /aggr0_sas_600g/plex0/rg2: 0a.02.0 7 20.47 0.00 .... . 7.69 55.57 119 12.78 18.03 178 0.00 .... . 0.00 .... . 0a.03.1 7 20.59 0.00 .... . 7.81 54.77 120 12.78 17.99 163 0.00 .... . 0.00 .... . 0a.01.0 28 66.68 44.17 2.32 3498 6.41 30.22 311 16.10 9.73 506 0.00 .... . 0.00 .... . 0a.02.1 26 64.67 42.92 2.31 3459 6.26 31.12 317 15.49 9.80 449 0.00 .... . 0.00 .... . 0a.03.2 26 65.77 43.57 2.47 3255 6.08 30.55 307 16.12 9.35 422 0.00 .... . 0.00 .... . 0a.01.1 27 67.36 45.53 2.36 3291 6.23 31.81 283 15.60 10.42 418 0.00 .... . 0.00 .... . 0a.02.3 27 66.85 45.00 2.19 3569 6.05 29.12 317 15.80 9.93 425 0.00 .... . 0.00 .... . 0a.03.6 27 70.77 47.82 2.21 3569 6.38 30.96 308 16.57 9.28 410 0.00 .... . 0.00 .... . 0a.01.7 27 65.07 42.47 2.37 3412 6.25 30.77 305 16.35 9.73 489 0.00 .... . 0.00 .... . 0a.02.4 27 65.73 44.65 2.46 3134 5.93 30.35 302 15.15 10.49 414 0.00 .... . 0.00 .... . 0a.03.10 27 66.80 44.37 2.24 3635 6.41 30.98 311 16.02 9.60 410 0.00 .... . 0.00 .... . 0a.01.9 27 66.53 44.56 2.25 3698 6.11 29.33 320 15.85 10.37 502 0.00 .... . 0.00 .... . 0a.02.6 26 64.67 42.16 2.38 3264 6.15 32.52 293 16.37 9.80 452 0.00 .... . 0.00 .... . 0a.03.15 28 68.97 46.26 2.12 3974 6.21 30.80 327 16.50 9.21 487 0.00 .... . 0.00 .... . 0a.01.11 27 67.25 44.28 2.27 3737 6.38 30.34 320 16.58 9.04 607 0.00 .... . 0.00 .... . 0a.02.8 26 65.32 41.81 2.43 3292 6.28 30.20 321 17.23 8.81 517 0.00 .... . 0.00 .... . 0a.03.17 28 67.45 44.95 2.20 3792 6.18 30.23 329 16.32 9.06 494 0.00 .... . 0.00 .... . 0a.01.18 27 66.80 44.78 2.28 3647 6.18 29.31 336 15.84 9.67 557 0.00 .... . 0.00 .... . 0a.02.11 27 64.69 42.47 2.27 3677 6.26 29.47 311 15.95 10.48 463 0.00 .... . 0.00 .... . 0a.03.22 27 65.58 43.82 2.34 3382 5.97 29.82 312 15.80 9.65 477 0.00 .... . 0.00 .... . 0a.01.20 28 67.93 45.13 2.47 3315 6.45 29.24 322 16.35 9.60 588 0.00 .... . 0.00 .... . 0a.02.17 28 67.00 43.77 2.34 3529 6.48 29.63 332 16.75 9.66 516 0.00 .... . 0.00 .... . 0a.03.23 27 70.05 46.68 2.25 3592 6.33 30.59 320 17.05 9.20 509 0.00 .... . 0.00 .... . /aggr0_sas_600g/plex0/rg3: 3b.05.0 6 18.91 0.00 .... . 7.86 56.64 108 11.05 15.07 148 0.00 .... . 0.00 .... . 0a.02.19 6 19.03 0.00 .... . 7.98 55.84 110 11.05 15.07 179 0.00 .... . 0.00 .... . 0a.01.22 30 74.03 51.36 2.46 3262 6.58 35.44 279 16.08 10.13 494 0.00 .... . 0.00 .... . 3b.05.1 31 77.13 54.15 2.37 3481 6.75 37.11 262 16.23 9.43 452 0.00 .... . 0.00 .... . 0a.02.22 31 77.30 54.24 2.49 3373 6.55 35.55 284 16.52 9.43 498 0.00 .... . 0.00 .... . 3b.05.2 30 73.74 50.76 2.35 3684 6.75 36.56 275 16.23 9.47 419 0.00 .... . 0.00 .... . 3b.05.20 31 76.57 54.02 2.50 3139 6.70 35.59 259 15.85 9.89 448 0.00 .... . 0.00 .... . 3b.05.4 30 76.29 54.19 2.34 3523 6.43 37.21 236 15.67 10.22 378 0.00 .... . 0.00 .... . 3b.05.5 29 74.09 52.11 2.32 3478 6.46 36.30 247 15.52 9.83 382 0.00 .... . 0.00 .... . 3b.05.6 30 74.94 53.11 2.60 3120 6.48 34.96 273 15.35 10.44 401 0.00 .... . 0.00 .... . 3b.05.7 30 76.80 54.24 2.51 3188 6.75 34.71 280 15.82 10.31 396 0.00 .... . 0.00 .... . 3b.05.8 31 79.71 56.46 2.40 3347 6.66 35.86 278 16.58 9.35 382 0.00 .... . 0.00 .... . 3b.05.9 30 74.57 52.08 2.41 3424 6.70 35.91 262 15.80 9.59 426 0.00 .... . 0.00 .... . 3b.05.10 30 76.72 54.43 2.52 3285 6.65 35.93 253 15.64 9.97 437 0.00 .... . 0.00 .... . 3b.05.11 32 76.53 54.07 2.38 3536 6.83 33.47 301 15.64 9.76 463 0.00 .... . 0.00 .... . 3b.05.12 31 77.58 55.02 2.42 3403 6.65 35.36 262 15.92 10.24 430 0.00 .... . 0.00 .... . 3b.05.13 31 75.65 52.47 2.35 3424 6.76 35.10 270 16.42 9.69 481 0.00 .... . 0.00 .... . 3b.05.14 31 75.22 52.61 2.53 3314 6.71 37.28 251 15.90 9.64 455 0.00 .... . 0.00 .... . 3b.05.15 30 74.26 51.59 2.52 3221 6.55 36.06 266 16.12 9.73 439 0.00 .... . 0.00 .... . 3b.05.16 32 78.20 55.80 2.37 3604 6.70 36.85 267 15.70 9.54 527 0.00 .... . 0.00 .... . 3b.05.17 31 75.70 53.24 2.42 3446 6.58 36.37 261 15.89 9.55 492 0.00 .... . 0.00 .... . 3b.05.18 31 77.58 55.10 2.35 3422 6.68 35.76 272 15.80 9.65 515 0.00 .... . 0.00 .... . 3b.05.19 30 75.87 53.40 2.57 3261 6.65 36.61 271 15.82 9.57 542 0.00 .... . 0.00 .... . Aggregate statistics: Minimum 6 18.91 0.00 5.65 11.05 0.00 0.00 Mean 25 62.98 40.64 6.40 15.92 0.00 0.00 Maximum 32 79.71 56.46 9.80 17.68 0.00 0.00 Alon Zeltser Solution Architect ,E&M Computing Tel: 972-35766966 (227) | Mobile: 972-544770108 alonz(a)emet.co.il| www.emet.co.il emet_logo

3 2

Which drives and shelves support full drive encryption?
by Mike Borkowski 06 Nov '13

06 Nov '13

We are considering using Netapp's full drive encryption, I can't seem to find a list of supported drives and shelves. Anyone able to point me at the information? Thanks. Mike Borkowski Manager, Storage & Servers, IST University of Waterloo

2 2

ndmpcopy and vol copy via ssh?
by Jeff Cleverley 01 Nov '13

01 Nov '13

Greetings, Is it acceptable to use ssh from an admin host to run ndmpcopy and vol copy commands? It seems to work initially, but both seem to die after varying periods of time with no useful explanations in the log files. I found a KB about someone connecting to the filer and running ctrl-c thinking the console is idle, but that doesn't apply here unless an ssh command from a cron job or something has the same effect. ndmpd status and vol copy status verify that there are no copies running. All filers are 8.1.2P4, source is a 6080, destination is a 6290, 10G networking on both. I'm doing restore time tests of entire volumes using various methods and these 2 are part of the list. The vol copy ran for ~10 minutes out of the 466 estimated, then stopped. No errors on the command line. It does run, so all the host.equiv entries and permissions are good. I'm sure it did not do the 11TB in 10 minutes :-) The ndmpcopy command ran for ~80 minutes and silently quit. It restored 217G, almost all are in smaller directories. The larger directories didn't show up and there are no abort messages in the logs. They just quit logging. For the ndmpcopy I had a script do an ndmpcopy for each directory (9 total). It looked like multiple ones were running at the same time which seems OK. I was hoping to use parallel threads to speed things up. Any ideas on the silent failures? Thanks, Jeff -- Jeff Cleverley Unix Systems Administrator 4380 Ziegler Road Fort Collins, Colorado 80525 970-288-4611

5 10

Methods to offline transfer a large amount of data between two filers
by Strauss Wolfram 01 Nov '13

01 Nov '13

The scenario is as follows: Main site with a filer used as a Nearstore System, in particular as a Snapvault destination. A branch site with a smaller filer backing up its data every night with Snapvault to the main site. Both sites are connected over a rather low bandwidth WAN connection. Restoring single files from the main site's Nearstore System to the branch site's filer is no big deal. But what if we have to restore a complete volume in the Terabyte size? Over the WAN connection this would take days to complete. Here comes in the offline transfer scenario like so 1. Make a dump of the volume from the Nearstore System to a portable raid box 2. Hop in the car with the raid box and drive to the branch site (about 100 kilometers away) 3. Restore the volume from the raid box to the branch site filer The crux is how to dump a volume to the portable raid box fast enough. I haven't found a way to directly connect external storage to a filer so we definitely need to use a computer to connect the raid box to and then dump the data from the filer over the network onto this computer. We expect to more or less saturate a 1 Gbits/s link. This would allow us to carry out a restore of a 1 Terabyte volume within one working day. So far we have tried - Copy the files directly over NFS / CIFS -> too slow as expected, especially if there are many files and what if the volume contains LUNs and/or qtrees? - Using NetApp's LREP tool (offline SnapVault) -> astonishingly too slow, we only achieved about 200 Mbits/s although neither the filer nor the laptop used in the scenario hit any resource limit - NetApp's dump command to dump the volume into a file on a second volume and then transfer the dump file over NFS onto the laptop and the raid box. On the branch site copy the dump file over NFS to a second volume on the filer and then restore the main volume from the dump file. -> Especially the creation of the dump file was quite slow and the method doubles space requirements on both filers Out of curiosity, since all the above methods were too slow, I tried the following. I connected a LUN over iSCSI to the laptop used in the scenario and copied the files from the LUN onto the raid box. I easily managed to saturate the 1 Gbits/s link even with many small files residing in the LUN. So basically, the filer, the laptop and the raid box are capable of saturating 1 Gbits/s. What I'm missing is a method to dump a hole volume, including all its sub structures like qtrees and LUNs, in a decent way out of a filer. Does anybody have tried something similar and found a performing solution or did anybody succeed with one of the three above mentions methods and if so, how exactly did your scenario look like? Regards, Wolfram Strauss

3 2

RE: Methods to offline transfer a large amount of data between two filers
by Strauss Wolfram 01 Nov '13

01 Nov '13

Hi Mike It doesn't have to be disk based storage but we have to be able to connect to it over Ethernet. So, if there is a way to have a portable tape drive with an Ethernet connection that can be used by smtape, then it would sound like a promising solution. It should be a portable tape drive since we have about ten branch sites and I don't think we are going to install tape drives at every site. Regards, Wolfram Strauss From: Mike Nye [mailto:Mike.Nye@xpanse.com.au] Sent: Freitag, 1. November 2013 11:15 To: Strauss Wolfram Subject: RE: Methods to offline transfer a large amount of data between two filers Hi Wolfram, Does it have to be a RAID box? How about SnapMirror to Tape (smtape) ? Kind regards, Mike Nye [Description: cid:image001.jpg@01C9F9A5.3B7B34A0] [Description: cid:image002.jpg@01C9F9A5.3B7B34A0] Mike Nye Team Lead Systems Engineer mike.nye(a)xpanse.com.au<mailto:mike.nye@xpanse.com.au> www.xpanse.com.au<http://www.xpanse.com.au/> Mob: Office: Fax: +61 407 772 465 +61 8 9322 6767 +61 8 9322 6077 18 Emerald Tce West Perth WA 6005 [Description: cid:image003.jpg@01C9F9A5.3B7B34A0] From: toasters-bounces(a)teaparty.net<mailto:toasters-bounces@teaparty.net> [mailto:toasters-bounces@teaparty.net] On Behalf Of Strauss Wolfram Sent: Friday, 1 November 2013 6:07 PM To: toasters(a)teaparty.net<mailto:toasters@teaparty.net> Subject: Methods to offline transfer a large amount of data between two filers The scenario is as follows: Main site with a filer used as a Nearstore System, in particular as a Snapvault destination. A branch site with a smaller filer backing up its data every night with Snapvault to the main site. Both sites are connected over a rather low bandwidth WAN connection. Restoring single files from the main site's Nearstore System to the branch site's filer is no big deal. But what if we have to restore a complete volume in the Terabyte size? Over the WAN connection this would take days to complete. Here comes in the offline transfer scenario like so 1. Make a dump of the volume from the Nearstore System to a portable raid box 2. Hop in the car with the raid box and drive to the branch site (about 100 kilometers away) 3. Restore the volume from the raid box to the branch site filer The crux is how to dump a volume to the portable raid box fast enough. I haven't found a way to directly connect external storage to a filer so we definitely need to use a computer to connect the raid box to and then dump the data from the filer over the network onto this computer. We expect to more or less saturate a 1 Gbits/s link. This would allow us to carry out a restore of a 1 Terabyte volume within one working day. So far we have tried - Copy the files directly over NFS / CIFS -> too slow as expected, especially if there are many files and what if the volume contains LUNs and/or qtrees? - Using NetApp's LREP tool (offline SnapVault) -> astonishingly too slow, we only achieved about 200 Mbits/s although neither the filer nor the laptop used in the scenario hit any resource limit - NetApp's dump command to dump the volume into a file on a second volume and then transfer the dump file over NFS onto the laptop and the raid box. On the branch site copy the dump file over NFS to a second volume on the filer and then restore the main volume from the dump file. -> Especially the creation of the dump file was quite slow and the method doubles space requirements on both filers Out of curiosity, since all the above methods were too slow, I tried the following. I connected a LUN over iSCSI to the laptop used in the scenario and copied the files from the LUN onto the raid box. I easily managed to saturate the 1 Gbits/s link even with many small files residing in the LUN. So basically, the filer, the laptop and the raid box are capable of saturating 1 Gbits/s. What I'm missing is a method to dump a hole volume, including all its sub structures like qtrees and LUNs, in a decent way out of a filer. Does anybody have tried something similar and found a performing solution or did anybody succeed with one of the three above mentions methods and if so, how exactly did your scenario look like? Regards, Wolfram Strauss

1 0

AW: Dedup killing performance (again)
by Alexander Griesser 27 Oct '13

27 Oct '13

Is it stable? I mean 8.1.3 is just in GA for now... Alexander Griesser System-Administrator ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-320 Telefax: +43-5-0556-500 E-Mail: ag(a)anexia.at<mailto:ag@anexia.at> Web: http://www.anexia.at<http://www.anexia.at/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 Von: Peter-Paul Witta [mailto:paul.witta@CUBiT.at] Gesendet: Samstag, 19. Oktober 2013 00:30 An: Alexander Griesser; Sebastian Goetze Cc: toasters(a)teaparty.net Betreff: Re: Dedup killing performance (again) I'd recommend 813p2 immediately... lg paul witta Am 19.10.13 00:19, schrieb Alexander Griesser: Yah, just found it while googling I guess, must be bug id #657692 :-/ Anyhow, I'm now stuck with this: > sis status Path State Status Progress /vol/CIFS_SHARES Enabled Active Fingerprint is being recovered from aggregate (6% complete) It's doing that since quite some time now, although it now actually jumped from 0% to 6%, so it at least is still alive. This bug is first fixed in 8.1.2P4 as it seems, so I might need to upgrade ASAP; for now, I'll check if it at least finishes this process now, my volume is filling up quickly, but there's still some aggregate space left so I'll just increase the volume size before it goes full. I'll go above the magic 80% fill of an aggregate then, but better that then having the volume go full. I'll see how it goes and if it troubles me too much, I hope there is a way to abort sis in that state. Alexander Griesser System-Administrator ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-320 Telefax: +43-5-0556-500 E-Mail: ag(a)anexia.at<mailto:ag@anexia.at> Web: http://www.anexia.at<http://www.anexia.at/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 Von: Sebastian Goetze [mailto:spgoetze@gmail.com] Gesendet: Samstag, 19. Oktober 2013 00:16 An: Alexander Griesser Betreff: Re: Dedup killing performance (again) Isn't that the Dedupe bug again, that was fixed in 8.1.3? (since like 8.1.2P3 or P4)... There was a thread about it some time ago... Anyway, I'm off to bed, just wanted to point you in the right direction. BFN On 10/18/2013 11:52 PM, Alexander Griesser wrote: Hi there, I am just trying to dedupe some volumes on one of our filers and the first volume (2.4TB data, CIFS share) deduplicates since about 4 hours now. Sysstat -x 1 shows: CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 58% 115 0 0 116 2239 3635 281593 32 0 0 0s 98% 0% - 100% 0 0 1 0 0 0 0 63% 162 0 0 169 1682 4149 310829 0 0 0 0s 99% 0% - 100% 7 0 0 0 0 0 0 50% 381 0 0 384 11793 11325 182100 0 0 0 0s 96% 0% - 100% 0 0 3 0 0 0 0 55% 412 0 0 416 6487 6040 223036 24 0 0 0s 97% 0% - 100% 0 0 4 0 0 37 0 51% 19 0 0 19 185 9 239388 0 0 0 2 98% 0% - 100% 0 0 0 0 0 0 0 51% 77 0 0 80 348 906 273880 8 0 0 2 99% 0% - 100% 0 0 3 0 0 0 0 60% 48 0 0 49 174 410 196179 271 0 0 2 98% 35% Tn 100% 0 0 1 0 0 0 0 45% 102 0 0 110 337 337 144387 21222 0 0 2 96% 100% :f 100% 8 0 0 0 0 0 0 71% 143 0 0 149 258 2814 109728 24268 0 0 2 85% 100% :f 86% 0 0 6 0 0 25 0 53% 64 0 0 65 203 1368 236620 14152 0 0 2 98% 100% :f 100% 0 0 1 0 0 0 0 49% 96 0 0 96 319 1317 185552 19748 0 0 2 96% 100% :f 100% 0 0 0 0 0 0 0 35% 109 0 0 117 381 1962 62613 18010 0 0 2 88% 100% :f 100% 5 0 3 0 0 0 0 55% 38 0 0 39 159 1168 239408 1980 0 0 2 97% 100% :f 100% 0 0 1 0 0 0 0 57% 177 0 0 179 308 6692 136568 520 0 0 2 92% 82% : 99% 0 0 2 0 0 37 0 39% 256 0 0 271 403 12647 117284 0 0 0 2 93% 0% - 97% 12 0 3 0 0 0 0 40% 232 0 0 473 808 7506 162368 32 0 0 2 96% 0% - 100% 240 0 1 0 0 0 0 38% 86 0 0 86 246 3413 160092 0 0 0 2 96% 0% - 100% 0 0 0 0 0 0 0 47% 113 0 0 117 314 1738 199516 0 0 0 2 96% 0% - 100% 1 0 3 0 0 0 0 59% 116 0 0 120 476 2751 288048 24 0 0 0s 99% 0% - 100% 0 0 4 0 0 24 0 45% 230 0 0 230 2879 6547 185787 120 0 0 0s 98% 0% - 100% 0 0 0 0 0 0 0 > version NetApp Release 8.1.2P2 7-Mode: Wed Feb 20 19:58:27 PST 2013 I'm aware that deduplication _SHOULD_ be a background process hugging up as much resources as it can but when host i/o is needed, it will be preferred over deduplication. There are no other tasks running at the same time (snapvault, etc.), and the virtual machinse on this system are _VERY_ slow - f.ex. logging in via RDP on one of the virtual machines now took > 1 minute for the login window to appear. This is a FAS2240 with 24x600 SAS 10k and basically my question now is: Is there any way to tell a-sis to really run with lower priority or is this a bug in the OnTap version I'm running? I've used a-sis a lot in the past and never had performance issues like this when it was running... I'm thankful for any ideas how to track down this issue or improve performance while a-sis is running. Thanks, Alexander Griesser System-Administrator ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-320 Telefax: +43-5-0556-500 E-Mail: ag(a)anexia.at<mailto:ag@anexia.at> Web: http://www.anexia.at<http://www.anexia.at/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 _______________________________________________ Toasters mailing list Toasters(a)teaparty.net<mailto:Toasters@teaparty.net> http://www.teaparty.net/mailman/listinfo/toasters _______________________________________________ Toasters mailing list Toasters(a)teaparty.net<mailto:Toasters@teaparty.net> http://www.teaparty.net/mailman/listinfo/toasters -- kind regards, +43-664-4542287 Ing. Peter-Paul Witta +43-1-7189880-0 Chief Technology Officer www.CUBiT.at<http://www.CUBiT.at>

4 5

Thin and Over subscription
by Iluhes 25 Oct '13

25 Oct '13

Dear toasters, Did anyone seen any script that provides an oversubscription rate for a volume. Something like 1. Check for thin lun size allocated size 2. Compare that to "Occupied" size 3. Compare that to volume size And give a oversubscription rate Thanks in advance

1 0

cDOT CIFS setup
by Aaron Lewis 25 Oct '13

25 Oct '13

Has anyone seen the error listed below while setting up CIFS in cDOT? The filer is running 8.2P3. This has been a very straightforward process in the past for me, but with the newness of cDOT, I'm not finding a lot of information when I search. Thanks! netapp::vserver cifs> create -cifs-server "testcifs" -domain " XXXXXX.local" In order to create an Active Directory machine account for the CIFS server, you must supply the name and password of a Windows account with sufficient privileges to add computers to the "CN=Computers" container within the " XXXXXX.local" domain. Enter the user name: XXXXXX Enter the password: Error: CIFS server creation procedure failed [ 27] Loaded the preliminary configuration. [ 62] Created a machine account for the Cifs server in the domain [ 62] SID to name translations of Domain Users and Admins completed successfully **[ 23082] FAILURE: Kerberos password set for ** 'TESTCIFS$(a)XXXXXX.LOCAL' failed with Cannot contact any ** KDC for requested realm [ 23104] Deleted existing account 'CN=TESTCIFS,CN=Computers,DC=xxxxxxx,DC=local' Error: command failed: Failed to create CIFS server TESTCIFS. Reason: Kerberos Error: KDC Unreachable. netapp::vserver cifs> Regards, Aaron

7 7

CIFS Scanner
by Jordan Slingerland 23 Oct '13

23 Oct '13

Does anyone know of a good tool that would scan a cifs share for meta data and possibly put the information in a database that can be queried for Creator Owner Information. For example, the netapp "quota report" command shows a user owns 100GB and 5000 files. I would like to give said user a list of the 5000 files and their sizes. Alternatively, a less sophisticated too that accomplishes the same result that is better than a power shell script that could scan for this information for single user or as needed. In the past I have gathered this information to a text file with a power shell script and it was very messy and certainly less then optimal. --JMS For the fourth consecutive year, Independent Health is the highest ranked health insurance plan in the New York/New Jersey region by J.D. Power and Associates 2013 Member Health Plan StudySM. For the sixth consecutive year, Independent Health was named one of the best companies to work for in New York State. Go to www.independenthealth.com to view careers and available positions. We are an Equal Opportunity Employer. CONFIDENTIALITY NOTICE: This e-mail and its attachments (collectively referred to as "e-mail") may contain confidential information that is privileged and protected from disclosure by Federal and State confidentiality laws, rules or regulations. This e-mail is intended for the designated addressee only. If you are not the designated addressee, you are notified that any disclosure, copying or distribution of this e-mail may be unlawful and may subject you to legal consequences. If you have received this e-mail in error, please contact me immediately by telephone at (716) 631 - 3001 and delete the e-mail from your computer immediately. Thank you for your attention.

3 4

Deswizzling and Reallocate
by Martin 23 Oct '13

23 Oct '13

I've recently destroyed one aggregate and merged the spared disks it into another aggregate on the same Filer. I now need to snapmirror the volumes out of this aggregate into another on another Filer (musical volumes!). I didn't run a volume reallocation on the volumes after I added the disks from the recently destroyed aggregate as I'm going to be snapmirroring the data out of it. I'm not clear on whether I will need to run a volume level reallocate on the volumes once they are in the new aggregate? Will the deswizzle post snapmirror on the destination aggregate optimise the layout on the aggregate? I did a Google for "reallocate deswizzle" and looked on the NOW site but didn't find many hits and none were relevant. I can run a reallocate measure once the volumes are on the new Filer and aggregate but wonder how the volume reallocate and deswizzling are related. -- View this message in context: http://network-appliance-toasters.10978.n7.nabble.com/Deswizzling-and-Reall… Sent from the Network Appliance - Toasters mailing list archive at Nabble.com.

3 2

unexplained read operation
by Alon Zeltser 23 Oct '13

23 Oct '13

Hello toasters I encounter rather strange situation I have a system FAS2040 with 1 aggr of 23x450g raid group ontap version 8.1.3p2 Every write operation being generated on this aggregate automatically caused a read operation When I try to create 100% sequential write operation using tools such as sio_ntap or filersio Instead of the expected 150Mb throughput I get 100MB write and 50MB read On another FAS2040 system using the same commands I get 100% write and no reads As you can see filersio asyncio_active 0 -r 0 64k 0 15g 60 1 /vol/test/testfile CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s 93% 0 0 0 5 26 1 24242 118671 0 0 0s 99% 72% Fn 43% 0 0 5 0 0 24 0 90% 0 0 0 4 4 1 40959 102501 0 0 19 100% 89% Fn 43% 1 0 3 0 0 2 0 86% 0 0 0 3 3 1 23126 125022 0 0 0s 99% 69% : 59% 0 0 3 0 0 2 0 93% 0 0 0 7 1 1 28200 135740 0 0 0s 99% 97% Ff 48% 7 0 0 0 0 0 0 92% 0 0 0 6 15 1 48764 115324 0 0 0s 99% 91% Ff 52% 0 0 6 0 0 14 0 90% 0 0 0 6 16 1 36148 156492 0 0 0s 99% 83% Ff 56% 0 0 6 0 0 14 0 95% 0 0 0 0 4 4 32784 114555 0 0 0s 99% 77% Fs 48% 0 0 0 0 0 0 0 92% 0 0 0 18 90 12 34931 104059 0 0 0s 99% 88% Fn 42% 0 0 18 0 0 79 0 88% 0 0 0 8 3 1 20463 124800 0 0 0s 100% 83% Fn 45% 5 0 3 0 0 2 0 89% 0 0 0 2 2 1 34050 112166 0 0 0s 99% 87% Fn 44% 0 0 2 0 0 1 0 92% 0 0 0 3 7 1 33025 112816 0 0 0s 99% 88% Fn 46% 0 0 3 0 0 5 0 91% 0 0 0 3 3 1 30929 130833 0 0 0s 100% 73% : 50% 0 0 3 0 0 2 0 92% 0 0 0 7 24 2 36484 134260 0 0 0s 100% 98% Ff 50% 0 0 7 0 0 22 0 86% 0 0 0 6 1 1 38608 125140 0 0 0s 99% 90% Ff 44% 5 0 1 0 0 1 0 93% 0 0 0 4 3 1 28748 104544 0 0 0s 100% 75% Ff 43% 0 0 4 0 0 2 0 87% 0 0 0 5 18 1 28128 158608 0 0 0s 100% 90% Fs 55% 0 0 5 0 0 17 0 93% 0 0 0 2 3 34 33115 104980 0 0 0s 99% 85% Fn 41% 0 0 2 0 0 1 32 84% 0 0 0 5 28 7 37185 109425 0 0 0s 100% 78% Fn 48% 0 0 5 0 0 21 0 91% 0 0 0 9 3 1 30224 118322 0 0 0s 99% 97% Fn 47% 7 0 2 0 0 1 0 89% 0 0 0 2 3 4 27649 118335 0 0 0s 99% 88% : 45% 0 0 2 0 0 1 0 CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 88% 0 0 0 14 57 39 43284 133480 0 0 0s 100% 98% Ff 50% 0 0 14 0 0 52 0 92% 0 0 0 2 6 16 47936 109432 0 0 0s 99% 86% Ff 46% 0 0 2 0 0 1 0 88% 0 0 0 23 86 3 45340 141014 0 0 0s 99% 73% Ff 55% 3 0 20 0 0 82 0 92% 0 0 0 10 5 4 21787 123980 0 0 0s 99% 80% Fn 44% 7 0 3 0 0 1 0 95% 0 0 0 2 4 3 34751 98158 0 0 0s 99% 82% Fn 44% 0 0 2 0 0 1 0 91% 0 0 0 1 3 1 35022 118549 0 0 0s 100% 60% Fn 48% 0 0 1 0 0 0 0 92% 0 0 0 87 3 1 25455 123066 0 0 0s 100% 80% Fn 47% 84 0 3 0 0 1 0 83% 0 0 0 8 32 2 36956 121303 0 0 0s 100% 88% : 46% 0 0 8 0 0 30 0 85% 0 0 0 5 2 1 62064 140088 0 0 0s 100% 92% Ff 59% 5 0 0 0 0 0 0 95% 0 0 0 3 3 1 39352 117364 0 0 0s 99% 71% Ff 48% 0 0 3 0 0 2 0 94% 0 0 0 3 3 1 42008 136943 0 0 0s 99% 94% Fs 53% 0 0 3 0 0 2 0 92% 0 0 0 0 2 7 26147 111286 0 0 0s 99% 58% Fn 44% 0 0 0 0 0 0 0 90% 0 0 0 3 3 1 29567 110665 0 0 20 99% 87% Fn 44% 0 0 3 0 0 1 0 90% 0 0 0 9 4 1 40031 111273 0 0 16 99% 86% Fn 43% 5 0 4 0 0 2 0 84% 0 0 0 0 1 1 36760 105709 0 0 0s 100% 84% Fn 43% 0 0 0 0 0 0 0 92% 0 0 0 3 3 1 31033 137891 0 0 0s 100% 99% : 53% 0 0 3 0 0 2 0 83% 0 0 0 4 6 1 32991 140148 0 0 0s 100% 89% F 51% 0 0 4 0 0 5 0 Any thoughts why this could happens ? Nothing else is running on this netapp except for this test Thanks Alon Z

5 12

Slow copy of a directory full of files via an NFS client across a WAN
by Arnold de Leon 21 Oct '13

21 Oct '13

I have an NFS client (CentOS 5) that is mounting a volume from filer across a WAN with about 20ms of latency (40 ms round trip). We have an application that makes a "backup" copy of a directory of data before modifying it. The directory can contain a lot tiny of files (for example 600 files, taking up less than 12MB). I'm using NFS V3 over TCP (also tried V3 over UDP). The NFS copy (using cp -a) is glacial. Looking at the WireShark packet captures shows the client and the server in a deadly lock step request/reply situation (typically GETATTR and SETATTR). As far as I can tell the client/server are not allowing any windowing/pipelining of requests. Is there a setting I am missing to enable more outstanding requests (on the client or the server)? All the searches I've done for tuning seem to be about fixing issues for large bulk transfers. Is the NFS protocol inherently limited in this way. I expected this kind of behavior in SMB 1 but not with NFS. Thanks. arnold

13 20

AW: Dedup killing performance (again)
by Alexander Griesser 18 Oct '13

18 Oct '13

Yah, just found it while googling I guess, must be bug id #657692 :-/ Anyhow, I'm now stuck with this: > sis status Path State Status Progress /vol/CIFS_SHARES Enabled Active Fingerprint is being recovered from aggregate (6% complete) It's doing that since quite some time now, although it now actually jumped from 0% to 6%, so it at least is still alive. This bug is first fixed in 8.1.2P4 as it seems, so I might need to upgrade ASAP; for now, I'll check if it at least finishes this process now, my volume is filling up quickly, but there's still some aggregate space left so I'll just increase the volume size before it goes full. I'll go above the magic 80% fill of an aggregate then, but better that then having the volume go full. I'll see how it goes and if it troubles me too much, I hope there is a way to abort sis in that state. Alexander Griesser System-Administrator ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-320 Telefax: +43-5-0556-500 E-Mail: ag(a)anexia.at<mailto:ag@anexia.at> Web: http://www.anexia.at<http://www.anexia.at/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 Von: Sebastian Goetze [mailto:spgoetze@gmail.com] Gesendet: Samstag, 19. Oktober 2013 00:16 An: Alexander Griesser Betreff: Re: Dedup killing performance (again) Isn't that the Dedupe bug again, that was fixed in 8.1.3? (since like 8.1.2P3 or P4)... There was a thread about it some time ago... Anyway, I'm off to bed, just wanted to point you in the right direction. BFN On 10/18/2013 11:52 PM, Alexander Griesser wrote: Hi there, I am just trying to dedupe some volumes on one of our filers and the first volume (2.4TB data, CIFS share) deduplicates since about 4 hours now. Sysstat -x 1 shows: CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 58% 115 0 0 116 2239 3635 281593 32 0 0 0s 98% 0% - 100% 0 0 1 0 0 0 0 63% 162 0 0 169 1682 4149 310829 0 0 0 0s 99% 0% - 100% 7 0 0 0 0 0 0 50% 381 0 0 384 11793 11325 182100 0 0 0 0s 96% 0% - 100% 0 0 3 0 0 0 0 55% 412 0 0 416 6487 6040 223036 24 0 0 0s 97% 0% - 100% 0 0 4 0 0 37 0 51% 19 0 0 19 185 9 239388 0 0 0 2 98% 0% - 100% 0 0 0 0 0 0 0 51% 77 0 0 80 348 906 273880 8 0 0 2 99% 0% - 100% 0 0 3 0 0 0 0 60% 48 0 0 49 174 410 196179 271 0 0 2 98% 35% Tn 100% 0 0 1 0 0 0 0 45% 102 0 0 110 337 337 144387 21222 0 0 2 96% 100% :f 100% 8 0 0 0 0 0 0 71% 143 0 0 149 258 2814 109728 24268 0 0 2 85% 100% :f 86% 0 0 6 0 0 25 0 53% 64 0 0 65 203 1368 236620 14152 0 0 2 98% 100% :f 100% 0 0 1 0 0 0 0 49% 96 0 0 96 319 1317 185552 19748 0 0 2 96% 100% :f 100% 0 0 0 0 0 0 0 35% 109 0 0 117 381 1962 62613 18010 0 0 2 88% 100% :f 100% 5 0 3 0 0 0 0 55% 38 0 0 39 159 1168 239408 1980 0 0 2 97% 100% :f 100% 0 0 1 0 0 0 0 57% 177 0 0 179 308 6692 136568 520 0 0 2 92% 82% : 99% 0 0 2 0 0 37 0 39% 256 0 0 271 403 12647 117284 0 0 0 2 93% 0% - 97% 12 0 3 0 0 0 0 40% 232 0 0 473 808 7506 162368 32 0 0 2 96% 0% - 100% 240 0 1 0 0 0 0 38% 86 0 0 86 246 3413 160092 0 0 0 2 96% 0% - 100% 0 0 0 0 0 0 0 47% 113 0 0 117 314 1738 199516 0 0 0 2 96% 0% - 100% 1 0 3 0 0 0 0 59% 116 0 0 120 476 2751 288048 24 0 0 0s 99% 0% - 100% 0 0 4 0 0 24 0 45% 230 0 0 230 2879 6547 185787 120 0 0 0s 98% 0% - 100% 0 0 0 0 0 0 0 > version NetApp Release 8.1.2P2 7-Mode: Wed Feb 20 19:58:27 PST 2013 I'm aware that deduplication _SHOULD_ be a background process hugging up as much resources as it can but when host i/o is needed, it will be preferred over deduplication. There are no other tasks running at the same time (snapvault, etc.), and the virtual machinse on this system are _VERY_ slow - f.ex. logging in via RDP on one of the virtual machines now took > 1 minute for the login window to appear. This is a FAS2240 with 24x600 SAS 10k and basically my question now is: Is there any way to tell a-sis to really run with lower priority or is this a bug in the OnTap version I'm running? I've used a-sis a lot in the past and never had performance issues like this when it was running... I'm thankful for any ideas how to track down this issue or improve performance while a-sis is running. Thanks, Alexander Griesser System-Administrator ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-320 Telefax: +43-5-0556-500 E-Mail: ag(a)anexia.at<mailto:ag@anexia.at> Web: http://www.anexia.at<http://www.anexia.at/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601 _______________________________________________ Toasters mailing list Toasters(a)teaparty.net<mailto:Toasters@teaparty.net> http://www.teaparty.net/mailman/listinfo/toasters

1 0

Dedup killing performance (again)
by Alexander Griesser 18 Oct '13

18 Oct '13

Hi there, I am just trying to dedupe some volumes on one of our filers and the first volume (2.4TB data, CIFS share) deduplicates since about 4 hours now. Sysstat -x 1 shows: CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out 58% 115 0 0 116 2239 3635 281593 32 0 0 0s 98% 0% - 100% 0 0 1 0 0 0 0 63% 162 0 0 169 1682 4149 310829 0 0 0 0s 99% 0% - 100% 7 0 0 0 0 0 0 50% 381 0 0 384 11793 11325 182100 0 0 0 0s 96% 0% - 100% 0 0 3 0 0 0 0 55% 412 0 0 416 6487 6040 223036 24 0 0 0s 97% 0% - 100% 0 0 4 0 0 37 0 51% 19 0 0 19 185 9 239388 0 0 0 2 98% 0% - 100% 0 0 0 0 0 0 0 51% 77 0 0 80 348 906 273880 8 0 0 2 99% 0% - 100% 0 0 3 0 0 0 0 60% 48 0 0 49 174 410 196179 271 0 0 2 98% 35% Tn 100% 0 0 1 0 0 0 0 45% 102 0 0 110 337 337 144387 21222 0 0 2 96% 100% :f 100% 8 0 0 0 0 0 0 71% 143 0 0 149 258 2814 109728 24268 0 0 2 85% 100% :f 86% 0 0 6 0 0 25 0 53% 64 0 0 65 203 1368 236620 14152 0 0 2 98% 100% :f 100% 0 0 1 0 0 0 0 49% 96 0 0 96 319 1317 185552 19748 0 0 2 96% 100% :f 100% 0 0 0 0 0 0 0 35% 109 0 0 117 381 1962 62613 18010 0 0 2 88% 100% :f 100% 5 0 3 0 0 0 0 55% 38 0 0 39 159 1168 239408 1980 0 0 2 97% 100% :f 100% 0 0 1 0 0 0 0 57% 177 0 0 179 308 6692 136568 520 0 0 2 92% 82% : 99% 0 0 2 0 0 37 0 39% 256 0 0 271 403 12647 117284 0 0 0 2 93% 0% - 97% 12 0 3 0 0 0 0 40% 232 0 0 473 808 7506 162368 32 0 0 2 96% 0% - 100% 240 0 1 0 0 0 0 38% 86 0 0 86 246 3413 160092 0 0 0 2 96% 0% - 100% 0 0 0 0 0 0 0 47% 113 0 0 117 314 1738 199516 0 0 0 2 96% 0% - 100% 1 0 3 0 0 0 0 59% 116 0 0 120 476 2751 288048 24 0 0 0s 99% 0% - 100% 0 0 4 0 0 24 0 45% 230 0 0 230 2879 6547 185787 120 0 0 0s 98% 0% - 100% 0 0 0 0 0 0 0 > version NetApp Release 8.1.2P2 7-Mode: Wed Feb 20 19:58:27 PST 2013 I'm aware that deduplication _SHOULD_ be a background process hugging up as much resources as it can but when host i/o is needed, it will be preferred over deduplication. There are no other tasks running at the same time (snapvault, etc.), and the virtual machinse on this system are _VERY_ slow - f.ex. logging in via RDP on one of the virtual machines now took > 1 minute for the login window to appear. This is a FAS2240 with 24x600 SAS 10k and basically my question now is: Is there any way to tell a-sis to really run with lower priority or is this a bug in the OnTap version I'm running? I've used a-sis a lot in the past and never had performance issues like this when it was running... I'm thankful for any ideas how to track down this issue or improve performance while a-sis is running. Thanks, Alexander Griesser System-Administrator ANEXIA Internetdienstleistungs GmbH Telefon: +43-5-0556-320 Telefax: +43-5-0556-500 E-Mail: ag(a)anexia.at<mailto:ag@anexia.at> Web: http://www.anexia.at<http://www.anexia.at/> Anschrift Hauptsitz Klagenfurt: Feldkirchnerstraße 140, 9020 Klagenfurt Geschäftsführer: Alexander Windbichler Firmenbuch: FN 289918a | Gerichtsstand: Klagenfurt | UID-Nummer: AT U63216601

1 0

Aftermarket Hardware Support?
by Randy Rue 17 Oct '13

17 Oct '13

Hello All, This is not a NetApp question but a general storage question. We have an older system with two years left in it maximum. Its support contract is coming up. We're considering going with Sherlock Services for hardware support. There's potentially a savings over the next two years that significant enough to warrant the risk. Anyone gone this route? Anyone familiar with this provider? Hope to hear from you, Randy

2 2

Cluster mode root volume recovery
by Mike Thompson 17 Oct '13

17 Oct '13

Hey all, I've got a 8.0.2 c-mode cluster that recently had a single node joined to it, and a few empty aggregates created on it. We had a extended power outtage that required a lot of gear in the data center to get shut down, and since this node in the cluster didn't have any live data or VIFs on it, it got shut down. A few days later, and we are now powering it up, but get this upon login: "The contents of the root volume may have changed and the local management databases may be out of sync with the replicated databases due to corruption of NVLOG data during takeover. This node is not fully operational. Contact support personnel for the root volume recovery procedures." The node comes up fine, can see all it's aggregates, and the other nodes in the cluster can see it via the cluster network, but the node is indeed not fully functional and part of the cluster again. It's aggregates and other info are not visible from the other nodes in the cluster. Did a wafl_check of the root aggr and vol0 and that came back clean. I seem to recall having been through this before in the past, but can't find anything in my notes. This particular cluster is not under support, due to some genius decisions by management, so I'm on my own with this. There are a few empty aggregates on this node, no volumes other than the root vol. Maybe I can force unjoin it from the cluster and rebuild it? Would rather not try to do that. If there is a way to sync up the dbs on the rootvol so it will come back into the cluster, that would be ideal. Any ideas?

6 12

Script to analizyse over subscription rate
by Iluhes 14 Oct '13

14 Oct '13

1 0

Options reduce support renewal
by Fletcher Cocquyt 10 Oct '13

10 Oct '13

We have two 3270 clusters (active/standby) coming off 3 yr warranty support (10 trays of DS4243 + 5 trays of MK14 each) Our software ELA is also expiring - resulting in the Netapp support renewal quote being over 2 times last year's total (and budget) - they want ~$160K I'm looking for advice on support options. I've already got a quote from Zeriva for the HW support (we just need the parts, we can handle the replacement ourselves) What other creative ideas might help lower the support renewal? thanks, Fletcher

2 1

Re: Options reduce support renewal
by Mark Waterhouse 10 Oct '13

10 Oct '13

This probably isnt what you want to hear....but we have generally found that replacing the hardware works out cheaper than renewing support. This would give you a great excuse to move to Cluster Mode as well From: Fletcher Cocquyt Sent: Thursday, October 10, 2013 6:19 AM To: mailto:toasters@teaparty.net Subject: Options reduce support renewal We have two 3270 clusters (active/standby) coming off 3 yr warranty support (10 trays of DS4243 + 5 trays of MK14 each) Our software ELA is also expiring - resulting in the Netapp support renewal quote being over 2 times last year's total (and budget) - they want ~$160K I'm looking for advice on support options. I've already got a quote from Zeriva for the HW support (we just need the parts, we can handle the replacement ourselves) What other creative ideas might help lower the support renewal? thanks, Fletcher -- To report this message as spam, click here This message has been scanned for viruses and dangerous content by DFK Systems Limited, and is believed to be clean. ------------------------------------------------------------------------------ _______________________________________________ Toasters mailing list Toasters(a)teaparty.net http://www.teaparty.net/mailman/listinfo/toasters -- This message has been scanned for viruses and dangerous content by DFK Systems Limited, and is believed to be clean. To report this message as spam, please click on this link - https://secure.dfk-systems.com/quarantine/spam.php?srv=m2&id=r9ACE2Q4000933

3 2

Alternatives to DFM/Oncommand Core
by Martin 30 Sep '13

30 Sep '13

We've recently been told that vFiler DR is not currently support with Oncommand Core. As cDOT doesn't have the concept of vFilers and from what I have read 5.2 for 7-mode is the last release of Oncommand Core I doubt this functionality will ever be added. We currently rely on vFiler DR to allow fast DR recovery but can't use Oncommand Core due to this limitation. I'm looking at alternatives to Oncommand Core. From past experience there is: Solarwinds Logicmonitor Various Nagios based solutions Does anyone have any other suggestions for monitoring Netapp Filers? Thanks Martin -- View this message in context: http://network-appliance-toasters.10978.n7.nabble.com/Alternatives-to-DFM-O… Sent from the Network Appliance - Toasters mailing list archive at Nabble.com.

1 0

ndmpd password error
by Eric Peng 28 Sep '13

28 Sep '13

Has anyone encountered this error before? filer> ndmpd password cvbackup Cannot generate NDMP password. filer> Fri Sep 27 17:12:40 PDT [filer:useradmin.unauthorized.user:warning]: User 'cvbackup' denied access - missing required capability: 'login-ndmp' This used to work, but we recently upgraded ONTAP to 8.1.2 P4 and am wondering if the upgrade may have broken this feature? Nothing has changed on the role, group, or user account "cvbackup" on the filer. I have verified that 'login-ndmp' capability exists on the role that is associated with the group, which is associated with the user. As a test I added the 'cvbackup' user into the default "Backup Operators" group which has the 'login-ndmp' role capability, but got the same results. Even tried running "ndmpd password administrator" and still got the same error message. Any thoughts? Thanks, Eric Peng | Systems Administrator Esri | 380 New York St. | Redlands, CA 92373 | USA T 909 793 2853 x3567 | M 909 557 0259 epeng(a)esri.com<mailto:epeng@esri.com> | esri.com<http://esri.com/>

3 2

vol size not having any effect on volume
by Phil Gardner 27 Sep '13

27 Sep '13

I recently attempted to expand a volume on one of our 2240's, but it seems to not have any effect on the free space. From the random reading I've done, it seems that having volumes and aggregates pushing >90% space utilization is not good in any case. Unfortunately the aggregate that this volume lives on is currently at 96% used (150G free), and the volume that I'm trying grow is at 95% used (124GB free). I attempted to grow the volume by 100G with the 'vol size +100g $volume_name' command. It took about 30s to come back to the prompt, but it did say that the volume has been resized to 100G larger than what df was showing. However, both df and df -A are not currently showing any change in size. Am I out of luck here? Is there any way to see a "queue" of things the controller is waiting to run or attempting to run? Feel free to school me on the badness of pushing anything close to 100% usage. This controller is sitting betweeen 70-95% CPU util as well. FWIW, this specific controller is used for VM storage, and I probably have some misalignment going on. -- _____________________ Phil Gardner PGP Key ID 0xFECC890C OTR Fingerprint 6707E9B8 BD6062D3 5010FE8B 36D614E3 D2F80538

4 9

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters