Hi
I have a FAS3240 running 8.0.5 with 256g flashcache
The filer has single aggregate with 92 X 15k disks is using for vmware (nfs over 10g)
I have very good performance and latency no complains here
The filer is running with average of 13,000 iops and I'm considering adding another 7000 iops to this controller
Trying to decide if it's a good idea or not (planning to add 2 new disk shelves)
Disk utilization is low 40% average
Analyzing perfstat I see I have average CPU utilization of 80-85% (see logs below)
First question do you think 80% disk utilization without any other process (no dedupe, no compression , no scrub , no snapmirror, no backups) only user activity, is 80% CPU is normal for this platform with this load ?
SPM say the controller should be around 40% CPU with this load
Another question I can't figure out
Looking at FlashCache statistics I see excellent hit rate over 80% and 24,500 reads replaced per second
How can it be that over 24,000 iops replaced per second when sysstat show much less total iops
Is the ext_cache_obj show only the maximum or average for the perfstat duration ?
ext_cache_obj:ec0:usage:77%ext_cache_obj:ec0:accesses_total:424571082367ext_cache_obj:ec0:accesses:45930/sext_cache_obj:ec0:disk_reads_replaced:24502/sext_cache_obj:ec0:hit:39725/sext_cache_obj:ec0:hit_percent:86% The last question do you think I should add the 7000 iops to this controller or get a new system ?
Thanks
------- ext_cache_obj ----------
TIME: Tue Nov 05 09:54:47 GMT 2013TIME_DELTA: 60:34 (3634s)ext_cache_obj:ec0:type:IOMEM-FLASHext_cache_obj:ec0:uptime:1302310171ext_cache_obj:ec0:blocks:67108864ext_cache_obj:ec0:size:256ext_cache_obj:ec0:media:Flashext_cache_obj:ec0:associativity:8ext_cache_obj:ec0:sets:16777216ext_cache_obj:ec0:usage:77%ext_cache_obj:ec0:accesses_total:424571082367ext_cache_obj:ec0:accesses:45930/sext_cache_obj:ec0:disk_reads_replaced:24502/sext_cache_obj:ec0:hit:39725/sext_cache_obj:ec0:hit_flushq:0/sext_cache_obj:ec0:hit_once:0/sext_cache_obj:ec0:hit_age:0/sext_cache_obj:ec0:hit_normal_lev0:34737/sext_cache_obj:ec0:hit_metadata_file:3833/sext_cache_obj:ec0:hit_directory:0/sext_cache_obj:ec0:hit_redirect:0/sext_cache_obj:ec0:hit_indirect:1238/sext_cache_obj:ec0:hit_type.l0.old/new inomap file:11ext_cache_obj:ec0:hit_type.l0.normal file:124863682ext_cache_obj:ec0:hit_type.l0.directory:1190ext_cache_obj:ec0:hit_type.l0.active bitmap:7498789ext_cache_obj:ec0:hit_type.l0.snap summary:6262825ext_cache_obj:ec0:hit_type.l0.block types:21600ext_cache_obj:ec0:hit_type.l0.spacemap:211740ext_cache_obj:ec0:hit_type.l0.metafile:258ext_cache_obj:ec0:hit_type.l0.private inode file:6ext_cache_obj:ec0:hit_type.l0.public inode file:7159ext_cache_obj:ec0:hit_type.l0.block refcount:853578ext_cache_obj:ec0:hit_type.l0.private inode map:6ext_cache_obj:ec0:hit_type.l1.normal file:2477006ext_cache_obj:ec0:hit_type.l1.directory:11ext_cache_obj:ec0:hit_type.l1.special file:1ext_cache_obj:ec0:hit_type.l1.active bitmap:1617ext_cache_obj:ec0:hit_type.l1.snap summary:36ext_cache_obj:ec0:hit_type.l1.block types:24413ext_cache_obj:ec0:hit_type.l1.vvol container:203275ext_cache_obj:ec0:hit_type.l1.private inode file:2ext_cache_obj:ec0:hit_type.l1.public inode file:47ext_cache_obj:ec0:hit_type.l1.block refcount:278778ext_cache_obj:ec0:hit_type.l1.delete log:1170345ext_cache_obj:ec0:hit_type.l1.purge file:359ext_cache_obj:ec0:hit_type.l2.normal file:290697ext_cache_obj:ec0:hit_type.l2.active bitmap:6ext_cache_obj:ec0:hit_type.l2.block types:183ext_cache_obj:ec0:hit_type.l2.vvol container:44698ext_cache_obj:ec0:hit_type.l2.delete log:2577ext_cache_obj:ec0:hit_type.l3.normal file:2567ext_cache_obj:ec0:hit_type.l3.vvol container:711ext_cache_obj:ec0:hit_type.l3.delete log:40ext_cache_obj:ec0:hit_partial:101/sext_cache_obj:ec0:total_metadata_hits:5072/sext_cache_obj:ec0:total_metadata_misses:784/sext_cache_obj:ec0:miss:6204/sext_cache_obj:ec0:miss_flushq:3851/sext_cache_obj:ec0:miss_once:2808/sext_cache_obj:ec0:miss_age:0/sext_cache_obj:ec0:miss_normal_lev0:5420/sext_cache_obj:ec0:miss_metadata_file:249/sext_cache_obj:ec0:miss_directory:0/sext_cache_obj:ec0:miss_indirect:535/sext_cache_obj:ec0:miss_ssd:0/sext_cache_obj:ec0:miss_type.l0.old/new inomap file:1ext_cache_obj:ec0:miss_type.l0.normal file:18874217ext_cache_obj:ec0:miss_type.l0.directory:244ext_cache_obj:ec0:miss_type.l0.active bitmap:1415914ext_cache_obj:ec0:miss_type.l0.snap summary:79062ext_cache_obj:ec0:miss_type.l0.block types:34523ext_cache_obj:ec0:miss_type.l0.spacemap:8153ext_cache_obj:ec0:miss_type.l0.metafile:1ext_cache_obj:ec0:miss_type.l0.private inode file:1ext_cache_obj:ec0:miss_type.l0.public inode file:5431ext_cache_obj:ec0:miss_type.l0.block refcount:165473ext_cache_obj:ec0:miss_type.l0.private inode map:1ext_cache_obj:ec0:miss_type.l1.normal file:325314ext_cache_obj:ec0:miss_type.l1.special file:12ext_cache_obj:ec0:miss_type.l1.active bitmap:1432ext_cache_obj:ec0:miss_type.l1.block types:928ext_cache_obj:ec0:miss_type.l1.vvol container:1518334ext_cache_obj:ec0:miss_type.l1.block refcount:1460ext_cache_obj:ec0:miss_type.l1.delete log:76923ext_cache_obj:ec0:miss_type.l1.purge file:19ext_cache_obj:ec0:miss_type.l2.normal file:13912ext_cache_obj:ec0:miss_type.l2.special file:1ext_cache_obj:ec0:miss_type.l2.active bitmap:3ext_cache_obj:ec0:miss_type.l2.block types:11ext_cache_obj:ec0:miss_type.l2.vvol container:3838ext_cache_obj:ec0:miss_type.l2.delete log:102ext_cache_obj:ec0:miss_type.l3.normal file:154ext_cache_obj:ec0:miss_type.l3.vvol container:27ext_cache_obj:ec0:miss_type.l3.delete log:2ext_cache_obj:ec0:lookup_reject:1506/sext_cache_obj:ec0:lookup_reject_normal_l0:0/sext_cache_obj:ec0:lookup_reject_io:0/sext_cache_obj:ec0:lookup_reject_lock:0/sext_cache_obj:ec0:lookup_generation_mismatch:3007ext_cache_obj:ec0:lookup_checksum_mismatch:2ext_cache_obj:ec0:lookup_context_mismatch:0ext_cache_obj:ec0:lookup_disk_ios_issued:3009ext_cache_obj:ec0:lookup_chains:28388/sext_cache_obj:ec0:lookup_chain_cnt:50724/sext_cache_obj:ec0:lookup_lwr_mismatch:0/sext_cache_obj:ec0:hit_percent:86%ext_cache_obj:ec0:inserts:3227/sext_cache_obj:ec0:inserts_mbuf:20/sext_cache_obj:ec0:inserts_mbuf_zerocopy:0/sext_cache_obj:ec0:inserts_flushq:0/sext_cache_obj:ec0:inserts_once:0/sext_cache_obj:ec0:inserts_age:20/sext_cache_obj:ec0:inserts_normal_lev0:1472/sext_cache_obj:ec0:inserts_metadata_file:812/sext_cache_obj:ec0:inserts_directory:0/sext_cache_obj:ec0:inserts_indirect:955/sext_cache_obj:ec0:insert_rejects_misc:0/sext_cache_obj:ec0:insert_rejects_compressed:0/sext_cache_obj:ec0:insert_rejects_ssd:0/sext_cache_obj:ec0:insert_rejects_present:36702/sext_cache_obj:ec0:insert_rejects_flushq:16717/sext_cache_obj:ec0:insert_rejects_normal_lev0:0/sext_cache_obj:ec0:insert_rejects_throttle:144/sext_cache_obj:ec0:insert_rejects_throttle_io:0/sext_cache_obj:ec0:insert_rejects_throttle_refill:144/sext_cache_obj:ec0:insert_rejects_throttle_mem:0/sext_cache_obj:ec0:insert_rejects_cache_reuse:0/sext_cache_obj:ec0:insert_rejects_vbn_invalid:3227/sext_cache_obj:ec0:insert_rejects_pvbn_aliased:0/sext_cache_obj:ec0:insert_rejects_mbuf_nobuf:0/sext_cache_obj:ec0:insert_rejects_mbuf_low:2189/sext_cache_obj:ec0:insert_rejects_throttle_rate:0/sext_cache_obj:ec0:insert_rejects_erase_stripe_state:0/sext_cache_obj:ec0:insert_rejects_erase_queue:0/sext_cache_obj:ec0:insert_rejects_mbuf_off:0/sext_cache_obj:ec0:insert_rejects_no_wip:987/sext_cache_obj:ec0:insert_rejects_vol_not_mounted:0/sext_cache_obj:ec0:insert_rejects_delete_log:0/sext_cache_obj:ec0:insert_rejects_not_valid:104/sext_cache_obj:ec0:insert_rejects_errored:0/sext_cache_obj:ec0:insert_rejects_no_context:0/sext_cache_obj:ec0:reuse_percent:1230%ext_cache_obj:ec0:evicts:36/sext_cache_obj:ec0:evicts_ref:16/sext_cache_obj:ec0:readio_solitary:17462/sext_cache_obj:ec0:readio_chains:24501/sext_cache_obj:ec0:readio_blocks:39369/sext_cache_obj:ec0:readio_in_flight:0ext_cache_obj:ec0:readio_max_in_flight:511ext_cache_obj:ec0:readio_general_errors:0ext_cache_obj:ec0:readio_transient_errors:0ext_cache_obj:ec0:readio_avg_chainlength:1.61ext_cache_obj:ec0:readio_avg_latency:1.70msext_cache_obj:ec0:readio_latency.0:20693251ext_cache_obj:ec0:readio_latency.1:39478840ext_cache_obj:ec0:readio_latency.2:11080740ext_cache_obj:ec0:readio_latency.3:7053418ext_cache_obj:ec0:readio_latency.4:3915379ext_cache_obj:ec0:readio_latency.5:2101373ext_cache_obj:ec0:readio_latency.6:1295253ext_cache_obj:ec0:readio_latency.7:877946ext_cache_obj:ec0:readio_latency.<16:2131721ext_cache_obj:ec0:readio_latency.<24:245311ext_cache_obj:ec0:readio_latency.<32:38177ext_cache_obj:ec0:readio_latency.<40:18450ext_cache_obj:ec0:readio_latency.<48:8463ext_cache_obj:ec0:readio_latency.<56:4540ext_cache_obj:ec0:readio_latency.<64:2248ext_cache_obj:ec0:readio_latency.<128:3371ext_cache_obj:ec0:readio_latency.<192:19ext_cache_obj:ec0:readio_chain_lengths.1:63394306ext_cache_obj:ec0:readio_chain_lengths.2:5874105ext_cache_obj:ec0:readio_chain_lengths.3:16774665ext_cache_obj:ec0:readio_chain_lengths.4:1088869ext_cache_obj:ec0:readio_chain_lengths.5:306521ext_cache_obj:ec0:readio_chain_lengths.6:871098ext_cache_obj:ec0:readio_chain_lengths.7:157207ext_cache_obj:ec0:readio_chain_lengths.<16:434548ext_cache_obj:ec0:readio_chain_lengths.<24:36558ext_cache_obj:ec0:readio_chain_lengths.<32:5244ext_cache_obj:ec0:readio_chain_lengths.<40:3359ext_cache_obj:ec0:readio_chain_lengths.<48:936ext_cache_obj:ec0:readio_chain_lengths.<56:632ext_cache_obj:ec0:readio_chain_lengths.<64:389ext_cache_obj:ec0:readio_chain_lengths.<128:53ext_cache_obj:ec0:writeio_solitary:0/sext_cache_obj:ec0:writeio_chains:50/sext_cache_obj:ec0:writeio_blocks:3227/sext_cache_obj:ec0:writeio_in_flight:0ext_cache_obj:ec0:writeio_max_in_flight:107ext_cache_obj:ec0:writeio_general_errors:0ext_cache_obj:ec0:writeio_transient_errors:0ext_cache_obj:ec0:writeio_avg_chainlength:64.00ext_cache_obj:ec0:writeio_avg_latency:4.62msext_cache_obj:ec0:writeio_latency.0:358ext_cache_obj:ec0:writeio_latency.1:23690ext_cache_obj:ec0:writeio_latency.2:41774ext_cache_obj:ec0:writeio_latency.3:27620ext_cache_obj:ec0:writeio_latency.4:17766ext_cache_obj:ec0:writeio_latency.5:14558ext_cache_obj:ec0:writeio_latency.6:12651ext_cache_obj:ec0:writeio_latency.7:12580ext_cache_obj:ec0:writeio_latency.<16:29129ext_cache_obj:ec0:writeio_latency.<24:2504ext_cache_obj:ec0:writeio_latency.<32:337ext_cache_obj:ec0:writeio_latency.<40:83ext_cache_obj:ec0:writeio_latency.<48:27ext_cache_obj:ec0:writeio_latency.<56:5ext_cache_obj:ec0:writeio_latency.<64:2ext_cache_obj:ec0:writeio_latency.<128:4ext_cache_obj:ec0:writeio_chain_lengths.<128:183088ext_cache_obj:ec0:blocks_ref0:41301539ext_cache_obj:ec0:blocks_ref1:18120071ext_cache_obj:ec0:blocks_ref2:2121072ext_cache_obj:ec0:blocks_ref3:1084969ext_cache_obj:ec0:blocks_ref4:599609ext_cache_obj:ec0:blocks_ref5:417174ext_cache_obj:ec0:blocks_ref6:286331ext_cache_obj:ec0:blocks_ref7:3178099ext_cache_obj:ec0:blocks_ref0_arrivals:27596243622ext_cache_obj:ec0:blocks_ref1_arrivals:27622050947ext_cache_obj:ec0:blocks_ref2_arrivals:10423310000ext_cache_obj:ec0:blocks_ref3_arrivals:6427750307ext_cache_obj:ec0:blocks_ref4_arrivals:4495536161ext_cache_obj:ec0:blocks_ref5_arrivals:3557830045ext_cache_obj:ec0:blocks_ref6_arrivals:2957867966ext_cache_obj:ec0:blocks_ref7_arrivals:117062905990ext_cache_obj:ec0:lru_ticks:3563396220ext_cache_obj:ec0:invalidates:3526/sext_cache_obj:ec0:erase_stripe_invalidates:6/sext_cache_obj:ec0:erase_stripe_blocks_invalidates:3226/sext_cache_obj:ec0:cache_utilization:77%ext_cache_obj:ec0:pcs_invalidates_flash:0/sext_cache_obj:ec0:pcs_wraps:0ext_cache_obj:ec0:pcs_wrap_time:0ext_cache_obj:ec0:pcs_confidence:100ext_cache_obj:ec0:pcs_total_ops_insert:0/sext_cache_obj:ec0:pcs_total_ops_lookup:0/sext_cache_obj:ec0:pcs_total_ops_invalidate:0/sext_cache_obj:ec0:pcs_total_ops_misc:0/sext_cache_obj:ec0:pcs_sampled_ops_insert:0/sext_cache_obj:ec0:pcs_sampled_ops_lookup:0/sext_cache_obj:ec0:pcs_sampled_ops_invalidate:0/sext_cache_obj:ec0:pcs_sampled_ops_misc:0/sext_cache_obj:ec0:pcs_sampling_rate:0---------- sysstat –x 1 -----------
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s
in out read write read write age hit time ty util in out in out
96% 16951 0 0 16951 22579 180997 108622 57263 0 0 4s 67% 100% :f 50% 0 0 0 0 0 0 0
89% 20267 0 0 20267 45941 229706 111661 93433 0 0 5s 51% 100% :f 44% 0 0 0 0 0 0 0
90% 17307 0 0 17307 56792 223833 118109 105672 0 0 3s 48% 100% :f 44% 0 0 0 0 0 0 0
90% 20658 0 0 20658 34301 267718 41860 22109 0 0 7s 50% 38% : 39% 0 0 0 0 0 0 0
77% 12840 0 0 12843 44520 142228 49804 0 0 0 8s 58% 0% - 47% 3 0 0 0 0 0 0
95% 16191 0 0 16191 33156 180821 149909 121486 0 0 5s 53% 58% Dn 46% 0 0 0 0 0 0 0
97% 15198 0 0 15198 23309 199482 100311 43518 0 0 4s 58% 100% :s 42% 0 0 0 0 0 0 0
91% 15363 0 0 15363 22212 216210 82631 62381 0 0 5s 62% 100% :f 46% 0 0 0 0 0 0 0
87% 17662 0 0 17662 36828 209514 118430 79394 0 0 6s 53% 100% :f 49% 0 0 0 0 0 0 0
80% 14307 0 0 14310 47874 166153 74677 54134 0 0 6s 55% 100% :f 49% 3 0 0 0 0 0 0
80% 13905 0 0 13905 42593 202792 49344 49444 0 0 5s 53% 57% : 48% 0 0 0 0 0 0 0
88% 16720 0 0 16720 59662 244810 41227 0 0 0 3s 48% 0% - 44% 0 0 0 0 0 0 0
93% 14717 0 0 14717 17933 223608 155425 102506 0 0 7s 55% 46% Dn 46% 0 0 0 0 0 0 0
96% 13937 0 0 13937 17215 164887 112227 64931 0 0 7s 54% 100% :s 43% 0 0 0 0 0 0 0
92% 13494 0 0 13497 28609 195722 71916 54271 0 0 5s 64% 100% :f 46% 3 0 0 0 0 0 0
93% 22023 0 0 22023 28545 254401 73108 96616 0 0 4s 48% 100% :f 41% 0 0 0 0 0 0 0
88% 23112 0 0 23112 24806 213939 64348 73510 0 0 5s 53% 100% :f 44% 0 0 0 0 0 0 0
84% 17240 0 0 17240 30511 203088 45549 22405 0 0 3s 54% 36% : 41% 0 0 0 0 0 0 0
90% 20676 0 0 20676 61721 269589 35822 24 0 0 4s 51% 0% - 36% 0 0 0 0 0 0 0
84% 19089 0 0 19100 42507 229811 34014 0 0 0 5s 54% 0% - 38% 11 0 0 0 0 0 0
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s
in out read write read write age hit time ty util in out in out
92% 17690 0 0 17690 32740 195063 106798 63482 0 0 5s 51% 33% Dn 40% 0 0 0 0 0 0 0
96% 13450 0 0 13450 18298 175936 97234 80224 0 0 4s 60% 100% :s 39% 0 0 0 0 0 0 0
94% 14813 0 0 14813 13928 204597 69566 61750 0 0 5s 57% 100% :f 39% 0 0 0 0 0 0 0
89% 18505 0 0 18509 23968 247285 104228 69749 0 0 6s 54% 100% :f 47% 4 0 0 0 0 0 0
91% 21963 0 0 21967 60710 223419 78861 66270 0 0 3s 54% 100% :f 42% 4 0 0 0 0 0 0
89% 19919 0 0 19919 33959 215890 79309 59648 0 0 5s 53% 100% :f 45% 0 0 0 0 0 0 0
90% 19707 0 0 19707 27499 247462 35972 33722 0 0 6s 50% 59% : 35% 0 0 0 0 0 0 0
86% 19456 0 0 19456 43880 233384 37526 0 0 0 4s 55% 0% - 38% 0 0 0 0 0 0 0
94% 20130 0 0 20130 63247 229293 78845 48829 0 0 2s 54% 31% Dn 35% 0 0 0 0 0 0 0
96% 10225 0 0 10228 45026 101768 193293 199449 0 0 3s 68% 100% :s 47% 3 0 0 0 0 0 0
95% 13359 0 0 13359 39462 191730 99143 52233 0 0 4s 61% 100% :f 46% 0 0 0 0 0 0 0
87% 15657 0 0 15657 21113 257858 95879 61368 0 0 5s 53% 100% :f 41% 0 0 0 0 0 0 0
89% 19590 0 0 19590 51499 252323 57820 65116 0 0 6s 51% 100% :v 39% 0 0 0 0 0 0 0
77% 14530 0 0 14530 31816 178572 28376 32 0 0 6s 53% 7% : 39% 0 0 0 0 0 0 0
95% 14670 0 0 14673 36120 218937 136512 118201 0 0 5s 55% 54% Dn 45% 3 0 0 0 0 0 0
97% 15592 0 0 15592 24149 170596 136341 76393 0 0 6s 56% 100% :s 41% 0 0 0 0 0 0 0
95% 15562 0 0 15562 46759 158751 158750 58356 0 0 7s 72% 100% :f 59% 0 0 0 0 0 0 0
84% 15471 0 0 15471 15530 229961 145671 73636 0 0 2s 55% 100% :f 47% 0 0 0 0 0 0 0
85% 17753 0 0 17753 53878 214190 53348 46938 0 0 2s 50% 76% : 43% 0 0 0 0 0 0 0
84% 15532 0 0 15535 33703 231932 52108 0 0 0 10s 54% 0% - 48% 3 0 0 0 0 0 0
Summary Statistics ( 3595 samples 0.1 secs/sample) CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in outMin 31% 3582 0 0 3582 9751 7799 8404 0 0 0 0s 42% 0% * 12% 0 0 0 0 0 0 0Avg 87% 13331 0 0 13340 64420 184870 114575 102713 0 0 5s 56% 73% * 43% 8 0 0 0 0 0 0Max 99% 25894 0 0 25894 215371 661784 322220 330317 0 0 21s 94% 100% * 90% 547 0 0 0 0 0 0--------- Sysstat –M ---------
ANY1+ ANY2+ ANY3+ ANY4+ AVG CPU0 CPU1 CPU2 CPU3 Network Protocol Cluster Storage Raid Target Kahuna WAFL_Ex(Kahu) WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CP 98% 88% 74% 56% 79% 75% 78% 79% 86% 46% 0% 0% 15% 28% 0% 40% 109%( 52%) 10% 0% 0% 65% 4% 1% 13358 81% 98% 88% 73% 55% 79% 75% 77% 80% 86% 46% 0% 0% 13% 27% 0% 41% 108%( 52%) 11% 0% 0% 65% 4% 2% 13152 78% 95% 81% 64% 46% 72% 67% 68% 73% 80% 43% 0% 0% 11% 21% 0% 38% 102%( 50%) 7% 0% 0% 61% 5% 1% 12848 73% 92% 79% 64% 47% 71% 68% 69% 73% 75% 44% 0% 0% 11% 23% 0% 32% 104%( 51%) 8% 0% 0% 57% 5% 1% 13178 64% 90% 78% 64% 48% 71% 68% 69% 72% 73% 44% 0% 0% 11% 24% 0% 30% 101%( 50%) 8% 0% 0% 58% 4% 1% 12550 70% 93% 80% 66% 49% 73% 70% 71% 75% 75% 47% 0% 0% 12% 24% 0% 31% 106%( 53%) 8% 0% 0% 58% 5% 2% 13090 64% 87% 71% 55% 40% 64% 60% 62% 67% 68% 40% 0% 0% 9% 18% 0% 27% 97%( 49%) 6% 0% 0% 51% 5% 1% 12890 67% 91% 77% 61% 45% 69% 65% 67% 71% 74% 43% 0% 0% 11% 22% 0% 32% 100%( 50%) 8% 0% 0% 55% 4% 1% 12697 73% 89% 73% 57% 42% 66% 62% 63% 69% 69% 41% 0% 0% 10% 20% 0% 28% 99%( 50%) 7% 0% 0% 52% 5% 1% 13111 66% 90% 76% 60% 44% 68% 66% 66% 69% 71% 43% 0% 0% 11% 22% 0% 29% 99%( 50%) 8% 0% 0% 55% 4% 2% 12771 72% 87% 71% 55% 40% 64% 60% 62% 66% 67% 41% 0% 0% 10% 18% 0% 27% 96%( 49%) 6% 0% 0% 51% 5% 1% 12675 66% 86% 71% 56% 41% 64% 61% 62% 67% 68% 41% 0% 0% 10% 19% 0% 28% 97%( 48%) 7% 0% 0% 52% 5% 1% 12678 62% 92% 77% 62% 45% 70% 66% 68% 72% 74% 44% 0% 0% 11% 22% 0% 31% 103%( 52%) 8% 0% 0% 55% 5% 1% 13402 75% 94% 81% 67% 49% 74% 69% 72% 76% 77% 45% 0% 0% 12% 26% 0% 33% 104%( 52%) 10% 0% 0% 58% 4% 2% 13289 76% 89% 75% 60% 44% 67% 64% 65% 71% 70% 43% 0% 0% 10% 21% 0% 29% 99%( 50%) 8% 0% 0% 54% 5% 1% 12985 66% 85% 68% 52% 37% 62% 58% 60% 64% 64% 40% 0% 0% 9% 16% 0% 24% 96%( 49%) 6% 0% 0% 49% 5% 1% 12931 64% 90% 75% 61% 45% 69% 64% 66% 72% 72% 42% 0% 0% 10% 21% 0% 30% 103%( 51%) 8% 0% 0% 55% 5% 1% 13163 66% 90% 75% 58% 42% 67% 64% 65% 70% 70% 44% 0% 0% 10% 19% 0% 27% 104%( 52%) 6% 0% 0% 53% 5% 2% 13929 71% 99% 94% 84% 68% 87% 83% 86% 90% 87% 51% 0% 0% 15% 37% 0% 33% 123%( 61%) 14% 0% 0% 69% 4% 1% 14002 81%--Summary Statistics ( 58 samples 6.0 secs/sample)ANY1+ ANY2+ ANY3+ ANY4+ AVG CPU0 CPU1 CPU2 CPU3 Network Protocol Cluster Storage Raid Target Kahuna WAFL_Ex(Kahu) WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CPMin 85% 68% 52% 37% 62% 58% 60% 64% 64% 40% 0% 0% 9% 16% 0% 24% 95%( 46%) 6% 0% 0% 49% 3% 1% 10226 58%Avg 95% 84% 70% 54% 76% 73% 75% 78% 80% 46% 0% 0% 13% 28% 0% 35% 106%( 51%) 9% 0% 0% 61% 4% 1% 13234 73%Max 100% 96% 87% 70% 89% 84% 93% 90% 91% 56% 0% 0% 26% 49% 0% 49% 123%( 61%) 18% 0% 0% 72% 5% 2% 16583 95%---------- STATIT -----------
NetApp Release 8.0.5 7-Mode: Sun Jan 20 21:53:54 PST 2013 <1O> Start time: Tue Nov 5 11:25:18 IST 2013 CPU Statistics 60.182572 time (seconds) 100 % 173.781160 system time 289 % 2.716133 rupt time 5 % (1632432 rupts x 2 usec/rupt) 171.065027 non-rupt system time 284 % 66.949128 idle time 111 % 41.250883 time in CP 69 % 100 % 1.790287 rupt time in CP 4 % (1107164 rupts x 2 usec/rupt) Multiprocessor Statistics (per second) cpu0 cpu1 cpu2 cpu3 totalsk switches 58254.32 59427.55 57820.04 45838.67 221340.59hard switches 29026.92 30923.44 35440.13 16979.75 112370.24domain switches 1174.79 2076.38 238.46 316.05 3805.69CP rupts 14284.20 1370.85 1370.83 1370.88 18396.75nonCP rupts 6840.54 629.14 629.14 629.10 8727.91IPI rupts 0.00 0.00 0.00 0.00 0.00grab kahuna 0.00 0.98 0.91 0.00 1.89grab w_xcleaner 0.00 0.18 0.81 268.87 269.86 grab kahuna usec 0.00 488.56 910.73 0.00 1399.29grab w_xcleaner usec 0.00 123.31 347.31 156723.71 157194.33 CP rupt usec 26808.77 651.73 1779.62 507.44 29747.60nonCP rupt usec 14115.33 288.34 768.00 212.25 15383.96idle 318889.16 304923.34 259322.25 229299.07 1112433.85kahuna 0.47 0.00 0.00 332371.27 332371.75storage 84402.34 12364.19 16805.60 0.00 113572.15exempt 199610.16 176903.95 212622.70 13.92 589150.76raid 815.39 200269.12 22897.81 0.00 223982.33target 15.04 17.15 16.67 0.00 48.88netcache 0.00 0.00 0.20 0.00 0.20netcache2 0.00 0.00 0.00 0.00 0.00cifs 3.66 3.72 2.77 0.00 10.19wafl_exempt 254690.06 231801.77 136534.46 437596.00 1060622.33wafl_xcleaner 33024.31 19066.62 23468.94 0.00 75559.88sm_exempt 19.36 22.46 21.42 0.00 63.27cluster 0.00 0.00 0.00 0.00 0.00protocol 26.95 25.87 26.54 0.00 79.38nwk_exclusive 975.88 511.33 1150.90 0.00 2638.12nwk_exempt 261.45 359.64 366.10 0.00 987.23nwk_legacy 1585.44 1830.31 1686.75 0.00 5102.52nwk_ctx1 360.39 24181.75 303022.64 0.00 327564.81nwk_ctx2 59706.95 22852.83 16500.72 0.00 99060.52nwk_ctx3 0.00 0.00 0.00 0.00 0.00nwk_ctx4 0.00 0.00 0.00 0.00 0.00hostOS 4688.73 3925.69 3005.77 0.00 11620.19 56.181715 seconds with one or more CPUs active ( 93%) 48.289199 seconds with 2 or more CPUs active ( 80%) 38.846369 seconds with 3 or more CPUs active ( 65%) 7.892516 seconds with one CPU active ( 13%) 9.442829 seconds with 2 CPUs active ( 16%) 10.092179 seconds with 3 CPUs active ( 17%) 28.754190 seconds with all CPUs active ( 48%) Domain Utilization of Shared Domains (per second) 0.00 idle 516212.17 kahuna 0.00 storage 0.00 exempt 0.00 raid 0.00 target 0.00 netcache 0.00 netcache2 0.00 cifs 0.00 wafl_exempt 0.00 wafl_xcleaner 0.00 sm_exempt 0.00 cluster 0.00 protocol 392429.57 nwk_exclusive 0.00 nwk_exempt 0.00 nwk_legacy 0.00 nwk_ctx1 0.00 nwk_ctx2 0.00 nwk_ctx3 0.00 nwk_ctx4 0.00 hostOS Miscellaneous Statistics (per second) 112370.24 hard context switches 13517.83 NFS operations 0.00 CIFS operations 0.00 HTTP operations 44559.59 network KB received 184368.21 network KB transmitted 94098.07 disk KB read 78261.13 disk KB written 40535.46 NVRAM KB written 0.00 nolog KB written 45831.13 WAFL bufs given to clients 0.80 checksum cache hits ( 0%) 45787.34 no checksum - partial buffer 0.00 FCP operations 0.00 iSCSI operations WAFL Statistics (per second) 81.17 name cache hits ( 77%) 24.81 name cache misses ( 23%) 432838.23 buf hash hits ( 87%) 62737.71 buf hash misses ( 13%) 29606.73 inode cache hits ( 100%) 0.00 inode cache misses ( 0%) 47745.08 buf cache hits ( 55%) 39604.57 buf cache misses ( 45%) 2150.64 blocks read 5882.00 blocks read-ahead 1252.07 chains read-ahead 3674.07 dummy reads 4269.61 blocks speculative read-ahead 15858.58 blocks written 30.57 stripes written 0.00 blocks over-written 0.00 wafl_timer generated CP 0.00 snapshot generated CP 0.00 wafl_avail_bufs generated CP 0.02 dirty_blk_cnt generated CP 0.02 full NV-log generated CP 0.00 back-to-back CP 0.00 flush generated CP 0.00 sync generated CP 0.00 deferred back-to-back CP 0.00 container-indirect-pin CP 0.03 low mbufs generated CP 0.13 low datavecs generated CP 59836.91 non-restart messages 516.21 IOWAIT suspends142731372.60 next nvlog nearly full msecs 0.00 dirty buffer susp msecs 0.00 nvlog full susp msecs 933124 buffers RAID Statistics (per second) 3433.15 xors 0.00 long dispatches [0] 0.00 long consumed [0] 0.00 long consumed hipri [0] 0.00 long low priority [0] 0.00 long high priority [0] 0.00 long monitor tics [0] 0.00 long monitor clears [0] 0.00 long dispatches [1] 0.00 long consumed [1] 0.00 long consumed hipri [1] 0.00 long low priority [1] 0.00 long high priority [1] 0.00 long monitor tics [1] 0.00 long monitor clears [1] 18 max batch 6.95 blocked mode xor 1220.85 timed mode xor 2.18 fast adjustments 2.63 slow adjustments 0 avg batch start 0 avg stripe/msec 3737.36 checksum dispatches 23508.15 checksum consumed 31.42 tetrises written 0.00 master tetrises 0.00 slave tetrises 1729.77 stripes written 1713.19 partial stripes 16.58 full stripes 16095.72 blocks written 11922.54 blocks read 126.73 1 blocks per stripe size 21 100.83 2 blocks per stripe size 21 94.76 3 blocks per stripe size 21 96.27 4 blocks per stripe size 21 96.14 5 blocks per stripe size 21 96.47 6 blocks per stripe size 21 100.00 7 blocks per stripe size 21 102.87 8 blocks per stripe size 21 100.89 9 blocks per stripe size 21 98.15 10 blocks per stripe size 21 95.36 11 blocks per stripe size 21 91.70 12 blocks per stripe size 21 86.60 13 blocks per stripe size 21 84.97 14 blocks per stripe size 21 78.96 15 blocks per stripe size 21 71.90 16 blocks per stripe size 21 60.37 17 blocks per stripe size 21 54.10 18 blocks per stripe size 21 44.07 19 blocks per stripe size 21 32.04 20 blocks per stripe size 21 16.58 21 blocks per stripe size 21 Network Interface Statistics (per second)iface side bytes packets multicasts errors collisions pkt dropse0a recv 3683.86 50.26 0.00 0.00 0.00 xmit 5048.36 19.94 0.00 0.00 0.00e0b recv 4408.92 59.15 0.00 0.00 0.00 xmit 20.14 0.10 0.00 0.00 0.00e2a recv 45620024.58 50126.21 0.00 0.00 0.00 xmit 188787476.51 29046.10 0.18 0.00 0.00e2b recv 134.66 1.88 0.00 0.00 0.00 xmit 6.98 0.17 0.17 0.00 0.00e6a recv 71.10 0.91 0.00 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00e6b recv 59.10 0.63 0.00 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00e6c recv 0.00 0.00 0.00 71365630.73 0.00 xmit 0.00 0.00 0.00 0.00 0.00e6d recv 0.00 0.00 0.00 71365630.72 0.00 xmit 0.00 0.00 0.00 0.00 0.00c0a recv 285.13 21.93 1.46 0.00 0.00 xmit 216.34 1.16 1.16 0.00 0.00c0b recv 285.13 1.46 1.46 0.00 0.00 xmit 216.34 1.16 1.16 0.00 0.00e0M recv 0.00 0.00 0.00 0.15 0.00 xmit 0.00 0.00 0.00 0.00 0.00e0P recv 80.09 0.93 0.08 0.00 0.00 xmit 66.50 0.85 0.00 0.00 0.00vh recv 0.00 0.00 0.00 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00vif0-manage recv 8092.77 109.42 89.15 0.00 0.00 xmit 5068.49 20.04 0.00 0.00 0.00vif1-10g recv 45620159.24 50128.08 3.74 0.70 0.00 xmit 188787483.49 29046.27 0.35 0.00 0.00vif-174-1g recv 130.20 1.55 1.55 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00 Disk Statistics (per second) ut% is the percent of time the disk was busy. xfers is the number of data-transfer commands issued per second. xfers = ureads + writes + cpreads + greads + gwrites chain is the average number of 4K blocks per command. usecs is the average disk round-trip time per 4K block. disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs/aggr0_sas_600g/plex0/rg0:0a.01.4 8 23.16 0.76 1.39 2859 9.29 46.62 120 13.11 19.58 181 0.00 .... . 0.00 .... .0a.01.5 9 23.76 0.80 1.38 3394 9.80 44.28 119 13.16 19.71 199 0.00 .... . 0.00 .... .0a.01.6 25 64.90 42.39 2.39 3282 6.23 27.14 297 16.28 9.66 441 0.00 .... . 0.00 .... .0a.02.13 25 67.20 44.22 2.14 3537 6.06 29.12 311 16.92 8.81 470 0.00 .... . 0.00 .... .0a.02.2 25 63.74 40.44 2.13 3620 6.25 29.57 297 17.05 9.75 393 0.00 .... . 0.00 .... .0a.03.19 24 62.74 38.78 2.40 3366 6.28 27.77 293 17.68 9.16 365 0.00 .... . 0.00 .... .0a.03.16 24 63.24 40.51 2.33 3237 6.10 29.41 282 16.63 9.34 403 0.00 .... . 0.00 .... .0a.02.9 24 61.31 37.85 2.26 3474 5.95 29.06 289 17.51 8.78 460 0.00 .... . 0.00 .... .0a.02.14 24 61.83 38.05 2.37 3185 6.16 27.20 309 17.61 8.69 474 0.00 .... . 0.00 .... .0a.02.10 23 60.75 37.97 2.33 3423 5.97 29.50 283 16.82 8.62 452 0.00 .... . 0.00 .... .0a.02.7 23 57.48 35.26 2.33 3412 5.85 28.95 310 16.37 8.65 440 0.00 .... . 0.00 .... .0a.03.11 24 61.75 38.92 2.24 3490 5.93 29.31 298 16.90 8.47 453 0.00 .... . 0.00 .... .0a.03.4 23 61.35 39.36 2.21 3562 5.78 28.82 291 16.20 9.83 365 0.00 .... . 0.00 .... .0a.02.16 24 61.70 38.00 2.40 3473 6.11 27.69 312 17.58 8.85 491 0.00 .... . 0.00 .... .0a.03.3 23 62.76 39.20 2.62 2912 6.08 29.18 281 17.48 8.68 386 0.00 .... . 0.00 .... .0a.03.7 24 62.99 40.16 2.42 3037 6.16 29.02 294 16.67 9.64 374 0.00 .... . 0.00 .... .0a.03.5 24 63.29 39.66 2.40 3344 6.15 27.10 315 17.48 8.68 428 0.00 .... . 0.00 .... .0a.01.19 24 62.24 39.45 2.51 3041 6.11 27.71 303 16.68 9.47 511 0.00 .... . 0.00 .... .0a.01.17 24 62.38 39.51 2.16 3351 6.02 29.16 308 16.85 8.97 564 0.00 .... . 0.00 .... .0a.01.16 24 61.46 38.62 2.34 3270 6.18 28.09 306 16.67 8.64 555 0.00 .... . 0.00 .... .0a.01.14 25 64.27 40.96 2.52 3063 6.16 28.16 296 17.15 9.97 454 0.00 .... . 0.00 .... .0a.03.13 24 62.54 39.08 2.75 2821 6.15 28.75 299 17.31 9.30 443 0.00 .... . 0.00 .... .0a.03.9 24 64.59 41.79 2.54 3015 5.93 28.06 323 16.87 8.91 428 0.00 .... . 0.00 .... ./aggr0_sas_600g/plex0/rg1:0a.03.8 7 21.60 0.00 .... . 7.78 54.57 120 13.82 19.67 161 0.00 .... . 0.00 .... .0a.02.21 8 21.72 0.00 .... . 7.89 53.80 124 13.82 19.64 194 0.00 .... . 0.00 .... .0a.03.18 25 63.57 41.21 2.68 2986 6.03 28.37 332 16.33 9.35 418 0.00 .... . 0.00 .... .0a.03.0 24 62.14 40.19 2.34 3235 6.05 28.49 307 15.90 9.74 336 0.00 .... . 0.00 .... .0a.02.18 25 62.19 41.27 2.16 3611 5.83 28.69 318 15.09 9.18 451 0.00 .... . 0.00 .... .0a.02.15 24 62.18 40.48 2.36 3201 5.95 27.20 327 15.75 8.67 456 0.00 .... . 0.00 .... .0a.03.21 24 63.62 42.40 2.29 3318 5.92 27.13 336 15.30 9.85 386 0.00 .... . 0.00 .... .0a.03.12 24 61.95 40.08 2.41 2970 5.82 27.61 334 16.05 9.34 380 0.00 .... . 0.00 .... .0a.02.20 25 63.19 41.42 2.41 3333 5.85 28.05 312 15.92 8.87 482 0.00 .... . 0.00 .... .0a.02.23 25 62.86 40.58 2.20 3548 6.02 27.25 324 16.27 8.83 520 0.00 .... . 0.00 .... .0a.02.12 25 64.39 42.82 2.43 3185 5.80 27.85 315 15.77 8.92 464 0.00 .... . 0.00 .... .0a.03.20 24 63.44 41.57 2.34 3159 5.87 27.65 303 16.00 8.72 423 0.00 .... . 0.00 .... .0a.03.14 24 62.74 40.76 2.38 3287 6.08 28.84 310 15.90 9.07 427 0.00 .... . 0.00 .... .0a.02.5 24 61.81 40.05 2.14 3729 6.02 28.72 311 15.75 8.73 506 0.00 .... . 0.00 .... .0a.01.2 26 62.73 41.37 2.37 3226 6.11 27.65 335 15.24 8.99 552 0.00 .... . 0.00 .... .0a.01.3 24 57.66 35.92 2.39 3542 5.82 26.97 338 15.92 8.93 516 0.00 .... . 0.00 .... .0a.01.10 24 61.45 39.43 2.40 3215 6.18 28.27 315 15.84 8.81 545 0.00 .... . 0.00 .... .0a.01.23 24 60.92 39.20 2.54 2881 5.90 26.77 344 15.82 9.01 574 0.00 .... . 0.00 .... .0a.01.15 24 60.35 38.05 2.33 3345 6.02 28.27 305 16.28 8.92 555 0.00 .... . 0.00 .... .0a.01.21 25 63.16 41.62 2.31 3374 5.65 29.21 306 15.89 8.90 572 0.00 .... . 0.00 .... .0a.01.13 25 62.38 40.58 2.51 3214 5.88 27.73 338 15.92 9.22 573 0.00 .... . 0.00 .... .0a.01.8 26 64.69 43.12 2.35 3346 5.80 27.22 330 15.77 9.34 530 0.00 .... . 0.00 .... .0a.01.12 25 62.34 40.58 2.29 3404 5.92 28.99 316 15.85 9.06 557 0.00 .... . 0.00 .... ./aggr0_sas_600g/plex0/rg2:0a.02.0 7 20.47 0.00 .... . 7.69 55.57 119 12.78 18.03 178 0.00 .... . 0.00 .... .0a.03.1 7 20.59 0.00 .... . 7.81 54.77 120 12.78 17.99 163 0.00 .... . 0.00 .... .0a.01.0 28 66.68 44.17 2.32 3498 6.41 30.22 311 16.10 9.73 506 0.00 .... . 0.00 .... .0a.02.1 26 64.67 42.92 2.31 3459 6.26 31.12 317 15.49 9.80 449 0.00 .... . 0.00 .... .0a.03.2 26 65.77 43.57 2.47 3255 6.08 30.55 307 16.12 9.35 422 0.00 .... . 0.00 .... .0a.01.1 27 67.36 45.53 2.36 3291 6.23 31.81 283 15.60 10.42 418 0.00 .... . 0.00 .... .0a.02.3 27 66.85 45.00 2.19 3569 6.05 29.12 317 15.80 9.93 425 0.00 .... . 0.00 .... .0a.03.6 27 70.77 47.82 2.21 3569 6.38 30.96 308 16.57 9.28 410 0.00 .... . 0.00 .... .0a.01.7 27 65.07 42.47 2.37 3412 6.25 30.77 305 16.35 9.73 489 0.00 .... . 0.00 .... .0a.02.4 27 65.73 44.65 2.46 3134 5.93 30.35 302 15.15 10.49 414 0.00 .... . 0.00 .... .0a.03.10 27 66.80 44.37 2.24 3635 6.41 30.98 311 16.02 9.60 410 0.00 .... . 0.00 .... .0a.01.9 27 66.53 44.56 2.25 3698 6.11 29.33 320 15.85 10.37 502 0.00 .... . 0.00 .... .0a.02.6 26 64.67 42.16 2.38 3264 6.15 32.52 293 16.37 9.80 452 0.00 .... . 0.00 .... .0a.03.15 28 68.97 46.26 2.12 3974 6.21 30.80 327 16.50 9.21 487 0.00 .... . 0.00 .... .0a.01.11 27 67.25 44.28 2.27 3737 6.38 30.34 320 16.58 9.04 607 0.00 .... . 0.00 .... .0a.02.8 26 65.32 41.81 2.43 3292 6.28 30.20 321 17.23 8.81 517 0.00 .... . 0.00 .... .0a.03.17 28 67.45 44.95 2.20 3792 6.18 30.23 329 16.32 9.06 494 0.00 .... . 0.00 .... .0a.01.18 27 66.80 44.78 2.28 3647 6.18 29.31 336 15.84 9.67 557 0.00 .... . 0.00 .... .0a.02.11 27 64.69 42.47 2.27 3677 6.26 29.47 311 15.95 10.48 463 0.00 .... . 0.00 .... .0a.03.22 27 65.58 43.82 2.34 3382 5.97 29.82 312 15.80 9.65 477 0.00 .... . 0.00 .... .0a.01.20 28 67.93 45.13 2.47 3315 6.45 29.24 322 16.35 9.60 588 0.00 .... . 0.00 .... .0a.02.17 28 67.00 43.77 2.34 3529 6.48 29.63 332 16.75 9.66 516 0.00 .... . 0.00 .... .0a.03.23 27 70.05 46.68 2.25 3592 6.33 30.59 320 17.05 9.20 509 0.00 .... . 0.00 .... ./aggr0_sas_600g/plex0/rg3:3b.05.0 6 18.91 0.00 .... . 7.86 56.64 108 11.05 15.07 148 0.00 .... . 0.00 .... .0a.02.19 6 19.03 0.00 .... . 7.98 55.84 110 11.05 15.07 179 0.00 .... . 0.00 .... .0a.01.22 30 74.03 51.36 2.46 3262 6.58 35.44 279 16.08 10.13 494 0.00 .... . 0.00 .... .3b.05.1 31 77.13 54.15 2.37 3481 6.75 37.11 262 16.23 9.43 452 0.00 .... . 0.00 .... .0a.02.22 31 77.30 54.24 2.49 3373 6.55 35.55 284 16.52 9.43 498 0.00 .... . 0.00 .... .3b.05.2 30 73.74 50.76 2.35 3684 6.75 36.56 275 16.23 9.47 419 0.00 .... . 0.00 .... .3b.05.20 31 76.57 54.02 2.50 3139 6.70 35.59 259 15.85 9.89 448 0.00 .... . 0.00 .... .3b.05.4 30 76.29 54.19 2.34 3523 6.43 37.21 236 15.67 10.22 378 0.00 .... . 0.00 .... .3b.05.5 29 74.09 52.11 2.32 3478 6.46 36.30 247 15.52 9.83 382 0.00 .... . 0.00 .... .3b.05.6 30 74.94 53.11 2.60 3120 6.48 34.96 273 15.35 10.44 401 0.00 .... . 0.00 .... .3b.05.7 30 76.80 54.24 2.51 3188 6.75 34.71 280 15.82 10.31 396 0.00 .... . 0.00 .... .3b.05.8 31 79.71 56.46 2.40 3347 6.66 35.86 278 16.58 9.35 382 0.00 .... . 0.00 .... .3b.05.9 30 74.57 52.08 2.41 3424 6.70 35.91 262 15.80 9.59 426 0.00 .... . 0.00 .... .3b.05.10 30 76.72 54.43 2.52 3285 6.65 35.93 253 15.64 9.97 437 0.00 .... . 0.00 .... .3b.05.11 32 76.53 54.07 2.38 3536 6.83 33.47 301 15.64 9.76 463 0.00 .... . 0.00 .... .3b.05.12 31 77.58 55.02 2.42 3403 6.65 35.36 262 15.92 10.24 430 0.00 .... . 0.00 .... .3b.05.13 31 75.65 52.47 2.35 3424 6.76 35.10 270 16.42 9.69 481 0.00 .... . 0.00 .... .3b.05.14 31 75.22 52.61 2.53 3314 6.71 37.28 251 15.90 9.64 455 0.00 .... . 0.00 .... .3b.05.15 30 74.26 51.59 2.52 3221 6.55 36.06 266 16.12 9.73 439 0.00 .... . 0.00 .... .3b.05.16 32 78.20 55.80 2.37 3604 6.70 36.85 267 15.70 9.54 527 0.00 .... . 0.00 .... .3b.05.17 31 75.70 53.24 2.42 3446 6.58 36.37 261 15.89 9.55 492 0.00 .... . 0.00 .... .3b.05.18 31 77.58 55.10 2.35 3422 6.68 35.76 272 15.80 9.65 515 0.00 .... . 0.00 .... .3b.05.19 30 75.87 53.40 2.57 3261 6.65 36.61 271 15.82 9.57 542 0.00 .... . 0.00 .... . Aggregate statistics:Minimum 6 18.91 0.00 5.65 11.05 0.00 0.00Mean 25 62.98 40.64 6.40 15.92 0.00 0.00Maximum 32 79.71 56.46 9.80 17.68 0.00 0.00Alon Zeltser Solution Architect ,E&M Computing Tel: 972-35766966 (227) | Mobile: 972-544770108 |
