Hi
I have a FAS3240 running 8.0.5 with 256g flashcache
The filer has single aggregate with 92 X 15k disks is using for vmware (nfs over 10g)
I have very good performance and latency no complains here
The filer is running with average of 13,000 iops and I'm considering adding another 7000 iops to this controller
Trying to decide if it's a good idea or not (planning to add 2 new disk shelves)
Disk utilization is low 40% average
Analyzing perfstat I see I have average CPU utilization of 80-85% (see logs below)
First question do you think 80% disk utilization without any other process (no dedupe, no compression , no scrub , no snapmirror, no backups) only user activity, is 80% CPU is normal for this platform with this load ?
SPM say the controller should be around 40% CPU with this load
Another question I can't figure out
Looking at FlashCache statistics I see excellent hit rate over 80% and 24,500 reads replaced per second
How can it be that over 24,000 iops replaced per second when sysstat show much less total iops
Is the ext_cache_obj show only the maximum or average for the perfstat duration ?
ext_cache_obj:ec0:usage:77% ext_cache_obj:ec0:accesses_total:424571082367 ext_cache_obj:ec0:accesses:45930/s ext_cache_obj:ec0:disk_reads_replaced:24502/s ext_cache_obj:ec0:hit:39725/s ext_cache_obj:ec0:hit_percent:86%
The last question do you think I should add the 7000 iops to this controller or get a new system ?
Thanks
------- ext_cache_obj ----------
TIME: Tue Nov 05 09:54:47 GMT 2013 TIME_DELTA: 60:34 (3634s) ext_cache_obj:ec0:type:IOMEM-FLASH ext_cache_obj:ec0:uptime:1302310171 ext_cache_obj:ec0:blocks:67108864 ext_cache_obj:ec0:size:256 ext_cache_obj:ec0:media:Flash ext_cache_obj:ec0:associativity:8 ext_cache_obj:ec0:sets:16777216 ext_cache_obj:ec0:usage:77% ext_cache_obj:ec0:accesses_total:424571082367 ext_cache_obj:ec0:accesses:45930/s ext_cache_obj:ec0:disk_reads_replaced:24502/s ext_cache_obj:ec0:hit:39725/s ext_cache_obj:ec0:hit_flushq:0/s ext_cache_obj:ec0:hit_once:0/s ext_cache_obj:ec0:hit_age:0/s ext_cache_obj:ec0:hit_normal_lev0:34737/s ext_cache_obj:ec0:hit_metadata_file:3833/s ext_cache_obj:ec0:hit_directory:0/s ext_cache_obj:ec0:hit_redirect:0/s ext_cache_obj:ec0:hit_indirect:1238/s ext_cache_obj:ec0:hit_type.l0.old/new inomap file:11 ext_cache_obj:ec0:hit_type.l0.normal file:124863682 ext_cache_obj:ec0:hit_type.l0.directory:1190 ext_cache_obj:ec0:hit_type.l0.active bitmap:7498789 ext_cache_obj:ec0:hit_type.l0.snap summary:6262825 ext_cache_obj:ec0:hit_type.l0.block types:21600 ext_cache_obj:ec0:hit_type.l0.spacemap:211740 ext_cache_obj:ec0:hit_type.l0.metafile:258 ext_cache_obj:ec0:hit_type.l0.private inode file:6 ext_cache_obj:ec0:hit_type.l0.public inode file:7159 ext_cache_obj:ec0:hit_type.l0.block refcount:853578 ext_cache_obj:ec0:hit_type.l0.private inode map:6 ext_cache_obj:ec0:hit_type.l1.normal file:2477006 ext_cache_obj:ec0:hit_type.l1.directory:11 ext_cache_obj:ec0:hit_type.l1.special file:1 ext_cache_obj:ec0:hit_type.l1.active bitmap:1617 ext_cache_obj:ec0:hit_type.l1.snap summary:36 ext_cache_obj:ec0:hit_type.l1.block types:24413 ext_cache_obj:ec0:hit_type.l1.vvol container:203275 ext_cache_obj:ec0:hit_type.l1.private inode file:2 ext_cache_obj:ec0:hit_type.l1.public inode file:47 ext_cache_obj:ec0:hit_type.l1.block refcount:278778 ext_cache_obj:ec0:hit_type.l1.delete log:1170345 ext_cache_obj:ec0:hit_type.l1.purge file:359 ext_cache_obj:ec0:hit_type.l2.normal file:290697 ext_cache_obj:ec0:hit_type.l2.active bitmap:6 ext_cache_obj:ec0:hit_type.l2.block types:183 ext_cache_obj:ec0:hit_type.l2.vvol container:44698 ext_cache_obj:ec0:hit_type.l2.delete log:2577 ext_cache_obj:ec0:hit_type.l3.normal file:2567 ext_cache_obj:ec0:hit_type.l3.vvol container:711 ext_cache_obj:ec0:hit_type.l3.delete log:40 ext_cache_obj:ec0:hit_partial:101/s ext_cache_obj:ec0:total_metadata_hits:5072/s ext_cache_obj:ec0:total_metadata_misses:784/s ext_cache_obj:ec0:miss:6204/s ext_cache_obj:ec0:miss_flushq:3851/s ext_cache_obj:ec0:miss_once:2808/s ext_cache_obj:ec0:miss_age:0/s ext_cache_obj:ec0:miss_normal_lev0:5420/s ext_cache_obj:ec0:miss_metadata_file:249/s ext_cache_obj:ec0:miss_directory:0/s ext_cache_obj:ec0:miss_indirect:535/s ext_cache_obj:ec0:miss_ssd:0/s ext_cache_obj:ec0:miss_type.l0.old/new inomap file:1 ext_cache_obj:ec0:miss_type.l0.normal file:18874217 ext_cache_obj:ec0:miss_type.l0.directory:244 ext_cache_obj:ec0:miss_type.l0.active bitmap:1415914 ext_cache_obj:ec0:miss_type.l0.snap summary:79062 ext_cache_obj:ec0:miss_type.l0.block types:34523 ext_cache_obj:ec0:miss_type.l0.spacemap:8153 ext_cache_obj:ec0:miss_type.l0.metafile:1 ext_cache_obj:ec0:miss_type.l0.private inode file:1 ext_cache_obj:ec0:miss_type.l0.public inode file:5431 ext_cache_obj:ec0:miss_type.l0.block refcount:165473 ext_cache_obj:ec0:miss_type.l0.private inode map:1 ext_cache_obj:ec0:miss_type.l1.normal file:325314 ext_cache_obj:ec0:miss_type.l1.special file:12 ext_cache_obj:ec0:miss_type.l1.active bitmap:1432 ext_cache_obj:ec0:miss_type.l1.block types:928 ext_cache_obj:ec0:miss_type.l1.vvol container:1518334 ext_cache_obj:ec0:miss_type.l1.block refcount:1460 ext_cache_obj:ec0:miss_type.l1.delete log:76923 ext_cache_obj:ec0:miss_type.l1.purge file:19 ext_cache_obj:ec0:miss_type.l2.normal file:13912 ext_cache_obj:ec0:miss_type.l2.special file:1 ext_cache_obj:ec0:miss_type.l2.active bitmap:3 ext_cache_obj:ec0:miss_type.l2.block types:11 ext_cache_obj:ec0:miss_type.l2.vvol container:3838 ext_cache_obj:ec0:miss_type.l2.delete log:102 ext_cache_obj:ec0:miss_type.l3.normal file:154 ext_cache_obj:ec0:miss_type.l3.vvol container:27 ext_cache_obj:ec0:miss_type.l3.delete log:2 ext_cache_obj:ec0:lookup_reject:1506/s ext_cache_obj:ec0:lookup_reject_normal_l0:0/s ext_cache_obj:ec0:lookup_reject_io:0/s ext_cache_obj:ec0:lookup_reject_lock:0/s ext_cache_obj:ec0:lookup_generation_mismatch:3007 ext_cache_obj:ec0:lookup_checksum_mismatch:2 ext_cache_obj:ec0:lookup_context_mismatch:0 ext_cache_obj:ec0:lookup_disk_ios_issued:3009 ext_cache_obj:ec0:lookup_chains:28388/s ext_cache_obj:ec0:lookup_chain_cnt:50724/s ext_cache_obj:ec0:lookup_lwr_mismatch:0/s ext_cache_obj:ec0:hit_percent:86% ext_cache_obj:ec0:inserts:3227/s ext_cache_obj:ec0:inserts_mbuf:20/s ext_cache_obj:ec0:inserts_mbuf_zerocopy:0/s ext_cache_obj:ec0:inserts_flushq:0/s ext_cache_obj:ec0:inserts_once:0/s ext_cache_obj:ec0:inserts_age:20/s ext_cache_obj:ec0:inserts_normal_lev0:1472/s ext_cache_obj:ec0:inserts_metadata_file:812/s ext_cache_obj:ec0:inserts_directory:0/s ext_cache_obj:ec0:inserts_indirect:955/s ext_cache_obj:ec0:insert_rejects_misc:0/s ext_cache_obj:ec0:insert_rejects_compressed:0/s ext_cache_obj:ec0:insert_rejects_ssd:0/s ext_cache_obj:ec0:insert_rejects_present:36702/s ext_cache_obj:ec0:insert_rejects_flushq:16717/s ext_cache_obj:ec0:insert_rejects_normal_lev0:0/s ext_cache_obj:ec0:insert_rejects_throttle:144/s ext_cache_obj:ec0:insert_rejects_throttle_io:0/s ext_cache_obj:ec0:insert_rejects_throttle_refill:144/s ext_cache_obj:ec0:insert_rejects_throttle_mem:0/s ext_cache_obj:ec0:insert_rejects_cache_reuse:0/s ext_cache_obj:ec0:insert_rejects_vbn_invalid:3227/s ext_cache_obj:ec0:insert_rejects_pvbn_aliased:0/s ext_cache_obj:ec0:insert_rejects_mbuf_nobuf:0/s ext_cache_obj:ec0:insert_rejects_mbuf_low:2189/s ext_cache_obj:ec0:insert_rejects_throttle_rate:0/s ext_cache_obj:ec0:insert_rejects_erase_stripe_state:0/s ext_cache_obj:ec0:insert_rejects_erase_queue:0/s ext_cache_obj:ec0:insert_rejects_mbuf_off:0/s ext_cache_obj:ec0:insert_rejects_no_wip:987/s ext_cache_obj:ec0:insert_rejects_vol_not_mounted:0/s ext_cache_obj:ec0:insert_rejects_delete_log:0/s ext_cache_obj:ec0:insert_rejects_not_valid:104/s ext_cache_obj:ec0:insert_rejects_errored:0/s ext_cache_obj:ec0:insert_rejects_no_context:0/s ext_cache_obj:ec0:reuse_percent:1230% ext_cache_obj:ec0:evicts:36/s ext_cache_obj:ec0:evicts_ref:16/s ext_cache_obj:ec0:readio_solitary:17462/s ext_cache_obj:ec0:readio_chains:24501/s ext_cache_obj:ec0:readio_blocks:39369/s ext_cache_obj:ec0:readio_in_flight:0 ext_cache_obj:ec0:readio_max_in_flight:511 ext_cache_obj:ec0:readio_general_errors:0 ext_cache_obj:ec0:readio_transient_errors:0 ext_cache_obj:ec0:readio_avg_chainlength:1.61 ext_cache_obj:ec0:readio_avg_latency:1.70ms ext_cache_obj:ec0:readio_latency.0:20693251 ext_cache_obj:ec0:readio_latency.1:39478840 ext_cache_obj:ec0:readio_latency.2:11080740 ext_cache_obj:ec0:readio_latency.3:7053418 ext_cache_obj:ec0:readio_latency.4:3915379 ext_cache_obj:ec0:readio_latency.5:2101373 ext_cache_obj:ec0:readio_latency.6:1295253 ext_cache_obj:ec0:readio_latency.7:877946 ext_cache_obj:ec0:readio_latency.<16:2131721 ext_cache_obj:ec0:readio_latency.<24:245311 ext_cache_obj:ec0:readio_latency.<32:38177 ext_cache_obj:ec0:readio_latency.<40:18450 ext_cache_obj:ec0:readio_latency.<48:8463 ext_cache_obj:ec0:readio_latency.<56:4540 ext_cache_obj:ec0:readio_latency.<64:2248 ext_cache_obj:ec0:readio_latency.<128:3371 ext_cache_obj:ec0:readio_latency.<192:19 ext_cache_obj:ec0:readio_chain_lengths.1:63394306 ext_cache_obj:ec0:readio_chain_lengths.2:5874105 ext_cache_obj:ec0:readio_chain_lengths.3:16774665 ext_cache_obj:ec0:readio_chain_lengths.4:1088869 ext_cache_obj:ec0:readio_chain_lengths.5:306521 ext_cache_obj:ec0:readio_chain_lengths.6:871098 ext_cache_obj:ec0:readio_chain_lengths.7:157207 ext_cache_obj:ec0:readio_chain_lengths.<16:434548 ext_cache_obj:ec0:readio_chain_lengths.<24:36558 ext_cache_obj:ec0:readio_chain_lengths.<32:5244 ext_cache_obj:ec0:readio_chain_lengths.<40:3359 ext_cache_obj:ec0:readio_chain_lengths.<48:936 ext_cache_obj:ec0:readio_chain_lengths.<56:632 ext_cache_obj:ec0:readio_chain_lengths.<64:389 ext_cache_obj:ec0:readio_chain_lengths.<128:53 ext_cache_obj:ec0:writeio_solitary:0/s ext_cache_obj:ec0:writeio_chains:50/s ext_cache_obj:ec0:writeio_blocks:3227/s ext_cache_obj:ec0:writeio_in_flight:0 ext_cache_obj:ec0:writeio_max_in_flight:107 ext_cache_obj:ec0:writeio_general_errors:0 ext_cache_obj:ec0:writeio_transient_errors:0 ext_cache_obj:ec0:writeio_avg_chainlength:64.00 ext_cache_obj:ec0:writeio_avg_latency:4.62ms ext_cache_obj:ec0:writeio_latency.0:358 ext_cache_obj:ec0:writeio_latency.1:23690 ext_cache_obj:ec0:writeio_latency.2:41774 ext_cache_obj:ec0:writeio_latency.3:27620 ext_cache_obj:ec0:writeio_latency.4:17766 ext_cache_obj:ec0:writeio_latency.5:14558 ext_cache_obj:ec0:writeio_latency.6:12651 ext_cache_obj:ec0:writeio_latency.7:12580 ext_cache_obj:ec0:writeio_latency.<16:29129 ext_cache_obj:ec0:writeio_latency.<24:2504 ext_cache_obj:ec0:writeio_latency.<32:337 ext_cache_obj:ec0:writeio_latency.<40:83 ext_cache_obj:ec0:writeio_latency.<48:27 ext_cache_obj:ec0:writeio_latency.<56:5 ext_cache_obj:ec0:writeio_latency.<64:2 ext_cache_obj:ec0:writeio_latency.<128:4 ext_cache_obj:ec0:writeio_chain_lengths.<128:183088 ext_cache_obj:ec0:blocks_ref0:41301539 ext_cache_obj:ec0:blocks_ref1:18120071 ext_cache_obj:ec0:blocks_ref2:2121072 ext_cache_obj:ec0:blocks_ref3:1084969 ext_cache_obj:ec0:blocks_ref4:599609 ext_cache_obj:ec0:blocks_ref5:417174 ext_cache_obj:ec0:blocks_ref6:286331 ext_cache_obj:ec0:blocks_ref7:3178099 ext_cache_obj:ec0:blocks_ref0_arrivals:27596243622 ext_cache_obj:ec0:blocks_ref1_arrivals:27622050947 ext_cache_obj:ec0:blocks_ref2_arrivals:10423310000 ext_cache_obj:ec0:blocks_ref3_arrivals:6427750307 ext_cache_obj:ec0:blocks_ref4_arrivals:4495536161 ext_cache_obj:ec0:blocks_ref5_arrivals:3557830045 ext_cache_obj:ec0:blocks_ref6_arrivals:2957867966 ext_cache_obj:ec0:blocks_ref7_arrivals:117062905990 ext_cache_obj:ec0:lru_ticks:3563396220 ext_cache_obj:ec0:invalidates:3526/s ext_cache_obj:ec0:erase_stripe_invalidates:6/s ext_cache_obj:ec0:erase_stripe_blocks_invalidates:3226/s ext_cache_obj:ec0:cache_utilization:77% ext_cache_obj:ec0:pcs_invalidates_flash:0/s ext_cache_obj:ec0:pcs_wraps:0 ext_cache_obj:ec0:pcs_wrap_time:0 ext_cache_obj:ec0:pcs_confidence:100 ext_cache_obj:ec0:pcs_total_ops_insert:0/s ext_cache_obj:ec0:pcs_total_ops_lookup:0/s ext_cache_obj:ec0:pcs_total_ops_invalidate:0/s ext_cache_obj:ec0:pcs_total_ops_misc:0/s ext_cache_obj:ec0:pcs_sampled_ops_insert:0/s ext_cache_obj:ec0:pcs_sampled_ops_lookup:0/s ext_cache_obj:ec0:pcs_sampled_ops_invalidate:0/s ext_cache_obj:ec0:pcs_sampled_ops_misc:0/s ext_cache_obj:ec0:pcs_sampling_rate:0
---------- sysstat -x 1 -----------
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s
in out read write read write age hit time ty util in out in out
96% 16951 0 0 16951 22579 180997 108622 57263 0 0 4s 67% 100% :f 50% 0 0 0 0 0 0 0
89% 20267 0 0 20267 45941 229706 111661 93433 0 0 5s 51% 100% :f 44% 0 0 0 0 0 0 0
90% 17307 0 0 17307 56792 223833 118109 105672 0 0 3s 48% 100% :f 44% 0 0 0 0 0 0 0
90% 20658 0 0 20658 34301 267718 41860 22109 0 0 7s 50% 38% : 39% 0 0 0 0 0 0 0
77% 12840 0 0 12843 44520 142228 49804 0 0 0 8s 58% 0% - 47% 3 0 0 0 0 0 0
95% 16191 0 0 16191 33156 180821 149909 121486 0 0 5s 53% 58% Dn 46% 0 0 0 0 0 0 0
97% 15198 0 0 15198 23309 199482 100311 43518 0 0 4s 58% 100% :s 42% 0 0 0 0 0 0 0
91% 15363 0 0 15363 22212 216210 82631 62381 0 0 5s 62% 100% :f 46% 0 0 0 0 0 0 0
87% 17662 0 0 17662 36828 209514 118430 79394 0 0 6s 53% 100% :f 49% 0 0 0 0 0 0 0
80% 14307 0 0 14310 47874 166153 74677 54134 0 0 6s 55% 100% :f 49% 3 0 0 0 0 0 0
80% 13905 0 0 13905 42593 202792 49344 49444 0 0 5s 53% 57% : 48% 0 0 0 0 0 0 0
88% 16720 0 0 16720 59662 244810 41227 0 0 0 3s 48% 0% - 44% 0 0 0 0 0 0 0
93% 14717 0 0 14717 17933 223608 155425 102506 0 0 7s 55% 46% Dn 46% 0 0 0 0 0 0 0
96% 13937 0 0 13937 17215 164887 112227 64931 0 0 7s 54% 100% :s 43% 0 0 0 0 0 0 0
92% 13494 0 0 13497 28609 195722 71916 54271 0 0 5s 64% 100% :f 46% 3 0 0 0 0 0 0
93% 22023 0 0 22023 28545 254401 73108 96616 0 0 4s 48% 100% :f 41% 0 0 0 0 0 0 0
88% 23112 0 0 23112 24806 213939 64348 73510 0 0 5s 53% 100% :f 44% 0 0 0 0 0 0 0
84% 17240 0 0 17240 30511 203088 45549 22405 0 0 3s 54% 36% : 41% 0 0 0 0 0 0 0
90% 20676 0 0 20676 61721 269589 35822 24 0 0 4s 51% 0% - 36% 0 0 0 0 0 0 0
84% 19089 0 0 19100 42507 229811 34014 0 0 0 5s 54% 0% - 38% 11 0 0 0 0 0 0
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s
in out read write read write age hit time ty util in out in out
92% 17690 0 0 17690 32740 195063 106798 63482 0 0 5s 51% 33% Dn 40% 0 0 0 0 0 0 0
96% 13450 0 0 13450 18298 175936 97234 80224 0 0 4s 60% 100% :s 39% 0 0 0 0 0 0 0
94% 14813 0 0 14813 13928 204597 69566 61750 0 0 5s 57% 100% :f 39% 0 0 0 0 0 0 0
89% 18505 0 0 18509 23968 247285 104228 69749 0 0 6s 54% 100% :f 47% 4 0 0 0 0 0 0
91% 21963 0 0 21967 60710 223419 78861 66270 0 0 3s 54% 100% :f 42% 4 0 0 0 0 0 0
89% 19919 0 0 19919 33959 215890 79309 59648 0 0 5s 53% 100% :f 45% 0 0 0 0 0 0 0
90% 19707 0 0 19707 27499 247462 35972 33722 0 0 6s 50% 59% : 35% 0 0 0 0 0 0 0
86% 19456 0 0 19456 43880 233384 37526 0 0 0 4s 55% 0% - 38% 0 0 0 0 0 0 0
94% 20130 0 0 20130 63247 229293 78845 48829 0 0 2s 54% 31% Dn 35% 0 0 0 0 0 0 0
96% 10225 0 0 10228 45026 101768 193293 199449 0 0 3s 68% 100% :s 47% 3 0 0 0 0 0 0
95% 13359 0 0 13359 39462 191730 99143 52233 0 0 4s 61% 100% :f 46% 0 0 0 0 0 0 0
87% 15657 0 0 15657 21113 257858 95879 61368 0 0 5s 53% 100% :f 41% 0 0 0 0 0 0 0
89% 19590 0 0 19590 51499 252323 57820 65116 0 0 6s 51% 100% :v 39% 0 0 0 0 0 0 0
77% 14530 0 0 14530 31816 178572 28376 32 0 0 6s 53% 7% : 39% 0 0 0 0 0 0 0
95% 14670 0 0 14673 36120 218937 136512 118201 0 0 5s 55% 54% Dn 45% 3 0 0 0 0 0 0
97% 15592 0 0 15592 24149 170596 136341 76393 0 0 6s 56% 100% :s 41% 0 0 0 0 0 0 0
95% 15562 0 0 15562 46759 158751 158750 58356 0 0 7s 72% 100% :f 59% 0 0 0 0 0 0 0
84% 15471 0 0 15471 15530 229961 145671 73636 0 0 2s 55% 100% :f 47% 0 0 0 0 0 0 0
85% 17753 0 0 17753 53878 214190 53348 46938 0 0 2s 50% 76% : 43% 0 0 0 0 0 0 0
84% 15532 0 0 15535 33703 231932 52108 0 0 0 10s 54% 0% - 48% 3 0 0 0 0 0 0
Summary Statistics ( 3595 samples 0.1 secs/sample) CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk OTHER FCP iSCSI FCP kB/s iSCSI kB/s in out read write read write age hit time ty util in out in out Min 31% 3582 0 0 3582 9751 7799 8404 0 0 0 0s 42% 0% * 12% 0 0 0 0 0 0 0 Avg 87% 13331 0 0 13340 64420 184870 114575 102713 0 0 5s 56% 73% * 43% 8 0 0 0 0 0 0 Max 99% 25894 0 0 25894 215371 661784 322220 330317 0 0 21s 94% 100% * 90% 547 0 0 0 0 0 0
--------- Sysstat -M ---------
ANY1+ ANY2+ ANY3+ ANY4+ AVG CPU0 CPU1 CPU2 CPU3 Network Protocol Cluster Storage Raid Target Kahuna WAFL_Ex(Kahu) WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CP 98% 88% 74% 56% 79% 75% 78% 79% 86% 46% 0% 0% 15% 28% 0% 40% 109%( 52%) 10% 0% 0% 65% 4% 1% 13358 81% 98% 88% 73% 55% 79% 75% 77% 80% 86% 46% 0% 0% 13% 27% 0% 41% 108%( 52%) 11% 0% 0% 65% 4% 2% 13152 78% 95% 81% 64% 46% 72% 67% 68% 73% 80% 43% 0% 0% 11% 21% 0% 38% 102%( 50%) 7% 0% 0% 61% 5% 1% 12848 73% 92% 79% 64% 47% 71% 68% 69% 73% 75% 44% 0% 0% 11% 23% 0% 32% 104%( 51%) 8% 0% 0% 57% 5% 1% 13178 64% 90% 78% 64% 48% 71% 68% 69% 72% 73% 44% 0% 0% 11% 24% 0% 30% 101%( 50%) 8% 0% 0% 58% 4% 1% 12550 70% 93% 80% 66% 49% 73% 70% 71% 75% 75% 47% 0% 0% 12% 24% 0% 31% 106%( 53%) 8% 0% 0% 58% 5% 2% 13090 64% 87% 71% 55% 40% 64% 60% 62% 67% 68% 40% 0% 0% 9% 18% 0% 27% 97%( 49%) 6% 0% 0% 51% 5% 1% 12890 67% 91% 77% 61% 45% 69% 65% 67% 71% 74% 43% 0% 0% 11% 22% 0% 32% 100%( 50%) 8% 0% 0% 55% 4% 1% 12697 73% 89% 73% 57% 42% 66% 62% 63% 69% 69% 41% 0% 0% 10% 20% 0% 28% 99%( 50%) 7% 0% 0% 52% 5% 1% 13111 66% 90% 76% 60% 44% 68% 66% 66% 69% 71% 43% 0% 0% 11% 22% 0% 29% 99%( 50%) 8% 0% 0% 55% 4% 2% 12771 72% 87% 71% 55% 40% 64% 60% 62% 66% 67% 41% 0% 0% 10% 18% 0% 27% 96%( 49%) 6% 0% 0% 51% 5% 1% 12675 66% 86% 71% 56% 41% 64% 61% 62% 67% 68% 41% 0% 0% 10% 19% 0% 28% 97%( 48%) 7% 0% 0% 52% 5% 1% 12678 62% 92% 77% 62% 45% 70% 66% 68% 72% 74% 44% 0% 0% 11% 22% 0% 31% 103%( 52%) 8% 0% 0% 55% 5% 1% 13402 75% 94% 81% 67% 49% 74% 69% 72% 76% 77% 45% 0% 0% 12% 26% 0% 33% 104%( 52%) 10% 0% 0% 58% 4% 2% 13289 76% 89% 75% 60% 44% 67% 64% 65% 71% 70% 43% 0% 0% 10% 21% 0% 29% 99%( 50%) 8% 0% 0% 54% 5% 1% 12985 66% 85% 68% 52% 37% 62% 58% 60% 64% 64% 40% 0% 0% 9% 16% 0% 24% 96%( 49%) 6% 0% 0% 49% 5% 1% 12931 64% 90% 75% 61% 45% 69% 64% 66% 72% 72% 42% 0% 0% 10% 21% 0% 30% 103%( 51%) 8% 0% 0% 55% 5% 1% 13163 66% 90% 75% 58% 42% 67% 64% 65% 70% 70% 44% 0% 0% 10% 19% 0% 27% 104%( 52%) 6% 0% 0% 53% 5% 2% 13929 71% 99% 94% 84% 68% 87% 83% 86% 90% 87% 51% 0% 0% 15% 37% 0% 33% 123%( 61%) 14% 0% 0% 69% 4% 1% 14002 81% -- Summary Statistics ( 58 samples 6.0 secs/sample) ANY1+ ANY2+ ANY3+ ANY4+ AVG CPU0 CPU1 CPU2 CPU3 Network Protocol Cluster Storage Raid Target Kahuna WAFL_Ex(Kahu) WAFL_XClean SM_Exempt Cifs Exempt Intr Host Ops/s CP Min 85% 68% 52% 37% 62% 58% 60% 64% 64% 40% 0% 0% 9% 16% 0% 24% 95%( 46%) 6% 0% 0% 49% 3% 1% 10226 58% Avg 95% 84% 70% 54% 76% 73% 75% 78% 80% 46% 0% 0% 13% 28% 0% 35% 106%( 51%) 9% 0% 0% 61% 4% 1% 13234 73% Max 100% 96% 87% 70% 89% 84% 93% 90% 91% 56% 0% 0% 26% 49% 0% 49% 123%( 61%) 18% 0% 0% 72% 5% 2% 16583 95%
---------- STATIT -----------
NetApp Release 8.0.5 7-Mode: Sun Jan 20 21:53:54 PST 2013 <1O> Start time: Tue Nov 5 11:25:18 IST 2013
CPU Statistics 60.182572 time (seconds) 100 % 173.781160 system time 289 % 2.716133 rupt time 5 % (1632432 rupts x 2 usec/rupt) 171.065027 non-rupt system time 284 % 66.949128 idle time 111 %
41.250883 time in CP 69 % 100 % 1.790287 rupt time in CP 4 % (1107164 rupts x 2 usec/rupt)
Multiprocessor Statistics (per second) cpu0 cpu1 cpu2 cpu3 total sk switches 58254.32 59427.55 57820.04 45838.67 221340.59 hard switches 29026.92 30923.44 35440.13 16979.75 112370.24 domain switches 1174.79 2076.38 238.46 316.05 3805.69 CP rupts 14284.20 1370.85 1370.83 1370.88 18396.75 nonCP rupts 6840.54 629.14 629.14 629.10 8727.91 IPI rupts 0.00 0.00 0.00 0.00 0.00 grab kahuna 0.00 0.98 0.91 0.00 1.89 grab w_xcleaner 0.00 0.18 0.81 268.87 269.86
grab kahuna usec 0.00 488.56 910.73 0.00 1399.29 grab w_xcleaner usec 0.00 123.31 347.31 156723.71 157194.33
CP rupt usec 26808.77 651.73 1779.62 507.44 29747.60 nonCP rupt usec 14115.33 288.34 768.00 212.25 15383.96 idle 318889.16 304923.34 259322.25 229299.07 1112433.85 kahuna 0.47 0.00 0.00 332371.27 332371.75 storage 84402.34 12364.19 16805.60 0.00 113572.15 exempt 199610.16 176903.95 212622.70 13.92 589150.76 raid 815.39 200269.12 22897.81 0.00 223982.33 target 15.04 17.15 16.67 0.00 48.88 netcache 0.00 0.00 0.20 0.00 0.20 netcache2 0.00 0.00 0.00 0.00 0.00 cifs 3.66 3.72 2.77 0.00 10.19 wafl_exempt 254690.06 231801.77 136534.46 437596.00 1060622.33 wafl_xcleaner 33024.31 19066.62 23468.94 0.00 75559.88 sm_exempt 19.36 22.46 21.42 0.00 63.27 cluster 0.00 0.00 0.00 0.00 0.00 protocol 26.95 25.87 26.54 0.00 79.38 nwk_exclusive 975.88 511.33 1150.90 0.00 2638.12 nwk_exempt 261.45 359.64 366.10 0.00 987.23 nwk_legacy 1585.44 1830.31 1686.75 0.00 5102.52 nwk_ctx1 360.39 24181.75 303022.64 0.00 327564.81 nwk_ctx2 59706.95 22852.83 16500.72 0.00 99060.52 nwk_ctx3 0.00 0.00 0.00 0.00 0.00 nwk_ctx4 0.00 0.00 0.00 0.00 0.00 hostOS 4688.73 3925.69 3005.77 0.00 11620.19
56.181715 seconds with one or more CPUs active ( 93%) 48.289199 seconds with 2 or more CPUs active ( 80%) 38.846369 seconds with 3 or more CPUs active ( 65%)
7.892516 seconds with one CPU active ( 13%) 9.442829 seconds with 2 CPUs active ( 16%) 10.092179 seconds with 3 CPUs active ( 17%) 28.754190 seconds with all CPUs active ( 48%)
Domain Utilization of Shared Domains (per second) 0.00 idle 516212.17 kahuna 0.00 storage 0.00 exempt 0.00 raid 0.00 target 0.00 netcache 0.00 netcache2 0.00 cifs 0.00 wafl_exempt 0.00 wafl_xcleaner 0.00 sm_exempt 0.00 cluster 0.00 protocol 392429.57 nwk_exclusive 0.00 nwk_exempt 0.00 nwk_legacy 0.00 nwk_ctx1 0.00 nwk_ctx2 0.00 nwk_ctx3 0.00 nwk_ctx4 0.00 hostOS
Miscellaneous Statistics (per second) 112370.24 hard context switches 13517.83 NFS operations 0.00 CIFS operations 0.00 HTTP operations 44559.59 network KB received 184368.21 network KB transmitted 94098.07 disk KB read 78261.13 disk KB written 40535.46 NVRAM KB written 0.00 nolog KB written 45831.13 WAFL bufs given to clients 0.80 checksum cache hits ( 0%) 45787.34 no checksum - partial buffer 0.00 FCP operations 0.00 iSCSI operations
WAFL Statistics (per second) 81.17 name cache hits ( 77%) 24.81 name cache misses ( 23%) 432838.23 buf hash hits ( 87%) 62737.71 buf hash misses ( 13%) 29606.73 inode cache hits ( 100%) 0.00 inode cache misses ( 0%) 47745.08 buf cache hits ( 55%) 39604.57 buf cache misses ( 45%) 2150.64 blocks read 5882.00 blocks read-ahead 1252.07 chains read-ahead 3674.07 dummy reads 4269.61 blocks speculative read-ahead 15858.58 blocks written 30.57 stripes written 0.00 blocks over-written 0.00 wafl_timer generated CP 0.00 snapshot generated CP 0.00 wafl_avail_bufs generated CP 0.02 dirty_blk_cnt generated CP 0.02 full NV-log generated CP 0.00 back-to-back CP 0.00 flush generated CP 0.00 sync generated CP 0.00 deferred back-to-back CP 0.00 container-indirect-pin CP 0.03 low mbufs generated CP 0.13 low datavecs generated CP 59836.91 non-restart messages 516.21 IOWAIT suspends 142731372.60 next nvlog nearly full msecs 0.00 dirty buffer susp msecs 0.00 nvlog full susp msecs 933124 buffers
RAID Statistics (per second) 3433.15 xors 0.00 long dispatches [0] 0.00 long consumed [0] 0.00 long consumed hipri [0] 0.00 long low priority [0] 0.00 long high priority [0] 0.00 long monitor tics [0] 0.00 long monitor clears [0] 0.00 long dispatches [1] 0.00 long consumed [1] 0.00 long consumed hipri [1] 0.00 long low priority [1] 0.00 long high priority [1] 0.00 long monitor tics [1] 0.00 long monitor clears [1] 18 max batch 6.95 blocked mode xor 1220.85 timed mode xor 2.18 fast adjustments 2.63 slow adjustments 0 avg batch start 0 avg stripe/msec 3737.36 checksum dispatches 23508.15 checksum consumed 31.42 tetrises written 0.00 master tetrises 0.00 slave tetrises 1729.77 stripes written 1713.19 partial stripes 16.58 full stripes 16095.72 blocks written 11922.54 blocks read 126.73 1 blocks per stripe size 21 100.83 2 blocks per stripe size 21 94.76 3 blocks per stripe size 21 96.27 4 blocks per stripe size 21 96.14 5 blocks per stripe size 21 96.47 6 blocks per stripe size 21 100.00 7 blocks per stripe size 21 102.87 8 blocks per stripe size 21 100.89 9 blocks per stripe size 21 98.15 10 blocks per stripe size 21 95.36 11 blocks per stripe size 21 91.70 12 blocks per stripe size 21 86.60 13 blocks per stripe size 21 84.97 14 blocks per stripe size 21 78.96 15 blocks per stripe size 21 71.90 16 blocks per stripe size 21 60.37 17 blocks per stripe size 21 54.10 18 blocks per stripe size 21 44.07 19 blocks per stripe size 21 32.04 20 blocks per stripe size 21 16.58 21 blocks per stripe size 21
Network Interface Statistics (per second) iface side bytes packets multicasts errors collisions pkt drops e0a recv 3683.86 50.26 0.00 0.00 0.00 xmit 5048.36 19.94 0.00 0.00 0.00 e0b recv 4408.92 59.15 0.00 0.00 0.00 xmit 20.14 0.10 0.00 0.00 0.00 e2a recv 45620024.58 50126.21 0.00 0.00 0.00 xmit 188787476.51 29046.10 0.18 0.00 0.00 e2b recv 134.66 1.88 0.00 0.00 0.00 xmit 6.98 0.17 0.17 0.00 0.00 e6a recv 71.10 0.91 0.00 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00 e6b recv 59.10 0.63 0.00 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00 e6c recv 0.00 0.00 0.00 71365630.73 0.00 xmit 0.00 0.00 0.00 0.00 0.00 e6d recv 0.00 0.00 0.00 71365630.72 0.00 xmit 0.00 0.00 0.00 0.00 0.00 c0a recv 285.13 21.93 1.46 0.00 0.00 xmit 216.34 1.16 1.16 0.00 0.00 c0b recv 285.13 1.46 1.46 0.00 0.00 xmit 216.34 1.16 1.16 0.00 0.00 e0M recv 0.00 0.00 0.00 0.15 0.00 xmit 0.00 0.00 0.00 0.00 0.00 e0P recv 80.09 0.93 0.08 0.00 0.00 xmit 66.50 0.85 0.00 0.00 0.00 vh recv 0.00 0.00 0.00 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00 vif0-manage recv 8092.77 109.42 89.15 0.00 0.00 xmit 5068.49 20.04 0.00 0.00 0.00 vif1-10g recv 45620159.24 50128.08 3.74 0.70 0.00 xmit 188787483.49 29046.27 0.35 0.00 0.00 vif-174-1g recv 130.20 1.55 1.55 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00
Disk Statistics (per second) ut% is the percent of time the disk was busy. xfers is the number of data-transfer commands issued per second. xfers = ureads + writes + cpreads + greads + gwrites chain is the average number of 4K blocks per command. usecs is the average disk round-trip time per 4K block.
disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs /aggr0_sas_600g/plex0/rg0: 0a.01.4 8 23.16 0.76 1.39 2859 9.29 46.62 120 13.11 19.58 181 0.00 .... . 0.00 .... . 0a.01.5 9 23.76 0.80 1.38 3394 9.80 44.28 119 13.16 19.71 199 0.00 .... . 0.00 .... . 0a.01.6 25 64.90 42.39 2.39 3282 6.23 27.14 297 16.28 9.66 441 0.00 .... . 0.00 .... . 0a.02.13 25 67.20 44.22 2.14 3537 6.06 29.12 311 16.92 8.81 470 0.00 .... . 0.00 .... . 0a.02.2 25 63.74 40.44 2.13 3620 6.25 29.57 297 17.05 9.75 393 0.00 .... . 0.00 .... . 0a.03.19 24 62.74 38.78 2.40 3366 6.28 27.77 293 17.68 9.16 365 0.00 .... . 0.00 .... . 0a.03.16 24 63.24 40.51 2.33 3237 6.10 29.41 282 16.63 9.34 403 0.00 .... . 0.00 .... . 0a.02.9 24 61.31 37.85 2.26 3474 5.95 29.06 289 17.51 8.78 460 0.00 .... . 0.00 .... . 0a.02.14 24 61.83 38.05 2.37 3185 6.16 27.20 309 17.61 8.69 474 0.00 .... . 0.00 .... . 0a.02.10 23 60.75 37.97 2.33 3423 5.97 29.50 283 16.82 8.62 452 0.00 .... . 0.00 .... . 0a.02.7 23 57.48 35.26 2.33 3412 5.85 28.95 310 16.37 8.65 440 0.00 .... . 0.00 .... . 0a.03.11 24 61.75 38.92 2.24 3490 5.93 29.31 298 16.90 8.47 453 0.00 .... . 0.00 .... . 0a.03.4 23 61.35 39.36 2.21 3562 5.78 28.82 291 16.20 9.83 365 0.00 .... . 0.00 .... . 0a.02.16 24 61.70 38.00 2.40 3473 6.11 27.69 312 17.58 8.85 491 0.00 .... . 0.00 .... . 0a.03.3 23 62.76 39.20 2.62 2912 6.08 29.18 281 17.48 8.68 386 0.00 .... . 0.00 .... . 0a.03.7 24 62.99 40.16 2.42 3037 6.16 29.02 294 16.67 9.64 374 0.00 .... . 0.00 .... . 0a.03.5 24 63.29 39.66 2.40 3344 6.15 27.10 315 17.48 8.68 428 0.00 .... . 0.00 .... . 0a.01.19 24 62.24 39.45 2.51 3041 6.11 27.71 303 16.68 9.47 511 0.00 .... . 0.00 .... . 0a.01.17 24 62.38 39.51 2.16 3351 6.02 29.16 308 16.85 8.97 564 0.00 .... . 0.00 .... . 0a.01.16 24 61.46 38.62 2.34 3270 6.18 28.09 306 16.67 8.64 555 0.00 .... . 0.00 .... . 0a.01.14 25 64.27 40.96 2.52 3063 6.16 28.16 296 17.15 9.97 454 0.00 .... . 0.00 .... . 0a.03.13 24 62.54 39.08 2.75 2821 6.15 28.75 299 17.31 9.30 443 0.00 .... . 0.00 .... . 0a.03.9 24 64.59 41.79 2.54 3015 5.93 28.06 323 16.87 8.91 428 0.00 .... . 0.00 .... . /aggr0_sas_600g/plex0/rg1: 0a.03.8 7 21.60 0.00 .... . 7.78 54.57 120 13.82 19.67 161 0.00 .... . 0.00 .... . 0a.02.21 8 21.72 0.00 .... . 7.89 53.80 124 13.82 19.64 194 0.00 .... . 0.00 .... . 0a.03.18 25 63.57 41.21 2.68 2986 6.03 28.37 332 16.33 9.35 418 0.00 .... . 0.00 .... . 0a.03.0 24 62.14 40.19 2.34 3235 6.05 28.49 307 15.90 9.74 336 0.00 .... . 0.00 .... . 0a.02.18 25 62.19 41.27 2.16 3611 5.83 28.69 318 15.09 9.18 451 0.00 .... . 0.00 .... . 0a.02.15 24 62.18 40.48 2.36 3201 5.95 27.20 327 15.75 8.67 456 0.00 .... . 0.00 .... . 0a.03.21 24 63.62 42.40 2.29 3318 5.92 27.13 336 15.30 9.85 386 0.00 .... . 0.00 .... . 0a.03.12 24 61.95 40.08 2.41 2970 5.82 27.61 334 16.05 9.34 380 0.00 .... . 0.00 .... . 0a.02.20 25 63.19 41.42 2.41 3333 5.85 28.05 312 15.92 8.87 482 0.00 .... . 0.00 .... . 0a.02.23 25 62.86 40.58 2.20 3548 6.02 27.25 324 16.27 8.83 520 0.00 .... . 0.00 .... . 0a.02.12 25 64.39 42.82 2.43 3185 5.80 27.85 315 15.77 8.92 464 0.00 .... . 0.00 .... . 0a.03.20 24 63.44 41.57 2.34 3159 5.87 27.65 303 16.00 8.72 423 0.00 .... . 0.00 .... . 0a.03.14 24 62.74 40.76 2.38 3287 6.08 28.84 310 15.90 9.07 427 0.00 .... . 0.00 .... . 0a.02.5 24 61.81 40.05 2.14 3729 6.02 28.72 311 15.75 8.73 506 0.00 .... . 0.00 .... . 0a.01.2 26 62.73 41.37 2.37 3226 6.11 27.65 335 15.24 8.99 552 0.00 .... . 0.00 .... . 0a.01.3 24 57.66 35.92 2.39 3542 5.82 26.97 338 15.92 8.93 516 0.00 .... . 0.00 .... . 0a.01.10 24 61.45 39.43 2.40 3215 6.18 28.27 315 15.84 8.81 545 0.00 .... . 0.00 .... . 0a.01.23 24 60.92 39.20 2.54 2881 5.90 26.77 344 15.82 9.01 574 0.00 .... . 0.00 .... . 0a.01.15 24 60.35 38.05 2.33 3345 6.02 28.27 305 16.28 8.92 555 0.00 .... . 0.00 .... . 0a.01.21 25 63.16 41.62 2.31 3374 5.65 29.21 306 15.89 8.90 572 0.00 .... . 0.00 .... . 0a.01.13 25 62.38 40.58 2.51 3214 5.88 27.73 338 15.92 9.22 573 0.00 .... . 0.00 .... . 0a.01.8 26 64.69 43.12 2.35 3346 5.80 27.22 330 15.77 9.34 530 0.00 .... . 0.00 .... . 0a.01.12 25 62.34 40.58 2.29 3404 5.92 28.99 316 15.85 9.06 557 0.00 .... . 0.00 .... . /aggr0_sas_600g/plex0/rg2: 0a.02.0 7 20.47 0.00 .... . 7.69 55.57 119 12.78 18.03 178 0.00 .... . 0.00 .... . 0a.03.1 7 20.59 0.00 .... . 7.81 54.77 120 12.78 17.99 163 0.00 .... . 0.00 .... . 0a.01.0 28 66.68 44.17 2.32 3498 6.41 30.22 311 16.10 9.73 506 0.00 .... . 0.00 .... . 0a.02.1 26 64.67 42.92 2.31 3459 6.26 31.12 317 15.49 9.80 449 0.00 .... . 0.00 .... . 0a.03.2 26 65.77 43.57 2.47 3255 6.08 30.55 307 16.12 9.35 422 0.00 .... . 0.00 .... . 0a.01.1 27 67.36 45.53 2.36 3291 6.23 31.81 283 15.60 10.42 418 0.00 .... . 0.00 .... . 0a.02.3 27 66.85 45.00 2.19 3569 6.05 29.12 317 15.80 9.93 425 0.00 .... . 0.00 .... . 0a.03.6 27 70.77 47.82 2.21 3569 6.38 30.96 308 16.57 9.28 410 0.00 .... . 0.00 .... . 0a.01.7 27 65.07 42.47 2.37 3412 6.25 30.77 305 16.35 9.73 489 0.00 .... . 0.00 .... . 0a.02.4 27 65.73 44.65 2.46 3134 5.93 30.35 302 15.15 10.49 414 0.00 .... . 0.00 .... . 0a.03.10 27 66.80 44.37 2.24 3635 6.41 30.98 311 16.02 9.60 410 0.00 .... . 0.00 .... . 0a.01.9 27 66.53 44.56 2.25 3698 6.11 29.33 320 15.85 10.37 502 0.00 .... . 0.00 .... . 0a.02.6 26 64.67 42.16 2.38 3264 6.15 32.52 293 16.37 9.80 452 0.00 .... . 0.00 .... . 0a.03.15 28 68.97 46.26 2.12 3974 6.21 30.80 327 16.50 9.21 487 0.00 .... . 0.00 .... . 0a.01.11 27 67.25 44.28 2.27 3737 6.38 30.34 320 16.58 9.04 607 0.00 .... . 0.00 .... . 0a.02.8 26 65.32 41.81 2.43 3292 6.28 30.20 321 17.23 8.81 517 0.00 .... . 0.00 .... . 0a.03.17 28 67.45 44.95 2.20 3792 6.18 30.23 329 16.32 9.06 494 0.00 .... . 0.00 .... . 0a.01.18 27 66.80 44.78 2.28 3647 6.18 29.31 336 15.84 9.67 557 0.00 .... . 0.00 .... . 0a.02.11 27 64.69 42.47 2.27 3677 6.26 29.47 311 15.95 10.48 463 0.00 .... . 0.00 .... . 0a.03.22 27 65.58 43.82 2.34 3382 5.97 29.82 312 15.80 9.65 477 0.00 .... . 0.00 .... . 0a.01.20 28 67.93 45.13 2.47 3315 6.45 29.24 322 16.35 9.60 588 0.00 .... . 0.00 .... . 0a.02.17 28 67.00 43.77 2.34 3529 6.48 29.63 332 16.75 9.66 516 0.00 .... . 0.00 .... . 0a.03.23 27 70.05 46.68 2.25 3592 6.33 30.59 320 17.05 9.20 509 0.00 .... . 0.00 .... . /aggr0_sas_600g/plex0/rg3: 3b.05.0 6 18.91 0.00 .... . 7.86 56.64 108 11.05 15.07 148 0.00 .... . 0.00 .... . 0a.02.19 6 19.03 0.00 .... . 7.98 55.84 110 11.05 15.07 179 0.00 .... . 0.00 .... . 0a.01.22 30 74.03 51.36 2.46 3262 6.58 35.44 279 16.08 10.13 494 0.00 .... . 0.00 .... . 3b.05.1 31 77.13 54.15 2.37 3481 6.75 37.11 262 16.23 9.43 452 0.00 .... . 0.00 .... . 0a.02.22 31 77.30 54.24 2.49 3373 6.55 35.55 284 16.52 9.43 498 0.00 .... . 0.00 .... . 3b.05.2 30 73.74 50.76 2.35 3684 6.75 36.56 275 16.23 9.47 419 0.00 .... . 0.00 .... . 3b.05.20 31 76.57 54.02 2.50 3139 6.70 35.59 259 15.85 9.89 448 0.00 .... . 0.00 .... . 3b.05.4 30 76.29 54.19 2.34 3523 6.43 37.21 236 15.67 10.22 378 0.00 .... . 0.00 .... . 3b.05.5 29 74.09 52.11 2.32 3478 6.46 36.30 247 15.52 9.83 382 0.00 .... . 0.00 .... . 3b.05.6 30 74.94 53.11 2.60 3120 6.48 34.96 273 15.35 10.44 401 0.00 .... . 0.00 .... . 3b.05.7 30 76.80 54.24 2.51 3188 6.75 34.71 280 15.82 10.31 396 0.00 .... . 0.00 .... . 3b.05.8 31 79.71 56.46 2.40 3347 6.66 35.86 278 16.58 9.35 382 0.00 .... . 0.00 .... . 3b.05.9 30 74.57 52.08 2.41 3424 6.70 35.91 262 15.80 9.59 426 0.00 .... . 0.00 .... . 3b.05.10 30 76.72 54.43 2.52 3285 6.65 35.93 253 15.64 9.97 437 0.00 .... . 0.00 .... . 3b.05.11 32 76.53 54.07 2.38 3536 6.83 33.47 301 15.64 9.76 463 0.00 .... . 0.00 .... . 3b.05.12 31 77.58 55.02 2.42 3403 6.65 35.36 262 15.92 10.24 430 0.00 .... . 0.00 .... . 3b.05.13 31 75.65 52.47 2.35 3424 6.76 35.10 270 16.42 9.69 481 0.00 .... . 0.00 .... . 3b.05.14 31 75.22 52.61 2.53 3314 6.71 37.28 251 15.90 9.64 455 0.00 .... . 0.00 .... . 3b.05.15 30 74.26 51.59 2.52 3221 6.55 36.06 266 16.12 9.73 439 0.00 .... . 0.00 .... . 3b.05.16 32 78.20 55.80 2.37 3604 6.70 36.85 267 15.70 9.54 527 0.00 .... . 0.00 .... . 3b.05.17 31 75.70 53.24 2.42 3446 6.58 36.37 261 15.89 9.55 492 0.00 .... . 0.00 .... . 3b.05.18 31 77.58 55.10 2.35 3422 6.68 35.76 272 15.80 9.65 515 0.00 .... . 0.00 .... . 3b.05.19 30 75.87 53.40 2.57 3261 6.65 36.61 271 15.82 9.57 542 0.00 .... . 0.00 .... .
Aggregate statistics: Minimum 6 18.91 0.00 5.65 11.05 0.00 0.00 Mean 25 62.98 40.64 6.40 15.92 0.00 0.00 Maximum 32 79.71 56.46 9.80 17.68 0.00 0.00
Alon Zeltser Solution Architect ,E&M Computing
Tel: 972-35766966 (227) | Mobile: 972-544770108
alonz@emet.co.il| www.emet.co.il
emet_logo
