Hi all,
We are having a problem with a FAS270 that appears to be having high cpu utilisation issues.
To be honest I have looked at sysstat and statit and I cannot see the problem. I would appreciate if someone could take a look at the below outputs and let me know what I'm missing.
Thanks
Matt
ukfiler*> sysstat -i 3 CPU NFS CIFS iSCSI Net kB/s Disk kB/s iSCSI kB/s Cache in out read write in out age 85% 18 0 26 203 13 661 0 91 0 24 68% 9 1 50 283 23 692 0 221 9 24 92% 7 4 25 144 11 1643 2443 90 3 24 78% 12 1 74 345 21 618 0 198 0 24 67% 8 0 45 288 18 722 0 205 5 24 96% 13 1 32 236 14 1365 1708 151 1 24 81% 8 4 39 222 16 843 867 167 4 24 85% 8 0 33 443 10 1212 1456 118 0 24 99% 16 2 43 370 21 912 0 168 4 24 99% 14 16 23 219 26 1023 0 98 0 24 99% 12 0 36 585 39 2452 3781 465 18 24 ukfiler*>
Hostname: ukfiler ID: 0084264545 Memory: 1022 MB NetApp Release 7.2.3: Thu Jul 5 10:06:16 PDT 2007 <6O> Start time: Wed Nov 5 22:45:03 GMT 2008
CPU Statistics 64.822501 time (seconds) 100 % 52.221602 system time 81 % 0.511694 rupt time 1 % (161581 rupts x 3 usec/rupt) 51.709908 non-rupt system time 80 % 77.423400 idle time 119 %
4.011144 time in CP 6 % 100 % 0.052061 rupt time in CP 1 % (12144 rupts x 4 usec/rupt)
Multiprocessor Statistics (per second) cpu0 cpu1 total sk switches 6581.74 7585.82 14167.57 hard switches 1485.87 1560.68 3046.55 domain switches 72.75 65.01 137.76 CP rupts 125.45 61.89 187.34 nonCP rupts 1350.03 955.30 2305.33 IPI rupts 18.23 17.32 35.56
CP rupt usec 565.04 238.10 803.13 nonCP rupt usec 4342.69 2747.95 7090.64 idle 632999.86 561391.13 1194390.83 kahuna 334624.38 409797.24 744421.77 network 6697.98 8506.20 15204.18 storage 4193.90 5622.94 9816.84 exempt 3941.24 4036.10 7977.34 raid 7872.95 4627.33 12500.27 target 4409.87 2761.70 7171.57 netcache 0.00 0.00 0.00 netcache2 0.00 0.00 0.00 cifs 352.07 271.33 623.39 wafl_exempt 0.00 0.00 0.00
49.342358 seconds with one or more CPUs active ( 76%)
46.750469 seconds with one CPU active ( 72%) 2.591888 seconds with both CPUs active ( 4%)
Domain Utilization By Exempt (per second) 0.00 idle 0.00 kahuna 0.00 network 0.00 storage 0.00 exempt 0.00 raid 0.00 target 0.00 netcache 0.00 netcache2 0.00 cifs 0.00 wafl_exempt
CSMP Domain Switches (per second) From\To idle kahuna network storage exempt raid target netcache netcache2 cifs wafl_exem pt idle 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0 kahuna 0.00 0.00 27.09 2.65 0.00 5.32 1.87 0.00 0.00 18.79 0.0 0 network 0.00 27.09 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.20 0.0 0 storage 0.00 2.65 0.00 0.00 0.00 12.96 0.00 0.00 0.00 0.00 0.0 0 exempt 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0 raid 0.00 5.32 0.00 12.96 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0 target 0.00 1.87 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0 netcache 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0 netcache2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0 cifs 0.00 18.79 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0 wafl_exempt 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0. 00
Miscellaneous Statistics (per second) 3046.55 hard context switches 10.38 NFS operations 1.42 CIFS operations 0.00 HTTP operations 0.00 NetCache URLs 0.00 streaming packets 1240.42 network KB received 100.32 network KB transmitted 655.39 disk KB read 1146.02 disk KB written 648.90 NVRAM KB written 0.00 nolog KB written 8.27 WAFL bufs given to clients 0.00 checksum cache hits ( 0%) 1.23 no checksum - partial buffer 0.00 FCP operations 74.48 iSCSI operations
WAFL Statistics (per second) 15.94 name cache hits ( 95%) 0.86 name cache misses ( 5%) 529973.33 buf hash hits ( 100%) 1710.32 buf hash misses ( 0%) 516.44 inode cache hits ( 100%) 0.06 inode cache misses ( 0%) 262496.95 buf cache hits ( 100%) 19.65 buf cache misses ( 0%) 30.17 blocks read 67.00 blocks read-ahead 58.10 chains read-ahead 0.99 dummy reads 14.25 blocks speculative read-ahead 226.31 blocks written 2.27 stripes written 0.00 blocks over-written 0.09 wafl_timer generated CP 0.00 snapshot generated CP 0.00 wafl_avail_bufs generated CP 0.00 dirty_blk_cnt generated CP 0.00 full NV-log generated CP 0.00 back-to-back CP 0.00 flush generated CP 0.00 sync generated CP 0.00 wafl_avail_vbufs generated CP 0.00 deferred back-to-back CP 0.00 low datavecs generated CP 304.72 non-restart messages 4.98 IOWAIT suspends 133872 buffers
RAID Statistics (per second) 40.91 xors 0.00 long dispatches [0] 0.00 long consumed [0] 0.00 long consumed hipri [0] 0.00 long low priority [0] 0.00 long high priority [0] 0.00 long monitor tics [0] 0.00 long monitor clears [0] 0.00 long dispatches [1] 0.00 long consumed [1] 0.00 long consumed hipri [1] 0.00 long low priority [1] 0.00 long high priority [1] 0.00 long monitor tics [1] 0.00 long monitor clears [1] 18 max batch 2.11 blocked mode xor 19.68 timed mode xor 0.05 fast adjustments 0.05 slow adjustments 0 avg batch start 0 avg stripe/msec 2.64 tetrises written 0.00 master tetrises 0.00 slave tetrises 29.70 stripes written 11.22 partial stripes 18.48 full stripes 226.68 blocks written 31.56 blocks read 1.13 1 blocks per stripe size 6 0.29 2 blocks per stripe size 6 0.17 3 blocks per stripe size 6 0.35 4 blocks per stripe size 6 1.80 5 blocks per stripe size 6 15.63 6 blocks per stripe size 6 1.11 1 blocks per stripe size 16 0.31 2 blocks per stripe size 16 0.32 3 blocks per stripe size 16 0.19 4 blocks per stripe size 16 0.09 5 blocks per stripe size 16 0.20 6 blocks per stripe size 16 0.05 7 blocks per stripe size 16 0.20 8 blocks per stripe size 16 0.12 9 blocks per stripe size 16 0.23 10 blocks per stripe size 16 0.23 11 blocks per stripe size 16 0.56 12 blocks per stripe size 16 0.83 13 blocks per stripe size 16 1.26 14 blocks per stripe size 16 1.76 15 blocks per stripe size 16 2.85 16 blocks per stripe size 16
Network Interface Statistics (per second) iface side bytes packets multicasts errors collisions pkt drops e0a recv 114768.96 92.82 0.00 0.00 0.00 xmit 193.85 1.43 0.05 0.00 0.00 e0b recv 521347.10 383.35 0.00 0.00 0.00 xmit 53015.11 280.33 0.14 0.00 0.00 vh recv 0.00 0.00 0.00 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00 mtrunk1 recv 636116.06 476.18 0.00 0.00 0.00 xmit 53208.96 281.77 0.25 0.00 0.00 mtrunk1-1 recv 588.41 5.26 2.10 0.00 0.00 xmit 665.54 2.92 0.08 0.00 0.00 mtrunk1-21 recv 261620.02 204.87 0.00 0.00 0.00 xmit 22000.14 120.08 0.05 0.00 0.00 mtrunk1-22 recv 0.00 0.00 0.00 0.00 0.00 xmit 5.28 0.05 0.05 0.00 0.00 mtrunk1-23 recv 371871.00 264.20 0.02 0.00 0.00 xmit 26855.71 156.13 0.05 0.00 0.00
Disk Statistics (per second) ut% is the percent of time the disk was busy. xfers is the number of data-transfer commands issued per second. xfers = ureads + writes + cpreads + greads + gwrites chain is the average number of 4K blocks per command. usecs is the average disk round-trip time per 4K block.
disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs /aggr2/plex0/rg0: 0b.45 1 1.97 0.19 1.00 12333 1.10 9.42 988 0.69 5.07 768 0.00 .... . 0.00 .... . 0b.44 2 2.16 0.19 1.00 38167 1.28 8.35 925 0.69 5.07 838 0.00 .... . 0.00 .... . 0b.43 5 6.03 4.29 1.04 14107 1.13 6.84 1489 0.62 2.85 2430 0.00 .... . 0.00 .... . 0b.42 4 5.20 3.72 1.05 15565 0.88 8.00 1197 0.60 5.03 939 0.00 .... . 0.00 .... . 0b.41 4 5.09 3.76 1.01 14919 0.88 8.98 1184 0.45 4.03 1325 0.00 .... . 0.00 .... . 0b.40 5 5.28 4.09 1.08 13686 0.74 9.42 1210 0.45 5.28 1477 0.00 .... . 0.00 .... . 0b.39 5 5.54 4.18 1.08 15908 0.83 9.63 1142 0.52 4.44 1788 0.00 .... . 0.00 .... . 0b.38 4 5.45 3.90 1.00 17528 0.91 8.51 1327 0.63 3.56 1349 0.00 .... . 0.00 .... . 0b.37 4 5.29 3.72 1.07 16442 0.97 7.95 1359 0.60 4.23 1509 0.00 .... . 0.00 .... . 0b.36 4 5.01 3.50 1.01 13113 0.89 8.66 1371 0.62 4.18 1413 0.00 .... . 0.00 .... . 0b.32 5 5.54 4.03 1.36 11093 0.91 8.22 1433 0.60 3.92 1255 0.00 .... . 0.00 .... . 0b.17 4 5.06 3.76 1.01 13752 0.83 9.11 1268 0.46 4.60 1254 0.00 .... . 0.00 .... . 0b.33 5 5.51 3.90 1.02 14456 0.88 7.89 1553 0.73 4.02 1280 0.00 .... . 0.00 .... . 0b.18 4 5.51 3.79 1.19 12478 0.97 7.76 1591 0.74 3.71 1472 0.00 .... . 0.00 .... . 0b.34 5 5.51 3.89 1.12 13188 0.89 8.95 1301 0.73 3.62 1653 0.00 .... . 0.00 .... . 0b.19 4 5.04 3.56 1.07 11947 0.83 8.94 1308 0.65 3.50 2238 0.00 .... . 0.00 .... . 0b.35 4 4.74 3.33 1.14 12020 0.80 9.21 1353 0.60 4.26 1416 0.00 .... . 0.00 .... . 0b.20 4 5.32 3.89 1.01 13341 0.76 9.27 1352 0.68 3.86 1729 0.00 .... . 0.00 .... . /aggr1/plex0/rg0: 0b.24 2 2.39 0.19 1.00 13667 1.73 11.25 655 0.48 5.06 1140 0.00 .... . 0.00 .... . 0b.29 5 7.30 4.97 1.26 17742 1.71 10.48 666 0.62 3.45 1543 0.00 .... . 0.00 .... . 0b.26 4 6.34 4.37 1.28 13500 1.42 12.46 595 0.56 3.97 1427 0.00 .... . 0.00 .... . 0b.28 5 7.05 4.95 1.14 19218 1.50 11.82 695 0.60 3.44 1642 0.00 .... . 0.00 .... . 0b.27 7 6.20 4.26 1.05 19422 1.45 12.28 659 0.49 4.56 1000 0.00 .... . 0.00 .... . 0b.22 5 6.90 4.81 1.21 18814 1.53 11.51 835 0.56 3.81 1591 0.00 .... . 0.00 .... . 0b.21 4 6.57 4.37 1.11 18990 1.51 11.73 843 0.69 3.80 1655 0.00 .... . 0.00 .... . 0b.23 2 2.36 0.19 1.00 9667 1.70 11.42 677 0.48 5.06 1178 0.00 .... . 0.00 .... .
Aggregate statistics: Minimum 1 1.97 0.19 0.74 0.45 0.00 0.00 Mean 4 5.15 3.44 1.11 0.59 0.00 0.00 Maximum 7 7.30 4.97 1.73 0.74 0.00 0.00
Spares and other disks: 0b.25 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
Spares and other disks: 0b.16 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
FCP Statistics (per second) 0.00 FCP Bytes recv 0.00 FCP Bytes sent 0.00 FCP ops
iSCSI Statistics (per second) 516537.68 iSCSI Bytes recv 22866.13 iSCSI Bytes xmit 74.48 iSCSI ops
Interrupt Statistics (per second) 1604.06 Clock (IRQ 5) 35.51 IPI (IRQ 8) 35.51 IPI (IRQ 9) 160.84 FCAL (IRQ 23) 259.37 SB MAC (IRQ 25) 41.22 SB MAC (IRQ 26) 35.56 IPI 2172.07 total
Data Mover Statistics (per second) 627.05 total dma transfer KB 408.55 dma descriptors 82.58 dma desc wait count 753.86 total dma transfer KB 523.52 dma descriptors 112.63 dma desc wait count 0.00 total dma transfer KB 0.00 dma descriptors 0.00 dma desc wait count 0.00 total dma transfer KB 0.00 dma descriptors 0.00 dma desc wait count
____________________________ Matt Davies Director of International IT Operations General Atlantic 83 Pall Mall London SW1Y 5ES
Tel: +44 207 484 3203 Fax: +44 207 484 2803 Mobile: +44 777 559 4265 ____________________________
_____________________________________________________________ This e-mail (including all attachments) is confidential and may be privileged. It is for the exclusive use of the addressee only. If you are not the addressee, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please erase all copies of the message and its attachments and notify us immediately at help@generalatlantic.com mailto:help@generalatlantic.com. Thank You.