Nils,
1, yes it is both a source and destination for snapmirror, there is no snapvault
2, see below.
ukfiler*> wafl scan status Volume CIFS_UK: Scan id Type of scan progress 3 active bitmap rearrangement fbn 1605 of 2230 w/ max_chain_len 3 191730 container block reclamation block 2042 of 2230 Volume root: Scan id Type of scan progress 5 active bitmap rearrangement fbn 63 of 107 w/ max_chain_len 3 191636 container block reclamation block 64 of 107 Volume UKEXCHDB: Scan id Type of scan progress 8 active bitmap rearrangement fbn 138 of 713 w/ max_chain_len 3 Volume UKEXCHLOGS: Scan id Type of scan progress 10 active bitmap rearrangement fbn 369 of 401 w/ max_chain_len 3 191608 container block reclamation block 391 of 401 Volume UKVMWARE: Scan id Type of scan progress 12 active bitmap rearrangement fbn 1538 of 4461 w/ max_chain_len 3 Volume DUEXCHLOGS: Scan id Type of scan progress 191713 container block reclamation block 879 of 892 Volume UKVIRTUALCENTRE: Scan id Type of scan progress 16 active bitmap rearrangement fbn 197 of 1115 w/ max_chain_len 3 Volume UKVMWARE1: Scan id Type of scan progress 18 active bitmap rearrangement fbn 4215 of 4461 w/ max_chain_len 3 Volume UKVMWARENFS: Scan id Type of scan progress 21 active bitmap rearrangement fbn 1019 of 1784 w/ max_chain_len 3 Volume TESTUKEXCHLOGS: Scan id Type of scan progress 191708 container block reclamation block 393 of 401 Volume UKEXCH3DB: Scan id Type of scan progress 152073 active bitmap rearrangement fbn 548 of 571 w/ max_chain_len 3 191729 container block reclamation block 356 of 571 Volume vol1: Scan id Type of scan progress 191630 container block reclamation block 0 of 0 Volume UKEXCH3LOGS: Scan id Type of scan progress 152075 active bitmap rearrangement fbn 29 of 321 w/ max_chain_len 3 191728 container block reclamation block 195 of 321
Thanks
Matt
-----Original Message----- From: Nils Vogels [mailto:bacardicoke@gmail.com] Sent: 06 November 2008 09:29 To: Davies,Matt Subject: Re: High CPU utilisation
Couple of questions ..
1) Do you have snapvault/snapmirror running? 2) What does "wafl scan status" tell you?
Greetings,
Nils
On Thu, Nov 6, 2008 at 10:09 AM, Davies,Matt MDAVIES@generalatlantic.com wrote:
Hi all,
We are having a problem with a FAS270 that appears to be having high
cpu utilisation issues.
To be honest I have looked at sysstat and statit and I cannot see the
problem. I would appreciate if someone could take a look at the below outputs and let me know what I'm missing.
Thanks
Matt
ukfiler*> sysstat -i 3 CPU NFS CIFS iSCSI Net kB/s Disk kB/s iSCSI kB/s
Cache
in out read write in out
age
85% 18 0 26 203 13 661 0 91 0
24
68% 9 1 50 283 23 692 0 221 9
24
92% 7 4 25 144 11 1643 2443 90 3
24
78% 12 1 74 345 21 618 0 198 0
24
67% 8 0 45 288 18 722 0 205 5
24
96% 13 1 32 236 14 1365 1708 151 1
24
81% 8 4 39 222 16 843 867 167 4
24
85% 8 0 33 443 10 1212 1456 118 0
24
99% 16 2 43 370 21 912 0 168 4
24
99% 14 16 23 219 26 1023 0 98 0
24
99% 12 0 36 585 39 2452 3781 465 18
24
ukfiler*>
Hostname: ukfiler ID: 0084264545 Memory: 1022 MB NetApp Release 7.2.3: Thu Jul 5 10:06:16 PDT 2007 <6O> Start time: Wed Nov 5 22:45:03 GMT 2008
CPU Statistics 64.822501 time (seconds) 100 % 52.221602 system time 81 % 0.511694 rupt time 1 % (161581 rupts x 3
usec/rupt)
51.709908 non-rupt system time 80 % 77.423400 idle time 119 % 4.011144 time in CP 6 % 100 % 0.052061 rupt time in CP 1 % (12144 rupts x 4
usec/rupt)
Multiprocessor Statistics (per second) cpu0 cpu1 totalsk switches 6581.74 7585.82 14167.57 hard switches 1485.87 1560.68 3046.55 domain switches 72.75 65.01 137.76 CP rupts 125.45 61.89 187.34 nonCP rupts 1350.03 955.30 2305.33 IPI rupts 18.23 17.32 35.56
CP rupt usec 565.04 238.10 803.13 nonCP rupt usec 4342.69 2747.95 7090.64 idle 632999.86 561391.13 1194390.83 kahuna 334624.38 409797.24 744421.77 network 6697.98 8506.20 15204.18 storage 4193.90 5622.94 9816.84 exempt 3941.24 4036.10 7977.34 raid 7872.95 4627.33 12500.27 target 4409.87 2761.70 7171.57 netcache 0.00 0.00 0.00 netcache2 0.00 0.00 0.00 cifs 352.07 271.33 623.39 wafl_exempt 0.00 0.00 0.00
49.342358 seconds with one or more CPUs active ( 76%) 46.750469 seconds with one CPU active ( 72%) 2.591888 seconds with both CPUs active ( 4%) Domain Utilization By Exempt (per second) 0.00 idle 0.00 kahuna 0.00 network 0.00 storage 0.00 exempt 0.00 raid 0.00 target 0.00 netcache 0.00 netcache2 0.00 cifs 0.00 wafl_exempt CSMP Domain Switches (per second)From\To idle kahuna network storage exempt
raid target netcache netcache2 cifs wafl_exem
pt idle 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.0
0 kahuna 0.00 0.00 27.09 2.65 0.00
5.32 1.87 0.00 0.00 18.79 0.0
0 network 0.00 27.09 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.20 0.0
0 storage 0.00 2.65 0.00 0.00 0.00
12.96 0.00 0.00 0.00 0.00 0.0
0 exempt 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.0
0 raid 0.00 5.32 0.00 12.96 0.00
0.00 0.00 0.00 0.00 0.00 0.0
0 target 0.00 1.87 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.0
0 netcache 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.0
0 netcache2 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.0
0 cifs 0.00 18.79 0.20 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.0
0 wafl_exempt 0.00 0.00 0.00 0.00 0.00
0.00 0.00 0.00 0.00 0.00 0.
00
Miscellaneous Statistics (per second)3046.55 hard context switches 10.38 NFS operations 1.42 CIFS operations 0.00 HTTP operations 0.00 NetCache URLs 0.00 streaming packets 1240.42 network KB received 100.32 network KB
transmitted
655.39 disk KB read 1146.02 disk KB written 648.90 NVRAM KB written 0.00 nolog KB written 8.27 WAFL bufs given to clients 0.00 checksum cache hits
( 0%)
1.23 no checksum - partial buffer 0.00 FCP operations 74.48 iSCSI operations WAFL Statistics (per second) 15.94 name cache hits ( 95%) 0.86 name cache misses
( 5%)
529973.33 buf hash hits ( 100%) 1710.32 buf hash misses
( 0%)
516.44 inode cache hits ( 100%) 0.06 inode cache misses
( 0%)
262496.95 buf cache hits ( 100%) 19.65 buf cache misses
( 0%)
30.17 blocks read 67.00 blocks read-ahead 58.10 chains read-ahead 0.99 dummy reads 14.25 blocks speculative read-ahead 226.31 blocks written 2.27 stripes written 0.00 blocks over-written 0.09 wafl_timer generated CP 0.00 snapshot generated
CP
0.00 wafl_avail_bufs generated CP 0.00 dirty_blk_cnt
generated CP
0.00 full NV-log generated CP 0.00 back-to-back CP 0.00 flush generated CP 0.00 sync generated CP 0.00 wafl_avail_vbufs generated CP 0.00 deferred
back-to-back CP
0.00 low datavecs generated CP 304.72 non-restart messages 4.98 IOWAIT suspends 133872 buffers RAID Statistics (per second) 40.91 xors 0.00 long dispatches [0] 0.00 long consumed [0] 0.00 long consumed hipri
[0]
0.00 long low priority [0] 0.00 long high priority
[0]
0.00 long monitor tics [0] 0.00 long monitor clears
[0]
0.00 long dispatches [1] 0.00 long consumed [1] 0.00 long consumed hipri [1] 0.00 long low priority
[1]
0.00 long high priority [1] 0.00 long monitor tics
[1]
0.00 long monitor clears [1] 18 max batch 2.11 blocked mode xor 19.68 timed mode xor 0.05 fast adjustments 0.05 slow adjustments 0 avg batch start 0 avg stripe/msec 2.64 tetrises written 0.00 master tetrises 0.00 slave tetrises 29.70 stripes written 11.22 partial stripes 18.48 full stripes 226.68 blocks written 31.56 blocks read 1.13 1 blocks per stripe size 6 0.29 2 blocks per stripe
size 6
0.17 3 blocks per stripe size 6 0.35 4 blocks per stripe
size 6
1.80 5 blocks per stripe size 6 15.63 6 blocks per stripe
size 6
1.11 1 blocks per stripe size 16 0.31 2 blocks per stripe
size 16
0.32 3 blocks per stripe size 16 0.19 4 blocks per stripe
size 16
0.09 5 blocks per stripe size 16 0.20 6 blocks per stripe
size 16
0.05 7 blocks per stripe size 16 0.20 8 blocks per stripe
size 16
0.12 9 blocks per stripe size 16 0.23 10 blocks per stripe
size 16
0.23 11 blocks per stripe size 16 0.56 12 blocks per stripe
size 16
0.83 13 blocks per stripe size 16 1.26 14 blocks per stripe
size 16
1.76 15 blocks per stripe size 16 2.85 16 blocks per stripe
size 16
Network Interface Statistics (per second)iface side bytes packets multicasts errors collisions
pkt drops
e0a recv 114768.96 92.82 0.00 0.00
0.00
xmit 193.85 1.43 0.05 0.00 0.00e0b recv 521347.10 383.35 0.00 0.00
0.00
xmit 53015.11 280.33 0.14 0.00 0.00vh recv 0.00 0.00 0.00 0.00
0.00
xmit 0.00 0.00 0.00 0.00 0.00mtrunk1 recv 636116.06 476.18 0.00 0.00
0.00
xmit 53208.96 281.77 0.25 0.00 0.00mtrunk1-1 recv 588.41 5.26 2.10 0.00
0.00
xmit 665.54 2.92 0.08 0.00 0.00mtrunk1-21 recv 261620.02 204.87 0.00 0.00
0.00
xmit 22000.14 120.08 0.05 0.00 0.00mtrunk1-22 recv 0.00 0.00 0.00 0.00
0.00
xmit 5.28 0.05 0.05 0.00 0.00mtrunk1-23 recv 371871.00 264.20 0.02 0.00
0.00
xmit 26855.71 156.13 0.05 0.00 0.00 Disk Statistics (per second) ut% is the percent of time the disk was busy. xfers is the number of data-transfer commands issued per
second.
xfers = ureads + writes + cpreads + greads + gwrites chain is the average number of 4K blocks per command. usecs is the average disk round-trip time per 4K block.disk ut% xfers ureads--chain-usecs writes--chain-usecs
cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs
/aggr2/plex0/rg0: 0b.45 1 1.97 0.19 1.00 12333 1.10 9.42 988
0.69 5.07 768 0.00 .... . 0.00 .... .
0b.44 2 2.16 0.19 1.00 38167 1.28 8.35 925
0.69 5.07 838 0.00 .... . 0.00 .... .
0b.43 5 6.03 4.29 1.04 14107 1.13 6.84 1489
0.62 2.85 2430 0.00 .... . 0.00 .... .
0b.42 4 5.20 3.72 1.05 15565 0.88 8.00 1197
0.60 5.03 939 0.00 .... . 0.00 .... .
0b.41 4 5.09 3.76 1.01 14919 0.88 8.98 1184
0.45 4.03 1325 0.00 .... . 0.00 .... .
0b.40 5 5.28 4.09 1.08 13686 0.74 9.42 1210
0.45 5.28 1477 0.00 .... . 0.00 .... .
0b.39 5 5.54 4.18 1.08 15908 0.83 9.63 1142
0.52 4.44 1788 0.00 .... . 0.00 .... .
0b.38 4 5.45 3.90 1.00 17528 0.91 8.51 1327
0.63 3.56 1349 0.00 .... . 0.00 .... .
0b.37 4 5.29 3.72 1.07 16442 0.97 7.95 1359
0.60 4.23 1509 0.00 .... . 0.00 .... .
0b.36 4 5.01 3.50 1.01 13113 0.89 8.66 1371
0.62 4.18 1413 0.00 .... . 0.00 .... .
0b.32 5 5.54 4.03 1.36 11093 0.91 8.22 1433
0.60 3.92 1255 0.00 .... . 0.00 .... .
0b.17 4 5.06 3.76 1.01 13752 0.83 9.11 1268
0.46 4.60 1254 0.00 .... . 0.00 .... .
0b.33 5 5.51 3.90 1.02 14456 0.88 7.89 1553
0.73 4.02 1280 0.00 .... . 0.00 .... .
0b.18 4 5.51 3.79 1.19 12478 0.97 7.76 1591
0.74 3.71 1472 0.00 .... . 0.00 .... .
0b.34 5 5.51 3.89 1.12 13188 0.89 8.95 1301
0.73 3.62 1653 0.00 .... . 0.00 .... .
0b.19 4 5.04 3.56 1.07 11947 0.83 8.94 1308
0.65 3.50 2238 0.00 .... . 0.00 .... .
0b.35 4 4.74 3.33 1.14 12020 0.80 9.21 1353
0.60 4.26 1416 0.00 .... . 0.00 .... .
0b.20 4 5.32 3.89 1.01 13341 0.76 9.27 1352
0.68 3.86 1729 0.00 .... . 0.00 .... .
/aggr1/plex0/rg0: 0b.24 2 2.39 0.19 1.00 13667 1.73 11.25 655
0.48 5.06 1140 0.00 .... . 0.00 .... .
0b.29 5 7.30 4.97 1.26 17742 1.71 10.48 666
0.62 3.45 1543 0.00 .... . 0.00 .... .
0b.26 4 6.34 4.37 1.28 13500 1.42 12.46 595
0.56 3.97 1427 0.00 .... . 0.00 .... .
0b.28 5 7.05 4.95 1.14 19218 1.50 11.82 695
0.60 3.44 1642 0.00 .... . 0.00 .... .
0b.27 7 6.20 4.26 1.05 19422 1.45 12.28 659
0.49 4.56 1000 0.00 .... . 0.00 .... .
0b.22 5 6.90 4.81 1.21 18814 1.53 11.51 835
0.56 3.81 1591 0.00 .... . 0.00 .... .
0b.21 4 6.57 4.37 1.11 18990 1.51 11.73 843
0.69 3.80 1655 0.00 .... . 0.00 .... .
0b.23 2 2.36 0.19 1.00 9667 1.70 11.42 677
0.48 5.06 1178 0.00 .... . 0.00 .... .
Aggregate statistics: Minimum 1 1.97 0.19 0.74
0.45 0.00 0.00
Mean 4 5.15 3.44 1.11
0.59 0.00 0.00
Maximum 7 7.30 4.97 1.73
0.74 0.00 0.00
Spares and other disks: 0b.25 0 0.00 0.00 .... . 0.00 .... .
0.00 .... . 0.00 .... . 0.00 .... .
Spares and other disks: 0b.16 0 0.00 0.00 .... . 0.00 .... .
0.00 .... . 0.00 .... . 0.00 .... .
FCP Statistics (per second) 0.00 FCP Bytes recv 0.00 FCP Bytes sent 0.00 FCP ops iSCSI Statistics (per second)516537.68 iSCSI Bytes recv 22866.13 iSCSI Bytes xmit 74.48 iSCSI ops
Interrupt Statistics (per second)1604.06 Clock (IRQ 5) 35.51 IPI (IRQ 8) 35.51 IPI (IRQ 9) 160.84 FCAL (IRQ 23) 259.37 SB MAC (IRQ 25) 41.22 SB MAC (IRQ 26) 35.56 IPI 2172.07 total
Data Mover Statistics (per second) 627.05 total dma transfer KB 408.55 dma descriptors 82.58 dma desc wait count 753.86 total dma transfer
KB
523.52 dma descriptors 112.63 dma desc wait count 0.00 total dma transfer KB 0.00 dma descriptors 0.00 dma desc wait count 0.00 total dma transfer
KB
0.00 dma descriptors 0.00 dma desc wait count
Matt Davies Director of International IT Operations General Atlantic 83 Pall Mall London SW1Y 5ES
Tel: +44 207 484 3203 Fax: +44 207 484 2803 Mobile: +44 777 559 4265 ____________________________
This e-mail (including all attachments) is confidential and may be
privileged.
It is for the exclusive use of the addressee only. If you are not the
addressee,
you are hereby notified that any dissemination of this communication
is strictly
prohibited. If you have received this communication in error, please
erase all
copies of the message and its attachments and notify us immediately at help@generalatlantic.com mailto:help@generalatlantic.com. Thank You.
-
Davies,Matt