toasters November 2008

toasters@lists.teaparty.net

54 participants
58 discussions

High CPU utilisation
by Davies,Matt 06 Nov '08

06 Nov '08

Hi all, We are having a problem with a FAS270 that appears to be having high cpu utilisation issues. To be honest I have looked at sysstat and statit and I cannot see the problem. I would appreciate if someone could take a look at the below outputs and let me know what I'm missing. Thanks Matt ukfiler*> sysstat -i 3 CPU NFS CIFS iSCSI Net kB/s Disk kB/s iSCSI kB/s Cache in out read write in out age 85% 18 0 26 203 13 661 0 91 0 24 68% 9 1 50 283 23 692 0 221 9 24 92% 7 4 25 144 11 1643 2443 90 3 24 78% 12 1 74 345 21 618 0 198 0 24 67% 8 0 45 288 18 722 0 205 5 24 96% 13 1 32 236 14 1365 1708 151 1 24 81% 8 4 39 222 16 843 867 167 4 24 85% 8 0 33 443 10 1212 1456 118 0 24 99% 16 2 43 370 21 912 0 168 4 24 99% 14 16 23 219 26 1023 0 98 0 24 99% 12 0 36 585 39 2452 3781 465 18 24 ukfiler*> Hostname: ukfiler ID: 0084264545 Memory: 1022 MB NetApp Release 7.2.3: Thu Jul 5 10:06:16 PDT 2007 <6O> Start time: Wed Nov 5 22:45:03 GMT 2008 CPU Statistics 64.822501 time (seconds) 100 % 52.221602 system time 81 % 0.511694 rupt time 1 % (161581 rupts x 3 usec/rupt) 51.709908 non-rupt system time 80 % 77.423400 idle time 119 % 4.011144 time in CP 6 % 100 % 0.052061 rupt time in CP 1 % (12144 rupts x 4 usec/rupt) Multiprocessor Statistics (per second) cpu0 cpu1 total sk switches 6581.74 7585.82 14167.57 hard switches 1485.87 1560.68 3046.55 domain switches 72.75 65.01 137.76 CP rupts 125.45 61.89 187.34 nonCP rupts 1350.03 955.30 2305.33 IPI rupts 18.23 17.32 35.56 CP rupt usec 565.04 238.10 803.13 nonCP rupt usec 4342.69 2747.95 7090.64 idle 632999.86 561391.13 1194390.83 kahuna 334624.38 409797.24 744421.77 network 6697.98 8506.20 15204.18 storage 4193.90 5622.94 9816.84 exempt 3941.24 4036.10 7977.34 raid 7872.95 4627.33 12500.27 target 4409.87 2761.70 7171.57 netcache 0.00 0.00 0.00 netcache2 0.00 0.00 0.00 cifs 352.07 271.33 623.39 wafl_exempt 0.00 0.00 0.00 49.342358 seconds with one or more CPUs active ( 76%) 46.750469 seconds with one CPU active ( 72%) 2.591888 seconds with both CPUs active ( 4%) Domain Utilization By Exempt (per second) 0.00 idle 0.00 kahuna 0.00 network 0.00 storage 0.00 exempt 0.00 raid 0.00 target 0.00 netcache 0.00 netcache2 0.00 cifs 0.00 wafl_exempt CSMP Domain Switches (per second) From\To idle kahuna network storage exempt raid target netcache netcache2 cifs wafl_exem pt idle 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0 kahuna 0.00 0.00 27.09 2.65 0.00 5.32 1.87 0.00 0.00 18.79 0.0 0 network 0.00 27.09 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.20 0.0 0 storage 0.00 2.65 0.00 0.00 0.00 12.96 0.00 0.00 0.00 0.00 0.0 0 exempt 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0 raid 0.00 5.32 0.00 12.96 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0 target 0.00 1.87 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0 netcache 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0 netcache2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0 cifs 0.00 18.79 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 0 wafl_exempt 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0. 00 Miscellaneous Statistics (per second) 3046.55 hard context switches 10.38 NFS operations 1.42 CIFS operations 0.00 HTTP operations 0.00 NetCache URLs 0.00 streaming packets 1240.42 network KB received 100.32 network KB transmitted 655.39 disk KB read 1146.02 disk KB written 648.90 NVRAM KB written 0.00 nolog KB written 8.27 WAFL bufs given to clients 0.00 checksum cache hits ( 0%) 1.23 no checksum - partial buffer 0.00 FCP operations 74.48 iSCSI operations WAFL Statistics (per second) 15.94 name cache hits ( 95%) 0.86 name cache misses ( 5%) 529973.33 buf hash hits ( 100%) 1710.32 buf hash misses ( 0%) 516.44 inode cache hits ( 100%) 0.06 inode cache misses ( 0%) 262496.95 buf cache hits ( 100%) 19.65 buf cache misses ( 0%) 30.17 blocks read 67.00 blocks read-ahead 58.10 chains read-ahead 0.99 dummy reads 14.25 blocks speculative read-ahead 226.31 blocks written 2.27 stripes written 0.00 blocks over-written 0.09 wafl_timer generated CP 0.00 snapshot generated CP 0.00 wafl_avail_bufs generated CP 0.00 dirty_blk_cnt generated CP 0.00 full NV-log generated CP 0.00 back-to-back CP 0.00 flush generated CP 0.00 sync generated CP 0.00 wafl_avail_vbufs generated CP 0.00 deferred back-to-back CP 0.00 low datavecs generated CP 304.72 non-restart messages 4.98 IOWAIT suspends 133872 buffers RAID Statistics (per second) 40.91 xors 0.00 long dispatches [0] 0.00 long consumed [0] 0.00 long consumed hipri [0] 0.00 long low priority [0] 0.00 long high priority [0] 0.00 long monitor tics [0] 0.00 long monitor clears [0] 0.00 long dispatches [1] 0.00 long consumed [1] 0.00 long consumed hipri [1] 0.00 long low priority [1] 0.00 long high priority [1] 0.00 long monitor tics [1] 0.00 long monitor clears [1] 18 max batch 2.11 blocked mode xor 19.68 timed mode xor 0.05 fast adjustments 0.05 slow adjustments 0 avg batch start 0 avg stripe/msec 2.64 tetrises written 0.00 master tetrises 0.00 slave tetrises 29.70 stripes written 11.22 partial stripes 18.48 full stripes 226.68 blocks written 31.56 blocks read 1.13 1 blocks per stripe size 6 0.29 2 blocks per stripe size 6 0.17 3 blocks per stripe size 6 0.35 4 blocks per stripe size 6 1.80 5 blocks per stripe size 6 15.63 6 blocks per stripe size 6 1.11 1 blocks per stripe size 16 0.31 2 blocks per stripe size 16 0.32 3 blocks per stripe size 16 0.19 4 blocks per stripe size 16 0.09 5 blocks per stripe size 16 0.20 6 blocks per stripe size 16 0.05 7 blocks per stripe size 16 0.20 8 blocks per stripe size 16 0.12 9 blocks per stripe size 16 0.23 10 blocks per stripe size 16 0.23 11 blocks per stripe size 16 0.56 12 blocks per stripe size 16 0.83 13 blocks per stripe size 16 1.26 14 blocks per stripe size 16 1.76 15 blocks per stripe size 16 2.85 16 blocks per stripe size 16 Network Interface Statistics (per second) iface side bytes packets multicasts errors collisions pkt drops e0a recv 114768.96 92.82 0.00 0.00 0.00 xmit 193.85 1.43 0.05 0.00 0.00 e0b recv 521347.10 383.35 0.00 0.00 0.00 xmit 53015.11 280.33 0.14 0.00 0.00 vh recv 0.00 0.00 0.00 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00 mtrunk1 recv 636116.06 476.18 0.00 0.00 0.00 xmit 53208.96 281.77 0.25 0.00 0.00 mtrunk1-1 recv 588.41 5.26 2.10 0.00 0.00 xmit 665.54 2.92 0.08 0.00 0.00 mtrunk1-21 recv 261620.02 204.87 0.00 0.00 0.00 xmit 22000.14 120.08 0.05 0.00 0.00 mtrunk1-22 recv 0.00 0.00 0.00 0.00 0.00 xmit 5.28 0.05 0.05 0.00 0.00 mtrunk1-23 recv 371871.00 264.20 0.02 0.00 0.00 xmit 26855.71 156.13 0.05 0.00 0.00 Disk Statistics (per second) ut% is the percent of time the disk was busy. xfers is the number of data-transfer commands issued per second. xfers = ureads + writes + cpreads + greads + gwrites chain is the average number of 4K blocks per command. usecs is the average disk round-trip time per 4K block. disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs /aggr2/plex0/rg0: 0b.45 1 1.97 0.19 1.00 12333 1.10 9.42 988 0.69 5.07 768 0.00 .... . 0.00 .... . 0b.44 2 2.16 0.19 1.00 38167 1.28 8.35 925 0.69 5.07 838 0.00 .... . 0.00 .... . 0b.43 5 6.03 4.29 1.04 14107 1.13 6.84 1489 0.62 2.85 2430 0.00 .... . 0.00 .... . 0b.42 4 5.20 3.72 1.05 15565 0.88 8.00 1197 0.60 5.03 939 0.00 .... . 0.00 .... . 0b.41 4 5.09 3.76 1.01 14919 0.88 8.98 1184 0.45 4.03 1325 0.00 .... . 0.00 .... . 0b.40 5 5.28 4.09 1.08 13686 0.74 9.42 1210 0.45 5.28 1477 0.00 .... . 0.00 .... . 0b.39 5 5.54 4.18 1.08 15908 0.83 9.63 1142 0.52 4.44 1788 0.00 .... . 0.00 .... . 0b.38 4 5.45 3.90 1.00 17528 0.91 8.51 1327 0.63 3.56 1349 0.00 .... . 0.00 .... . 0b.37 4 5.29 3.72 1.07 16442 0.97 7.95 1359 0.60 4.23 1509 0.00 .... . 0.00 .... . 0b.36 4 5.01 3.50 1.01 13113 0.89 8.66 1371 0.62 4.18 1413 0.00 .... . 0.00 .... . 0b.32 5 5.54 4.03 1.36 11093 0.91 8.22 1433 0.60 3.92 1255 0.00 .... . 0.00 .... . 0b.17 4 5.06 3.76 1.01 13752 0.83 9.11 1268 0.46 4.60 1254 0.00 .... . 0.00 .... . 0b.33 5 5.51 3.90 1.02 14456 0.88 7.89 1553 0.73 4.02 1280 0.00 .... . 0.00 .... . 0b.18 4 5.51 3.79 1.19 12478 0.97 7.76 1591 0.74 3.71 1472 0.00 .... . 0.00 .... . 0b.34 5 5.51 3.89 1.12 13188 0.89 8.95 1301 0.73 3.62 1653 0.00 .... . 0.00 .... . 0b.19 4 5.04 3.56 1.07 11947 0.83 8.94 1308 0.65 3.50 2238 0.00 .... . 0.00 .... . 0b.35 4 4.74 3.33 1.14 12020 0.80 9.21 1353 0.60 4.26 1416 0.00 .... . 0.00 .... . 0b.20 4 5.32 3.89 1.01 13341 0.76 9.27 1352 0.68 3.86 1729 0.00 .... . 0.00 .... . /aggr1/plex0/rg0: 0b.24 2 2.39 0.19 1.00 13667 1.73 11.25 655 0.48 5.06 1140 0.00 .... . 0.00 .... . 0b.29 5 7.30 4.97 1.26 17742 1.71 10.48 666 0.62 3.45 1543 0.00 .... . 0.00 .... . 0b.26 4 6.34 4.37 1.28 13500 1.42 12.46 595 0.56 3.97 1427 0.00 .... . 0.00 .... . 0b.28 5 7.05 4.95 1.14 19218 1.50 11.82 695 0.60 3.44 1642 0.00 .... . 0.00 .... . 0b.27 7 6.20 4.26 1.05 19422 1.45 12.28 659 0.49 4.56 1000 0.00 .... . 0.00 .... . 0b.22 5 6.90 4.81 1.21 18814 1.53 11.51 835 0.56 3.81 1591 0.00 .... . 0.00 .... . 0b.21 4 6.57 4.37 1.11 18990 1.51 11.73 843 0.69 3.80 1655 0.00 .... . 0.00 .... . 0b.23 2 2.36 0.19 1.00 9667 1.70 11.42 677 0.48 5.06 1178 0.00 .... . 0.00 .... . Aggregate statistics: Minimum 1 1.97 0.19 0.74 0.45 0.00 0.00 Mean 4 5.15 3.44 1.11 0.59 0.00 0.00 Maximum 7 7.30 4.97 1.73 0.74 0.00 0.00 Spares and other disks: 0b.25 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . Spares and other disks: 0b.16 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . FCP Statistics (per second) 0.00 FCP Bytes recv 0.00 FCP Bytes sent 0.00 FCP ops iSCSI Statistics (per second) 516537.68 iSCSI Bytes recv 22866.13 iSCSI Bytes xmit 74.48 iSCSI ops Interrupt Statistics (per second) 1604.06 Clock (IRQ 5) 35.51 IPI (IRQ 8) 35.51 IPI (IRQ 9) 160.84 FCAL (IRQ 23) 259.37 SB MAC (IRQ 25) 41.22 SB MAC (IRQ 26) 35.56 IPI 2172.07 total Data Mover Statistics (per second) 627.05 total dma transfer KB 408.55 dma descriptors 82.58 dma desc wait count 753.86 total dma transfer KB 523.52 dma descriptors 112.63 dma desc wait count 0.00 total dma transfer KB 0.00 dma descriptors 0.00 dma desc wait count 0.00 total dma transfer KB 0.00 dma descriptors 0.00 dma desc wait count ____________________________ Matt Davies Director of International IT Operations General Atlantic 83 Pall Mall London SW1Y 5ES Tel: +44 207 484 3203 Fax: +44 207 484 2803 Mobile: +44 777 559 4265 ____________________________ _____________________________________________________________ This e-mail (including all attachments) is confidential and may be privileged. It is for the exclusive use of the addressee only. If you are not the addressee, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please erase all copies of the message and its attachments and notify us immediately at help(a)generalatlantic.com <mailto:help@generalatlantic.com>. Thank You.

5 4

SMSharepoint 1.1.1 & SMSQL 5.0
by Alexander Schalek 06 Nov '08

06 Nov '08

Hi all, has anyone sucessfully installed SM Sharepoint 1.1.1 in a SMSQL 5.0 environment? thanks in advance! alex -- Alexander Schalek, Technical Services / IT-Infrastruktur Bacher Systems EDV GmbH FN 54202i, Handelsgericht Wien Clemens-Holzmeister-Strasse 4 A-1100 Wien, Business Park Vienna, Austria phone: +43 (1) 60 126-34 | fax: -4 e-mail: as(a)bacher.at<mailto:as@bacher.at> | web: www.bacher.at<http://www.bacher.at/> -- unsere nächsten Veranstaltungen: NetApp Innovation 2008: http://www.netapp-innovation.at<http://www.netapp-innovation.at/>

2 2

RE: High CPU utilisation
by Davies,Matt 06 Nov '08

06 Nov '08

Nils, 1, yes it is both a source and destination for snapmirror, there is no snapvault 2, see below. ukfiler*> wafl scan status Volume CIFS_UK: Scan id Type of scan progress 3 active bitmap rearrangement fbn 1605 of 2230 w/ max_chain_len 3 191730 container block reclamation block 2042 of 2230 Volume root: Scan id Type of scan progress 5 active bitmap rearrangement fbn 63 of 107 w/ max_chain_len 3 191636 container block reclamation block 64 of 107 Volume UKEXCHDB: Scan id Type of scan progress 8 active bitmap rearrangement fbn 138 of 713 w/ max_chain_len 3 Volume UKEXCHLOGS: Scan id Type of scan progress 10 active bitmap rearrangement fbn 369 of 401 w/ max_chain_len 3 191608 container block reclamation block 391 of 401 Volume UKVMWARE: Scan id Type of scan progress 12 active bitmap rearrangement fbn 1538 of 4461 w/ max_chain_len 3 Volume DUEXCHLOGS: Scan id Type of scan progress 191713 container block reclamation block 879 of 892 Volume UKVIRTUALCENTRE: Scan id Type of scan progress 16 active bitmap rearrangement fbn 197 of 1115 w/ max_chain_len 3 Volume UKVMWARE1: Scan id Type of scan progress 18 active bitmap rearrangement fbn 4215 of 4461 w/ max_chain_len 3 Volume UKVMWARENFS: Scan id Type of scan progress 21 active bitmap rearrangement fbn 1019 of 1784 w/ max_chain_len 3 Volume TESTUKEXCHLOGS: Scan id Type of scan progress 191708 container block reclamation block 393 of 401 Volume UKEXCH3DB: Scan id Type of scan progress 152073 active bitmap rearrangement fbn 548 of 571 w/ max_chain_len 3 191729 container block reclamation block 356 of 571 Volume vol1: Scan id Type of scan progress 191630 container block reclamation block 0 of 0 Volume UKEXCH3LOGS: Scan id Type of scan progress 152075 active bitmap rearrangement fbn 29 of 321 w/ max_chain_len 3 191728 container block reclamation block 195 of 321 Thanks Matt -----Original Message----- From: Nils Vogels [mailto:bacardicoke@gmail.com] Sent: 06 November 2008 09:29 To: Davies,Matt Subject: Re: High CPU utilisation Couple of questions .. 1) Do you have snapvault/snapmirror running? 2) What does "wafl scan status" tell you? Greetings, Nils On Thu, Nov 6, 2008 at 10:09 AM, Davies,Matt <MDAVIES(a)generalatlantic.com> wrote: > Hi all, > > We are having a problem with a FAS270 that appears to be having high cpu utilisation issues. > > To be honest I have looked at sysstat and statit and I cannot see the problem. I would appreciate if someone could take a look at the below outputs and let me know what I'm missing. > > Thanks > > Matt > > ukfiler*> sysstat -i 3 > CPU NFS CIFS iSCSI Net kB/s Disk kB/s iSCSI kB/s Cache > in out read write in out age > 85% 18 0 26 203 13 661 0 91 0 24 > 68% 9 1 50 283 23 692 0 221 9 24 > 92% 7 4 25 144 11 1643 2443 90 3 24 > 78% 12 1 74 345 21 618 0 198 0 24 > 67% 8 0 45 288 18 722 0 205 5 24 > 96% 13 1 32 236 14 1365 1708 151 1 24 > 81% 8 4 39 222 16 843 867 167 4 24 > 85% 8 0 33 443 10 1212 1456 118 0 24 > 99% 16 2 43 370 21 912 0 168 4 24 > 99% 14 16 23 219 26 1023 0 98 0 24 > 99% 12 0 36 585 39 2452 3781 465 18 24 > ukfiler*> > > Hostname: ukfiler ID: 0084264545 Memory: 1022 MB > NetApp Release 7.2.3: Thu Jul 5 10:06:16 PDT 2007 > <6O> > Start time: Wed Nov 5 22:45:03 GMT 2008 > > CPU Statistics > 64.822501 time (seconds) 100 % > 52.221602 system time 81 % > 0.511694 rupt time 1 % (161581 rupts x 3 usec/rupt) > 51.709908 non-rupt system time 80 % > 77.423400 idle time 119 % > > 4.011144 time in CP 6 % 100 % > 0.052061 rupt time in CP 1 % (12144 rupts x 4 usec/rupt) > > Multiprocessor Statistics (per second) > cpu0 cpu1 total > sk switches 6581.74 7585.82 14167.57 > hard switches 1485.87 1560.68 3046.55 > domain switches 72.75 65.01 137.76 > CP rupts 125.45 61.89 187.34 > nonCP rupts 1350.03 955.30 2305.33 > IPI rupts 18.23 17.32 35.56 > > CP rupt usec 565.04 238.10 803.13 > nonCP rupt usec 4342.69 2747.95 7090.64 > idle 632999.86 561391.13 1194390.83 > kahuna 334624.38 409797.24 744421.77 > network 6697.98 8506.20 15204.18 > storage 4193.90 5622.94 9816.84 > exempt 3941.24 4036.10 7977.34 > raid 7872.95 4627.33 12500.27 > target 4409.87 2761.70 7171.57 > netcache 0.00 0.00 0.00 > netcache2 0.00 0.00 0.00 > cifs 352.07 271.33 623.39 > wafl_exempt 0.00 0.00 0.00 > > 49.342358 seconds with one or more CPUs active ( 76%) > > 46.750469 seconds with one CPU active ( 72%) > 2.591888 seconds with both CPUs active ( 4%) > > Domain Utilization By Exempt (per second) > 0.00 idle 0.00 kahuna > 0.00 network 0.00 storage > 0.00 exempt 0.00 raid > 0.00 target 0.00 netcache > 0.00 netcache2 0.00 cifs > 0.00 wafl_exempt > > > CSMP Domain Switches (per second) > From\To idle kahuna network storage exempt raid target netcache netcache2 cifs wafl_exem > pt > idle 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 > 0 > kahuna 0.00 0.00 27.09 2.65 0.00 5.32 1.87 0.00 0.00 18.79 0.0 > 0 > network 0.00 27.09 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.20 0.0 > 0 > storage 0.00 2.65 0.00 0.00 0.00 12.96 0.00 0.00 0.00 0.00 0.0 > 0 > exempt 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 > 0 > raid 0.00 5.32 0.00 12.96 0.00 0.00 0.00 0.00 0.00 0.00 0.0 > 0 > target 0.00 1.87 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 > 0 > netcache 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 > 0 > netcache2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 > 0 > cifs 0.00 18.79 0.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.0 > 0 > wafl_exempt 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0. > 00 > > Miscellaneous Statistics (per second) > 3046.55 hard context switches 10.38 NFS operations > 1.42 CIFS operations 0.00 HTTP operations > 0.00 NetCache URLs 0.00 streaming packets > 1240.42 network KB received 100.32 network KB transmitted > 655.39 disk KB read 1146.02 disk KB written > 648.90 NVRAM KB written 0.00 nolog KB written > 8.27 WAFL bufs given to clients 0.00 checksum cache hits ( 0%) > 1.23 no checksum - partial buffer 0.00 FCP operations > 74.48 iSCSI operations > > WAFL Statistics (per second) > 15.94 name cache hits ( 95%) 0.86 name cache misses ( 5%) > 529973.33 buf hash hits ( 100%) 1710.32 buf hash misses ( 0%) > 516.44 inode cache hits ( 100%) 0.06 inode cache misses ( 0%) > 262496.95 buf cache hits ( 100%) 19.65 buf cache misses ( 0%) > 30.17 blocks read 67.00 blocks read-ahead > 58.10 chains read-ahead 0.99 dummy reads > 14.25 blocks speculative read-ahead 226.31 blocks written > 2.27 stripes written 0.00 blocks over-written > 0.09 wafl_timer generated CP 0.00 snapshot generated CP > 0.00 wafl_avail_bufs generated CP 0.00 dirty_blk_cnt generated CP > 0.00 full NV-log generated CP 0.00 back-to-back CP > 0.00 flush generated CP 0.00 sync generated CP > 0.00 wafl_avail_vbufs generated CP 0.00 deferred back-to-back CP > 0.00 low datavecs generated CP 304.72 non-restart messages > 4.98 IOWAIT suspends 133872 buffers > > RAID Statistics (per second) > 40.91 xors 0.00 long dispatches [0] > 0.00 long consumed [0] 0.00 long consumed hipri [0] > 0.00 long low priority [0] 0.00 long high priority [0] > 0.00 long monitor tics [0] 0.00 long monitor clears [0] > 0.00 long dispatches [1] 0.00 long consumed [1] > 0.00 long consumed hipri [1] 0.00 long low priority [1] > 0.00 long high priority [1] 0.00 long monitor tics [1] > 0.00 long monitor clears [1] 18 max batch > 2.11 blocked mode xor 19.68 timed mode xor > 0.05 fast adjustments 0.05 slow adjustments > 0 avg batch start 0 avg stripe/msec > 2.64 tetrises written 0.00 master tetrises > 0.00 slave tetrises 29.70 stripes written > 11.22 partial stripes 18.48 full stripes > 226.68 blocks written 31.56 blocks read > 1.13 1 blocks per stripe size 6 0.29 2 blocks per stripe size 6 > 0.17 3 blocks per stripe size 6 0.35 4 blocks per stripe size 6 > 1.80 5 blocks per stripe size 6 15.63 6 blocks per stripe size 6 > 1.11 1 blocks per stripe size 16 0.31 2 blocks per stripe size 16 > 0.32 3 blocks per stripe size 16 0.19 4 blocks per stripe size 16 > 0.09 5 blocks per stripe size 16 0.20 6 blocks per stripe size 16 > 0.05 7 blocks per stripe size 16 0.20 8 blocks per stripe size 16 > 0.12 9 blocks per stripe size 16 0.23 10 blocks per stripe size 16 > 0.23 11 blocks per stripe size 16 0.56 12 blocks per stripe size 16 > 0.83 13 blocks per stripe size 16 1.26 14 blocks per stripe size 16 > 1.76 15 blocks per stripe size 16 2.85 16 blocks per stripe size 16 > > Network Interface Statistics (per second) > iface side bytes packets multicasts errors collisions pkt drops > e0a recv 114768.96 92.82 0.00 0.00 0.00 > xmit 193.85 1.43 0.05 0.00 0.00 > e0b recv 521347.10 383.35 0.00 0.00 0.00 > xmit 53015.11 280.33 0.14 0.00 0.00 > vh recv 0.00 0.00 0.00 0.00 0.00 > xmit 0.00 0.00 0.00 0.00 0.00 > mtrunk1 recv 636116.06 476.18 0.00 0.00 0.00 > xmit 53208.96 281.77 0.25 0.00 0.00 > mtrunk1-1 recv 588.41 5.26 2.10 0.00 0.00 > xmit 665.54 2.92 0.08 0.00 0.00 > mtrunk1-21 recv 261620.02 204.87 0.00 0.00 0.00 > xmit 22000.14 120.08 0.05 0.00 0.00 > mtrunk1-22 recv 0.00 0.00 0.00 0.00 0.00 > xmit 5.28 0.05 0.05 0.00 0.00 > mtrunk1-23 recv 371871.00 264.20 0.02 0.00 0.00 > xmit 26855.71 156.13 0.05 0.00 0.00 > > Disk Statistics (per second) > ut% is the percent of time the disk was busy. > xfers is the number of data-transfer commands issued per second. > xfers = ureads + writes + cpreads + greads + gwrites > chain is the average number of 4K blocks per command. > usecs is the average disk round-trip time per 4K block. > > disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs > /aggr2/plex0/rg0: > 0b.45 1 1.97 0.19 1.00 12333 1.10 9.42 988 0.69 5.07 768 0.00 .... . 0.00 .... . > 0b.44 2 2.16 0.19 1.00 38167 1.28 8.35 925 0.69 5.07 838 0.00 .... . 0.00 .... . > 0b.43 5 6.03 4.29 1.04 14107 1.13 6.84 1489 0.62 2.85 2430 0.00 .... . 0.00 .... . > 0b.42 4 5.20 3.72 1.05 15565 0.88 8.00 1197 0.60 5.03 939 0.00 .... . 0.00 .... . > 0b.41 4 5.09 3.76 1.01 14919 0.88 8.98 1184 0.45 4.03 1325 0.00 .... . 0.00 .... . > 0b.40 5 5.28 4.09 1.08 13686 0.74 9.42 1210 0.45 5.28 1477 0.00 .... . 0.00 .... . > 0b.39 5 5.54 4.18 1.08 15908 0.83 9.63 1142 0.52 4.44 1788 0.00 .... . 0.00 .... . > 0b.38 4 5.45 3.90 1.00 17528 0.91 8.51 1327 0.63 3.56 1349 0.00 .... . 0.00 .... . > 0b.37 4 5.29 3.72 1.07 16442 0.97 7.95 1359 0.60 4.23 1509 0.00 .... . 0.00 .... . > 0b.36 4 5.01 3.50 1.01 13113 0.89 8.66 1371 0.62 4.18 1413 0.00 .... . 0.00 .... . > 0b.32 5 5.54 4.03 1.36 11093 0.91 8.22 1433 0.60 3.92 1255 0.00 .... . 0.00 .... . > 0b.17 4 5.06 3.76 1.01 13752 0.83 9.11 1268 0.46 4.60 1254 0.00 .... . 0.00 .... . > 0b.33 5 5.51 3.90 1.02 14456 0.88 7.89 1553 0.73 4.02 1280 0.00 .... . 0.00 .... . > 0b.18 4 5.51 3.79 1.19 12478 0.97 7.76 1591 0.74 3.71 1472 0.00 .... . 0.00 .... . > 0b.34 5 5.51 3.89 1.12 13188 0.89 8.95 1301 0.73 3.62 1653 0.00 .... . 0.00 .... . > 0b.19 4 5.04 3.56 1.07 11947 0.83 8.94 1308 0.65 3.50 2238 0.00 .... . 0.00 .... . > 0b.35 4 4.74 3.33 1.14 12020 0.80 9.21 1353 0.60 4.26 1416 0.00 .... . 0.00 .... . > 0b.20 4 5.32 3.89 1.01 13341 0.76 9.27 1352 0.68 3.86 1729 0.00 .... . 0.00 .... . > /aggr1/plex0/rg0: > 0b.24 2 2.39 0.19 1.00 13667 1.73 11.25 655 0.48 5.06 1140 0.00 .... . 0.00 .... . > 0b.29 5 7.30 4.97 1.26 17742 1.71 10.48 666 0.62 3.45 1543 0.00 .... . 0.00 .... . > 0b.26 4 6.34 4.37 1.28 13500 1.42 12.46 595 0.56 3.97 1427 0.00 .... . 0.00 .... . > 0b.28 5 7.05 4.95 1.14 19218 1.50 11.82 695 0.60 3.44 1642 0.00 .... . 0.00 .... . > 0b.27 7 6.20 4.26 1.05 19422 1.45 12.28 659 0.49 4.56 1000 0.00 .... . 0.00 .... . > 0b.22 5 6.90 4.81 1.21 18814 1.53 11.51 835 0.56 3.81 1591 0.00 .... . 0.00 .... . > 0b.21 4 6.57 4.37 1.11 18990 1.51 11.73 843 0.69 3.80 1655 0.00 .... . 0.00 .... . > 0b.23 2 2.36 0.19 1.00 9667 1.70 11.42 677 0.48 5.06 1178 0.00 .... . 0.00 .... . > > Aggregate statistics: > Minimum 1 1.97 0.19 0.74 0.45 0.00 0.00 > Mean 4 5.15 3.44 1.11 0.59 0.00 0.00 > Maximum 7 7.30 4.97 1.73 0.74 0.00 0.00 > > Spares and other disks: > 0b.25 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . > > Spares and other disks: > 0b.16 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . > > FCP Statistics (per second) > 0.00 FCP Bytes recv 0.00 FCP Bytes sent > 0.00 FCP ops > > iSCSI Statistics (per second) > 516537.68 iSCSI Bytes recv 22866.13 iSCSI Bytes xmit > 74.48 iSCSI ops > > Interrupt Statistics (per second) > 1604.06 Clock (IRQ 5) 35.51 IPI (IRQ 8) > 35.51 IPI (IRQ 9) 160.84 FCAL (IRQ 23) > 259.37 SB MAC (IRQ 25) 41.22 SB MAC (IRQ 26) > 35.56 IPI 2172.07 total > > Data Mover Statistics (per second) > 627.05 total dma transfer KB 408.55 dma descriptors > 82.58 dma desc wait count 753.86 total dma transfer KB > 523.52 dma descriptors 112.63 dma desc wait count > 0.00 total dma transfer KB 0.00 dma descriptors > 0.00 dma desc wait count 0.00 total dma transfer KB > 0.00 dma descriptors 0.00 dma desc wait count > > ____________________________ > Matt Davies > Director of International IT Operations > General Atlantic > 83 Pall Mall > London > SW1Y 5ES > > Tel: +44 207 484 3203 > Fax: +44 207 484 2803 > Mobile: +44 777 559 4265 > ____________________________ > > > > > _____________________________________________________________ > This e-mail (including all attachments) is confidential and may be privileged. > It is for the exclusive use of the addressee only. If you are not the addressee, > you are hereby notified that any dissemination of this communication is strictly > prohibited. If you have received this communication in error, please erase all > copies of the message and its attachments and notify us immediately at > help(a)generalatlantic.com <mailto:help@generalatlantic.com>. Thank You. > > -- Simple guidelines to happiness: Work like you don't need the money, Love like your heart has never been broken and Dance like no one can see you.

1 0

RE: Is the list still active
by Davies,Matt 06 Nov '08

06 Nov '08

I will re post and see what happens.... Thanks Matt -----Original Message----- From: De Wit Tom (Consultant) [mailto:tom.de.wit@consultant.volvo.com] Sent: 06 November 2008 09:06 To: Davies,Matt; toasters(a)mathworks.com Subject: RE: Is the list still active Well it works now ;-) Yes, the list is still active. Grtz, Tom -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Davies,Matt Sent: donderdag 6 november 2008 9:45 To: toasters(a)mathworks.com Subject: Is the list still active Have been trying to post a message but it does not appear to be working. _____________________________________________________________ This e-mail (including all attachments) is confidential and may be privileged. It is for the exclusive use of the addressee only. If you are not the addressee, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please erase all copies of the message and its attachments and notify us immediately at help(a)generalatlantic.com <mailto:help@generalatlantic.com>. Thank You.

1 0

Is the list still active
by Davies,Matt 06 Nov '08

06 Nov '08

Have been trying to post a message but it does not appear to be working. _____________________________________________________________ This e-mail (including all attachments) is confidential and may be privileged. It is for the exclusive use of the addressee only. If you are not the addressee, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please erase all copies of the message and its attachments and notify us immediately at help(a)generalatlantic.com <mailto:help@generalatlantic.com>. Thank You.

1 0

TR-3428 & Cross-stack Etherchannel
by Carl Howell 04 Nov '08

04 Nov '08

For those of you that are doing LACP/cross-stack Etherchannel in your ESX environments, like the example in section 6.8 on page 40 of TR-3428, what switches are you using? If you are doing LACP across Cisco 6500's, what SUP's are you using? Thanks, --Carl

12 15

SV: Brief outages on the filer?
by Karlsson Ulf Ibrahim :ULK 04 Nov '08

04 Nov '08

Maybe this from http://media.netapp.com/documents/tr-3428.pdf (Netapp+VMware storage best practices) When using VMware snapshots (VMsnaps) with NFS Datastores a condition exists where I/O to the VM is suspended while VMsnaps are being deleted (or more technically speaking the process of committing the redo logs to the VMDK files occur). This issue is experienced with any VMware technology that leverages VMsnaps such as VMware Consolidated Backup, Storage VMotion, Scalable Virtual Images, etc. and SnapManager for Virtual Infrastructure from NetApp. VMware has identified this behavior as a bug (SR195302591), and has released patch ESX350-200808401-BG which addresses this bug. At present time, this patch applies to ESX version 3.5, updates 1 and 2 only. If you plan on leveraging any of the applications that require the VMsnap process please apply this patch, and complete its installation requirements, on the ESX servers in your environment. If you are using scripts in order to take on disk snapshot backups and are unable to upgrade your systems, then VMware and NetApp recommend that you discontinue the use of the VMsnap process prior to executing the NetApp snapshot. /Uffe -----Ursprungligt meddelande----- Från: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] För Glenn Walker Skickat: den 3 november 2008 20:03 Till: Page, Jeremy; toasters(a)mathworks.com Ämne: RE: Brief outages on the filer? Any way you can predict when it will happen? Sysstat (or better yet, perfstat) would be of help here. Something I've noticed on my infrastructure: VMWare over NFS (unsure about other protocols) will have huge spikes where they write lots of data in a quick burst - happens only a few times a day on relatively quiet systems, but I can definitely see a spike on the filer. Perhaps you have the same thing going, just a SWAG... The impact on our side is not really felt - but the filer does go into back2back CPs from the massive spike (200MB/s - 350MB/s in a short window) and that could manifest itself as 'poor disk response time'. In our case, we're running VMWare over NFS and Exchange over iSCSI on the same filers, but no one is really complaining when the 'events' happen. Just something I've noticed for a while. FAS6070 and the busy time is recorded around 6000 NFS IOPS. That said, we did a stress test with about 25 guests running IOMeter and were able to push 15000 NFS OPS on node 1, 10000 NFS OPS on node 2 (a combined 400MB/s write, 300MB/s read) without any sort of reported performance problems. ________________________________ From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Page, Jeremy Sent: Monday, November 03, 2008 11:02 AM To: toasters(a)mathworks.com Subject: Brief outages on the filer? I am seeing brief outages where my VMs (NFS as the back end protocol) and SQL LUNs (FC) both complain of poor disk response time at the same time. I don't think it can be the infrastructure since one is IP and the other FC. The LUNs are on a different set of spindles/different aggr then the NFS volumes as well, so I don't think it's a disk bottleneck. I'm on a 3070 and rarely do we hit 3500 IOPS (and 90+% of that out of cache) or go above 40% for the busiest CPU (normally we're in the 15-25% range) so I am not sure what's going on here, any suggestions on how to troubleshoot it? We're running 7.2.4, I want to wait for 7.3.1 to upgrade since we are using NFS for VMware and there are several fixes that will be beneficial to us. Please be advised that this email may contain confidential information. If you are not the intended recipient, please do not read, copy or re-transmit this email. If you have received this email in error, please notify us by email by replying to the sender and by telephone (call us collect at +1 202-828-0850) and delete this message and any attachments. Thank you in advance for your cooperation and assistance. In addition, Danaher and its subsidiaries disclaim that the content of this email constitutes an offer to enter into, or the acceptance of, any contract or agreement or any amendment thereto; provided that the foregoing disclaimer does not invalidate the binding effect of any digital or other electronic reproduction of a manual signature that is included in any attachment to this email.

3 5

RE: Data OnTap 7.2.6
by Darragh, Stephen J (US SSA) 03 Nov '08

03 Nov '08

Hello, You mention you believe it was GA at one point, and I don't believe that is correct. It used to be the case, but at some point, they determined that if the base release was GD, then subsiqent released are auto-majically placed into GD status. This has caused us problems and I hope others chime in to see if NetApp will go back to the old release cycle of every version being GA before going to GD. Stephen Darragh BAE Systems IT Merrimack NH -----Original Message----- From: owner-toasters(a)mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Kevin Sheen Sent: Thursday, October 30, 2008 9:04 PM To: Jack Lyons; toasters(a)mathworks.com Subject: Re: Data OnTap 7.2.6 We upgraded a pair of clustered 3070's and a 3020 last Saturday. We do a small amount of CIFS, more NFS and lots of LUNs. We haven't had any issues. All of the hosts connected to the clustered 3070's are multipathed except for one. good luck with the issues, Kevin At 07:17 PM 10/30/2008, Jack Lyons wrote: Does NetApp still publish the number of appliances running different version of Data OnTap? We just upgraded to 7.2.6 on Sunday and we have had two (and possibly a third) panics causing failover on 1 of the 4 filers. Failover went smoothly for the most part (I bet the app team that didn't want to spend the money multipathing software / hba's will be singing another tune tomorrow) but both panics didn't create complete core dumps. We are working on the case with NetApp support but want to see what other people were seeing about 7.2.6. 7.2.6 went GD recently and I assume it was GA for awhile before that, but I can't figure release dates or the number of appliances running the different versions of Data OnTap. Thanks Jack p.s. why did we upgrade? http://now.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=281669 is the bug we are hitting...in addition to: http://now.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=253517 http://now.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=256975 http://now.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=271864

3 2

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters November 2008