Hi all,
I'm working to diagnose some problems with performance of a Lotus Notes server talking over iSCSI to a FAS2050 running 7.2.6.1P8 and figuring out what my options are.
The system has 20 X289 420Gb SATA/SAS disks on it currently, and we are looking to add some FC disks in a new shelf. I'm not seeing any network collisions or anything like that, so I don't think it's a hardware problem, but more of a resource problem.
There is some CIFS on the system in use as well, but it's mostly iSCSI traffic. Looking at the info, I don't see any terrible numbers, and the performance looks ok. Not great, but ok.
Here's some output from 'statit' which I left running for about 2 minutes or so. I can go longer if need be obviously.
Hostname: irvfs1 ID: 0135054398 Memory: 1792 MB NetApp Release 7.2.6.1P8: Fri Jul 17 09:03:14 PDT 2009 <8O> Start time: Fri Oct 1 11:30:24 PDT 2010
CPU Statistics 316.694026 time (seconds) 100 % 74.098481 system time 23 % 4.159199 rupt time 1 % (905915 rupts x 5 usec/rupt) 69.939282 non-rupt system time 22 % 242.595544 idle time 77 %
73.954047 time in CP 23 % 100 % 1.130218 rupt time in CP 2 % (226012 rupts x 5 usec/rupt)
Miscellaneous Statistics (per second) 10966.86 hard context switches 0.00 NFS operations 125.38 CIFS operations 0.00 HTTP operations 0.00 NetCache URLs 0.00 streaming packets 2870.35 network KB received 8529.71 network KB transmitted 11154.87 disk KB read 4770.64 disk KB written 2698.64 NVRAM KB written 0.00 nolog KB written 2034.43 WAFL bufs given to clients 0.00 checksum cache hits ( 0%) 0.00 no checksum - partial buffer 0.00 FCP operations 434.87 iSCSI operations
WAFL Statistics (per second) 218.61 name cache hits ( 80%) 55.35 name cache misses ( 20%) 20887.17 buf hash hits ( 82%) 4548.72 buf hash misses ( 18%) 1813.94 inode cache hits ( 100%) 6.17 inode cache misses ( 0%) 5871.88 buf cache hits ( 75%) 1963.55 buf cache misses ( 25%) 171.92 blocks read 2101.86 blocks read-ahead 384.55 chains read-ahead 88.86 dummy reads 344.25 blocks speculative read-ahead 831.96 blocks written 6.58 stripes written 0.00 blocks over-written 0.08 wafl_timer generated CP 0.00 snapshot generated CP 0.00 wafl_avail_bufs generated CP 0.00 dirty_blk_cnt generated CP 0.00 full NV-log generated CP 0.00 back-to-back CP 0.00 flush generated CP 0.00 sync generated CP 0.00 wafl_avail_vbufs generated CP 0.00 deferred back-to-back CP 0.00 low mbufs generated CP 0.00 low datavecs generated CP 1183.82 non-restart messages 121.62 IOWAIT suspends 314438 buffers
RAID Statistics (per second) 312.93 xors 0.00 long dispatches [0] 0.00 long consumed [0] 0.00 long consumed hipri [0] 0.00 long low priority [0] 0.00 long high priority [0] 0.00 long monitor tics [0] 0.00 long monitor clears [0] 0.00 long dispatches [1] 0.00 long consumed [1] 0.00 long consumed hipri [1] 0.00 long low priority [1] 0.00 long high priority [1] 0.00 long monitor tics [1] 0.00 long monitor clears [1] 18 max batch 0.56 blocked mode xor 112.52 timed mode xor 0.05 fast adjustments 0.05 slow adjustments 0 avg batch start 0 avg stripe/msec 6.73 tetrises written 0.00 master tetrises 0.00 slave tetrises 179.93 stripes written 133.00 partial stripes 46.93 full stripes 832.11 blocks written 361.72 blocks read 21.89 1 blocks per stripe size 7 17.47 2 blocks per stripe size 7 16.67 3 blocks per stripe size 7 20.33 4 blocks per stripe size 7 24.39 5 blocks per stripe size 7 32.24 6 blocks per stripe size 7 46.93 7 blocks per stripe size 7
Network Interface Statistics (per second) iface side bytes packets multicasts errors collisions pkt drops e0a recv 55529.33 122.23 0.00 0.00 0.00 xmit 227177.99 228.14 0.00 0.00 0.00 e0b recv 39175.77 117.07 0.00 0.00 0.00 xmit 79516.28 126.61 0.27 0.00 0.00 e1a recv 2844533.72 2570.46 0.00 0.00 0.00 xmit 8427700.87 6604.89 0.00 0.00 0.00 e1b recv 0.40 0.01 0.00 0.00 0.00 xmit 29.44 0.27 0.27 0.00 0.00 vh recv 0.00 0.00 0.00 0.00 0.00 xmit 0.00 0.00 0.00 0.00 0.00 vif1 recv 94724.52 239.47 0.08 0.00 0.00 xmit 306742.10 354.89 0.27 0.00 0.00 iscsi recv 2863035.80 2585.27 0.02 0.00 0.00 xmit 8459234.73 6632.46 0.27 0.00 0.00
Disk Statistics (per second) ut% is the percent of time the disk was busy. xfers is the number of data-transfer commands issued per second. xfers = ureads + writes + cpreads + greads + gwrites chain is the average number of 4K blocks per command. usecs is the average disk round-trip time per 4K block.
disk ut% xfers ureads--chain-usecs writes--chain-usecs cpreads-chain-usecs greads--chain-usecs gwrites-chain-usecs /aggr0/plex0/rg0: 0c.00.0 3 8.11 0.16 1.36 6206 5.20 15.58 423 2.75 6.96 377 0.00 .... . 0.00 .... . 0c.00.1 3 8.27 0.16 1.36 21309 5.38 15.13 423 2.74 7.08 402 0.00 .... . 0.00 .... . 0c.00.2 23 48.11 40.63 4.28 1733 4.37 11.31 875 3.11 8.07 616 0.00 .... . 0.00 .... . 0c.00.15 23 47.38 39.91 4.23 1708 4.27 11.16 936 3.19 8.09 544 0.00 .... . 0.00 .... . 0c.00.4 23 46.19 39.16 4.31 1739 3.89 12.76 813 3.14 7.67 631 0.00 .... . 0.00 .... . 0c.00.5 23 46.99 39.86 4.28 1761 3.98 12.16 814 3.15 8.14 529 0.00 .... . 0.00 .... . 0c.00.6 23 47.39 40.20 4.33 1711 3.95 11.92 832 3.25 8.61 542 0.00 .... . 0.00 .... . 0c.00.7 23 47.00 39.86 4.33 1738 3.92 11.96 806 3.21 8.51 567 0.00 .... . 0.00 .... . 0c.00.8 22 45.78 38.71 4.37 1688 3.87 12.57 839 3.20 7.94 563 0.00 .... . 0.00 .... . /aggr0/plex0/rg1: 0c.00.9 2 5.78 0.00 .... . 4.16 23.81 222 1.62 6.71 399 0.00 .... . 0.00 .... . 0c.00.10 2 5.78 0.00 .... . 4.18 23.68 231 1.60 6.80 384 0.00 .... . 0.00 .... . 0c.00.11 23 47.92 39.78 4.24 1740 4.77 14.60 656 3.37 8.05 550 0.00 .... . 0.00 .... . 0c.00.12 23 47.55 39.45 4.14 1860 4.79 14.57 649 3.31 8.08 514 0.00 .... . 0.00 .... . 0c.00.13 23 47.62 39.49 4.26 1725 4.94 14.52 649 3.19 7.79 532 0.00 .... . 0.00 .... . 0c.00.19 24 48.22 39.78 4.18 1849 5.07 13.97 689 3.37 7.91 512 0.00 .... . 0.00 .... . 0c.00.17 23 48.19 40.11 4.16 1811 4.82 14.66 641 3.25 8.15 553 0.00 .... . 0.00 .... . 0c.00.14 23 47.77 39.81 4.16 1834 4.74 14.92 609 3.21 8.00 530 0.00 .... . 0.00 .... . 0c.00.16 24 48.31 40.05 4.13 1829 4.98 14.28 688 3.27 7.88 584 0.00 .... . 0.00 .... .
Aggregate statistics: Minimum 2 5.78 0.00 3.87 1.60 0.00 0.00 Mean 18 38.46 30.95 4.52 2.99 0.00 0.00 Maximum 24 48.31 40.63 5.38 3.37 0.00 0.00
Spares and other disks: 0c.00.3 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
Spares and other disks: 0c.00.18 0 0.00 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... . 0.00 .... .
FCP Statistics (per second) 0.00 FCP Bytes recv 0.00 FCP Bytes sent 0.00 FCP ops
iSCSI Statistics (per second) 2672116.49 iSCSI Bytes recv 8064194.15 iSCSI Bytes xmit 434.87 iSCSI ops
Interrupt Statistics (per second) 999.38 Clock (IRQ 0) 0.05 Uart (IRQ 4) 1766.58 PCI direct (IRQ 16) 94.52 PCI direct (IRQ 17) 0.00 RTC 2860.54 total
NVRAM Statistics (per second) 0.00 total dma transfer KB 0.00 wafl write req data KB 0.00 dma transactions 0.00 dma destriptors 28.36 transactions not queued 67.01 transactions queued 12.38 total waittime (MS) 29.34 completion proc wakeups 18.39 completion proc waitdone 186.57 waitdone preempts 0.00 waitdone delays 2698.66 total nvlog KB 0.00 nvlog shadow header array full 0.00 channel1 dma transfer KB 0.00 channel1 dma transactions 0.00 channel1 dma descriptors
E7520 Data Mover Statistics (per second) 3810.35 total dma transfer KB 1.18 total bcopy transfer KB 4.22 total waittime (MS)
I've also got some 'sysstat -i 5' numbers:
sysstat -i 5
CPU NFS CIFS iSCSI Net kB/s Disk kB/s iSCSI kB/s Cache in out read write in out age 15% 0 195 247 1232 4944 7678 4267 1104 4629 1 16% 0 238 352 1959 6185 7414 3201 1770 4791 1 11% 0 135 264 1908 4533 4946 0 1770 4257 1 18% 0 171 246 1155 4372 6915 7230 1046 4084 1 14% 0 230 298 1868 4478 4892 0 1717 4184 2 17% 0 164 284 1113 4955 7014 5319 1001 4683 2 9% 0 18 238 1395 3836 4217 0 1303 3654 2 15% 0 12 245 1759 4110 7539 3378 1653 3912 3 13% 0 21 244 900 5029 6302 3205 824 4814 2 11% 0 22 282 1569 5152 5866 0 1461 4910 2 19% 0 21 389 2280 5927 9750 6982 2132 5644 2 14% 0 49 401 1746 6072 6506 0 1614 5775 3 27% 0 83 491 5246 8447 12100 7693 4961 7985 2 11% 0 126 263 1772 4800 5258 0 1642 4553 2 16% 0 20 487 802 7104 10182 718 694 6800 6 25% 0 15 518 1763 6440 10771 9322 1621 6144 2 15% 0 81 389 1745 6271 6844 0 1607 5979 3 20% 0 140 287 906 6280 10088 7681 782 5965 4 8% 0 70 201 1163 4381 4638 0 937 4148 2
Any thoughts?
Thnaks, John John Stoffel - Senior Staff Systems Administrator - System LSI Group Toshiba America Electronic Components, Inc. - http://www.toshiba.com/taec john.stoffel@taec.toshiba.com - 508-486-1087