Here's the sysstat. Interesting that this shows periodic 0 writes.
Tue Feb 5 22:22:30 GMT [tn_login_0]: root logged in from host: 192.168.21.16 sysstat 1 CPU NFS CIFS HTTP Net kB/s Disk kB/s Tape kB/s Cache in out read write read write age 54% 1012 0 1 393 1494 2430 2446 0 0 1 43% 1012 0 0 431 1549 2302 3947 0 0 1 46% 1278 0 0 471 1787 2708 1116 0 0 1 57% 1080 0 0 611 2019 2078 5177 0 0 1 31% 726 0 0 439 1056 840 1028 0 0 1 37% 962 0 0 388 1872 1561 0 0 0 1 32% 919 0 0 404 1786 1688 0 0 0 1 33% 1050 0 0 407 1815 1529 0 0 0 1 43% 1102 0 0 623 2148 1264 0 0 0 1 43% 857 0 1 347 1585 2154 0 0 0 1 38% 746 0 0 430 1878 1704 0 0 0 1 58% 463 0 1 255 800 2610 3675 0 0 1 52% 997 0 0 517 1544 2840 2084 0 0 1 48% 792 0 0 401 1350 2222 5365 0 0 1 50% 1028 0 0 429 2055 2264 2108 0 0 1 37% 864 0 0 304 1864 1893 2210 0 0 1 39% 1053 0 0 566 2154 988 8 0 0 1 45% 1143 0 0 604 2309 1733 0 0 0 1 35% 1034 0 0 420 2123 1860 0 0 0 1 CPU NFS CIFS HTTP Net kB/s Disk kB/s Tape kB/s Cache in out read write read write age 42% 1000 0 0 550 2016 1264 0 0 0 1 52% 1118 0 1 575 2541 1589 0 0 0 1 30% 791 0 0 327 1648 1880 0 0 0 1 48% 880 0 0 382 1896 2022 1905 0 0 1 47% 745 0 1 354 1354 2196 2828 0 0 1 57% 1287 0 0 581 1772 2366 3583 0 0 1 51% 1136 0 0 595 1929 2488 1168 0 0 1 48% 987 0 0 419 1273 2348 3504 0 0 1 35% 768 0 0 421 1623 1597 720 0 0 1 45% 1237 0 0 624 2465 1869 0 0 0 1 38% 1112 0 0 390 1995 1688 0 0 0 1 44% 1044 0 0 585 1817 1697 0 0 0 1
ifconfig output:
e0: flags=240043<UP,BROADCAST,RUNNING,UP_1ARY,LINK_UP> mtu 1500 inet 192.168.21.81 netmask 0xffffff00 broadcast 192.168.21.255 ether 00:a0:98:00:97:67 (auto-100tx-fd-up) lo: flags=240049<UP,LOOPBACK,RUNNING,UP_1ARY,LINK_UP> mtu 1536 inet 127.0.0.1 netmask 0xff000000 broadcast 127.0.0.1
Regards,
Edward.
-----Original Message----- From: Mike Ball [mailto:MBall@DATALINK.com] Sent: 05 February 2002 19:58 To: Edward Hibbert; toasters Subject: RE: Slow write performance
Edward, Telnet to the netapp and type "sysstat 1". Let it run for about 20 seconds and send us the output. Also, type "ifconfig -a" and send us the output. Mike
-----Original Message----- From: Edward Hibbert [mailto:EH@dataconnection.com] Sent: Tuesday, February 05, 2002 2:24 PM To: toasters Subject: Slow write performance
We're seeing some performance problems on an F720 which you guys might be able to help with (even though it's not exactly top of the range nowadays).
What we see is: - We're driving it over NFS v3. - We get about 1500 ops/sec out of it, of which one third are writes and two thirds are reads. There aren't many file open/close operations. - The operations are to random locations in large (10GB) files. - The CPU is running about 75%, and the network input and output below the throughput of the link we have to it. We've seen both CPU and network go higher if we do simple copy tests. - Looking it it via pktt trace, something approaching 15% of WRITE operations take long enough for the clients to time out and retransmit (so at least 1 second). None of the READ operations do. - The retransmissions appear to come in bunches. For example we'll see a few seconds where the filer doesn't respond, during which time the retransmissions will come in, then it will wake up and send some responses back. - The rest of the time, the WRITES are very fast (sub-ms).
This appears to have worsened recently. We tried a couple of things: - We thought that this might be because the disk had got full and fragmented, so we zapped a bunch of data. - We rebooted. Neither of these seemed to help much.
sysstat consistently shows a cache age of 1. This, and the bursty nature of the delays, suggest to me that I'm just hitting it too hard, and there's some kind of periodic cache-flushing operation going on, but do any of you folk have any other suggestions?
Edward Hibbert Internet Applications Group Data Connection Ltd Tel: +44 131 662 1212 Fax: +44 131 662 1345 Email: eh@dataconnection.com Web: http://www.dataconnection.com