Edward,
Try to disable TCP over NFS and only enable UDP. Also in a switched environment 10/100/1000 BT keep the xfersizes to 8192 this has proven to improve performance (at least for us). This is a known bug on certain switches, it's also listed on now.netapp.com.
Re, Magnus Swenson Cadence Design Systems Inc. Addison, TX
-----Original Message----- From: FPS France [mailto:fpsfrance@fps.fr] Sent: Thursday, February 21, 2002 3:59 AM To: edward valencia; toasters@mathworks.com Subject: Re: Slow writes
just an idea : did u well check on 100Mb machines (solaris) that they all are in full dupleix or all in half , as - as u may know as u look a guru - mixing the two types can greatlly disturb network performance. Check that in both solaris machines, on the switch, and on other machines in the same subnet. U could try to switch a machine in half dupleix just to c what happend. In any case, make sure u avoid auto negoiciation for 100Mb nic - it's ok for the giga.
Hope u will turn around this problem. Fanku to let me know.
----- Original Message ----- From: "edward valencia" edjv@corp.earthlink.net To: toasters@mathworks.com Sent: Wednesday, February 13, 2002 2:14 PM Subject: Slow writes
Hi all,
I hope that someone has gone through this as I have been trying to figure this out and its making me go nuts. On our solaris(2.8, 2.6) we usually do small file writes, our users do cvs checkouts to thier home directories and compile software that they are working on in that directory also. The problems is not the reads but the writes, I have changed the client machines to nfs3/tcp with rsize and wsize or 32k. The avg time todo a cvs checkout has been about 24 seconds, if this is done to local disk we are seeing the avg. of 4 seconds. The size of the module they are checking out is 4mb, this is what I have been doing tests with, but other cvs modules are also slow while being written to the nfs mount home directory. The filers also have the nfs.tcp.xfersize of 32k. Here are the configs of the filer:
F760 NetApp Release 6.0.1R3: Fri Feb 16 02:42:49 PST 2001
e0: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.174.32.67 netmask 0xffffff80 broadcast 10.174.32.127 ether 00:a0:98:00:56:a3 (auto-100tx-down) e4: flags=900042<BROADCAST,RUNNING,TCPCKSUM> mtu 1500 inet 10.174.32.126 netmask 0xffffff80 broadcast 10.174.32.127 ether 00:60:cf:20:31:fb (1000fx) e6: flags=848043<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.174.32.7 netmask 0xffffff80 broadcast 10.174.32.127 ether 00:90:27:f4:5b:0e (auto-1000sx-fd-up) flowcontrol full lo: flags=948049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 9188 inet 127.0.0.1 netmask 0xff000000 broadcast 127.0.0.1
We currently only use e6.
nfs.mount_rootonly on nfs.per_client_stats.enable off nfs.tcp.enable on nfs.tcp.xfersize 32768 nfs.udp.xfersize 8192 nfs.v2.df_2gb_lim off nfs.v3.enable on nfs.webnfs.enable off nfs.webnfs.rootdir XXX nfs.webnfs.rootdir.set off
malibu> sysconfig -r Volume vol0 (root, zoned checksums)
RAID group 0RAID Disk HA.ID HA SHELF BAY CHAN Used (MB/blks) Phys (MB/blks)
parity 7.13 7 1 5 FC:A 8600/17612800 8683/17783112 data 7.14 7 1 6 FC:A 8600/17612800 8683/17783112 data 7.1 7 0 1 FC:A 8600/17612800 8683/17783112 data 7.12 7 1 4 FC:A 8600/17612800 8683/17783112
Volume vol1 (zoned checksums)
RAID group 0RAID Disk HA.ID HA SHELF BAY CHAN Used (MB/blks) Phys (MB/blks)
parity 7.3 7 0 3 FC:A 8600/17612800 8683/17783112 data 7.2 7 0 2 FC:A 8600/17612800 8683/17783112 data 7.4 7 0 4 FC:A 8600/17612800 8683/17783112 data 7.5 7 0 5 FC:A 8600/17612800 8683/17783112 data 7.6 7 0 6 FC:A 8600/17612800 8683/17783112 data 7.8 7 1 0 FC:A 8600/17612800 8683/17783112 data 7.9 7 1 1 FC:A 8600/17612800 8683/17783112 data 7.10 7 1 2 FC:A 8600/17612800 8683/17783112
Here is a copy of sysstat when doing the write to the nfs dir:
malibu> sysstat -x 1 CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk in out read write read write age hit time ty util 2% 0 0 0 0 0 0 0 0 0 0
60 100% 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
60 100% 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
60 - 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
60 100% 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
60 100% 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
60 - 0% - 0%
3% 74 0 0 74 27 15 0 0 0 0
60 100% 0% - 0%
10% 244 0 0 244 341 59 825 973 0 0
60 100% 31% T 23%
8% 252 0 0 252 1145 74 0 0 0 0
60 100% 0% - 0%
6% 240 0 0 240 537 61 8 0 0 0
60 100% 0% - 2%
6% 272 0 0 272 205 62 0 0 0 0
60 100% 0% - 0%
5% 276 0 0 276 154 62 0 0 0 0
60 100% 0% - 0%
6% 280 0 0 280 153 63 8 0 0 0
60 100% 0% - 3%
5% 273 0 0 273 260 63 0 0 0 0
60 100% 0% - 0%
5% 255 0 0 255 163 58 8 0 0 0
60 100% 0% - 2%
5% 270 0 0 270 185 61 8 0 0 0
60 100% 0% - 2%
6% 270 0 0 270 202 61 8 0 0 0
60 100% 0% - 2%
6% 274 0 0 274 362 65 0 0 0 0
60 100% 0% - 0%
13% 271 0 0 271 146 61 272 3860 0 0
60 100% 32% T 22%
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk in out read write read write age hit time ty util 5% 270 0 0 270 105 60 0 0 0 0
60 100% 0% - 0%
6% 257 0 0 257 278 60 8 0 0 0
60 100% 0% - 2%
7% 267 0 0 267 567 67 0 0 0 0
60 100% 0% - 0%
6% 287 0 0 287 236 66 0 0 0 0
60 100% 0% - 0%
5% 271 0 0 271 180 61 8 0 0 0
60 100% 0% - 2%
5% 240 0 0 240 168 55 0 0 0 0
60 100% 0% - 0%
5% 269 0 0 269 233 62 8 0 0 0
60 100% 0% - 3%
6% 272 0 0 272 208 62 0 0 0 0
60 100% 0% - 0%
5% 275 0 0 275 148 62 8 0 0 0
60 100% 0% - 2%
5% 229 0 0 229 70 51 40 0 0 0
60 99% 0% - 5%
11% 267 0 0 267 89 59 280 2627 0 0
60 100% 29% T 20%
3% 32 0 0 32 12 7 0 0 0 0
60 100% 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
60 100% 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
60 100% 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
60 - 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
60 100% 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
60 100% 0% - 0%
malibu> nfsstat 1 getattr lookup readlink read write create remove readdir{plus} 38878962 17285524 51726 2550544 6300249 2724487 2282134 2642512 0 0 0 0 0 0 13 6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 29 8 0 0 2 4 0 0 68 72 0 0 38 21 1 0 60 60 0 0 61 15 0 0 68 68 0 0 47 17 0 0 72 72 0 0 36 18 0 0 72 72 0 0 36 18 0 0 72 72 0 0 36 18 0 0 70 76 0 0 37 18 0 0 76 76 0 0 38 19 0 0 72 72 0 0 36 18 0 0 72 72 0 0 36 18 0 0 72 72 0 0 40 18 0 0 72 72 0 0 37 18 0 0 72 72 0 0 36 18 0 0 getattr lookup readlink read write create remove readdir{plus} 38879981 17286532 51726 2550544 6300803 2724743 2282148 2642518 66 68 0 0 45 17 0 0 68 72 0 0 36 17 0 0 76 76 0 0 38 19 0 0 71 74 0 0 36 18 0 0 72 72 0 0 36 18 0 0 72 72 0 0 37 18 0 0 57 77 0 0 30 18 1 0 73 83 0 0 34 24 2 0 48 57 0 0 24 15 2 0 0 0 0 0 0 0 0 0
Client settings for nfs:
/nfs/home from malibu:/vol/vol1/home Flags:
vers=3,proto=tcp,sec=sys,hard,intr,link,symlink,rsize=32768,wsize=32768,retr ans=5
/nfs/dpkg from malibu:/vol/vol1/dpkg Flags:
vers=3,proto=tcp,sec=sys,hard,intr,link,symlink,acl,rsize=32768,wsize=32768, retrans=5
The /dev/tcp settings have been left as default, and the /dev/hme settings have been changed to do fdx/non-auto. The filers are on fiber, and the clients are on 100mb, we are using a foundry bigiron 8000 to connect all the clients to the netapps.
I hope you all can help.
cheers,
edwardv-