-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Well we are actually getting better writes from tcp, which is weird due to the small overhead that tcp would have. Here is what I have currently set on a 2.8 client:
/etc/system:
set nstrpush = 90 set ncsize = 8000 set nfs:nfs3_max_threads = 34 set nfs:nfs3_nra = 24 set nfs:nfs3_bsize = 65536 set nfs:nfs3_max_transfer_size = 65536
set nfs:nfs_max_threads = 24 set nfs:nfs_nra = 10
/dev/tcp:
ndd -set /dev/tcp tcp_recv_hiwat 65535 ndd -set /dev/tcp tcp_xmit_hiwat 65535
/dev/hme: transceiver_inuse 0 ipg1 8 ipg2 4 use_int_xcvr 0 pace_size 0 adv_autoneg_cap 0 adv_100T4_cap 0 adv_100fdx_cap 1 adv_100hdx_cap 0 adv_10fdx_cap 0 adv_10hdx_cap 0 autoneg_cap 1 100T4_cap 0 100fdx_cap 1 100hdx_cap 1 10fdx_cap 1 10hdx_cap 1 lp_autoneg_cap 0 lp_100T4_cap 0 lp_100fdx_cap 0 lp_100hdx_cap 0 lp_10fdx_cap 0 lp_10hdx_cap 0 instance 0 lance_mode 1 ipg0 16
Currently I'm getting about 6secs writting to the nfs mounts, and about 4secs writting to local disk. I dont know if I can get anymore from the nfs mounts, but this is using my tests which are about 4mb of small files being written. Do you think that this still can be improved?
Below are the difference on the filer: malibu> sysstat -x 1 CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk in out read write read write age hit time ty util 16% 709 0 0 709 2390 214 32 0 0 0
60 99% 0% - 5%
23% 1695 0 0 1695 1423 420 44 0 0 0
60 100% 0% - 2%
24% 1792 0 0 1792 1406 442 32 0 0 0
60 100% 0% - 5%
23% 1562 0 0 1562 1753 397 48 0 0 0
60 100% 0% - 3%
14% 1024 0 0 1024 429 244 64 0 0 0
60 99% 0% - 5%
2% 0 0 0 0 0 0 0 0 0 0
60 100% 0% - 0%
The nfs ops have increased, the only thing I do see if I do a large cvs checkout the writes are intermitent. I see zeros and then I see a large write, below is an example. Now is this due to the filer flushing what it has on memory to disk?
22% 1319 0 0 1319 1498 336 78 0 0 0
60 100% 2% F 7%
33% 1062 0 0 1062 1319 274 657 10387 0 0
60 99% 58% : 48%
malibu> sysstat -x 1 CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk in out read write read write age hit time ty util 2% 0 0 0 0 0 0 0 0 0 0
60 100% 0% - 0%
5% 102 0 0 102 20 23 164 384 0 0
60 99% 10% T 9%
7% 369 0 0 369 334 92 80 76 0 0
60 98% 15% : 15%
15% 1060 0 0 1060 616 257 56 0 0 0
60 99% 0% - 6%
20% 1343 0 0 1343 1483 340 56 0 0 0
60 99% 0% - 5%
20% 1195 0 0 1195 2047 318 60 0 0 0
60 99% 0% - 6%
21% 1498 0 0 1498 1251 371 72 0 0 0
60 99% 0% - 7%
23% 1904 0 0 1904 859 457 44 0 0 0
60 100% 0% - 6%
20% 1653 0 0 1653 752 397 60 0 0 0
60 99% 0% - 5%
22% 1436 0 0 1436 1754 368 72 0 0 0
60 99% 0% - 7%
22% 1319 0 0 1319 1498 336 78 0 0 0
60 100% 2% F 7%
33% 1062 0 0 1062 1319 274 657 10387 0 0
60 99% 58% : 48%
21% 1441 0 0 1441 1470 364 80 0 0 0
60 99% 0% - 5%
19% 1475 0 0 1475 724 356 116 0 0 0
60 99% 0% - 7%
21% 1563 0 0 1563 1137 384 56 0 0 0
60 99% 0% - 8%
3% 9 0 0 9 2 2 0 0 0 0
60 100% 0% - 0%
edwardv-
- ---------------------------------------------------------------------- - --------------------------------------------------- These are my opionions and my own, these do not reflect what my employer may have or belives in. - ---------------------------------------------------------------------- - ---------------------------------------------------
- ----- Original Message ----- From: "Magnus Swenson" magnuss@cadence.com To: "FPS France" fpsfrance@fps.fr; "edward valencia" edjv@corp.earthlink.net; toasters@mathworks.com Sent: Thursday, February 21, 2002 7:38 AM Subject: RE: Slow writes
Edward,
Try to disable TCP over NFS and only enable UDP. Also in a switched environment 10/100/1000 BT keep the xfersizes to 8192 this has proven to improve performance (at least for us). This is a known bug on certain switches, it's also listed on now.netapp.com.
Re, Magnus Swenson Cadence Design Systems Inc. Addison, TX
-----Original Message----- From: FPS France [mailto:fpsfrance@fps.fr] Sent: Thursday, February 21, 2002 3:59 AM To: edward valencia; toasters@mathworks.com Subject: Re: Slow writes
just an idea : did u well check on 100Mb machines (solaris) that they all are in full dupleix or all in half , as - as u may know as u look a guru - mixing the two types can greatlly disturb network performance. Check that in both solaris machines, on the switch, and on other machines in the same subnet. U could try to switch a machine in half dupleix just to c what happend. In any case, make sure u avoid auto negoiciation for 100Mb nic - it's ok for the giga.
Hope u will turn around this problem. Fanku to let me know.
----- Original Message ----- From: "edward valencia" edjv@corp.earthlink.net To: toasters@mathworks.com Sent: Wednesday, February 13, 2002 2:14 PM Subject: Slow writes
Hi all,
I hope that someone has gone through this as I have been trying to figure this out and its making me go nuts. On our solaris(2.8, 2.6) we usually do small file writes, our users do cvs checkouts to thier home directories and compile software that they are working on in that directory also. The problems is not the reads but the writes, I have changed the client machines to nfs3/tcp with rsize and wsize or 32k. The avg time todo a cvs checkout has been about 24 seconds, if this is done to local disk we are seeing the avg. of 4 seconds. The size of the module they are checking out is 4mb, this is what I have been doing tests with, but other cvs modules are also slow while being written to the nfs mount home directory. The filers also have the nfs.tcp.xfersize of 32k. Here are the configs of the filer:
F760 NetApp Release 6.0.1R3: Fri Feb 16 02:42:49 PST 2001
e0: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.174.32.67 netmask 0xffffff80 broadcast 10.174.32.127 ether 00:a0:98:00:56:a3 (auto-100tx-down) e4: flags=900042<BROADCAST,RUNNING,TCPCKSUM> mtu 1500 inet 10.174.32.126 netmask 0xffffff80 broadcast 10.174.32.127 ether 00:60:cf:20:31:fb (1000fx) e6: flags=848043<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.174.32.7 netmask 0xffffff80 broadcast 10.174.32.127 ether 00:90:27:f4:5b:0e (auto-1000sx-fd-up) flowcontrol full lo: flags=948049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 9188 inet 127.0.0.1 netmask 0xff000000 broadcast 127.0.0.1
We currently only use e6.
nfs.mount_rootonly on nfs.per_client_stats.enable off nfs.tcp.enable on nfs.tcp.xfersize 32768 nfs.udp.xfersize 8192 nfs.v2.df_2gb_lim off nfs.v3.enable on nfs.webnfs.enable off nfs.webnfs.rootdir XXX nfs.webnfs.rootdir.set off
malibu> sysconfig -r Volume vol0 (root, zoned checksums)
RAID group 0RAID Disk HA.ID HA SHELF BAY CHAN Used (MB/blks) Phys (MB/blks)
parity 7.13 7 1 5 FC:A 8600/17612800 8683/17783112 data 7.14 7 1 6 FC:A 8600/17612800 8683/17783112 data 7.1 7 0 1 FC:A 8600/17612800 8683/17783112 data 7.12 7 1 4 FC:A 8600/17612800 8683/17783112
Volume vol1 (zoned checksums)
RAID group 0RAID Disk HA.ID HA SHELF BAY CHAN Used (MB/blks) Phys (MB/blks)
parity 7.3 7 0 3 FC:A 8600/17612800 8683/17783112 data 7.2 7 0 2 FC:A 8600/17612800 8683/17783112 data 7.4 7 0 4 FC:A 8600/17612800 8683/17783112 data 7.5 7 0 5 FC:A 8600/17612800 8683/17783112 data 7.6 7 0 6 FC:A 8600/17612800 8683/17783112 data 7.8 7 1 0 FC:A 8600/17612800 8683/17783112 data 7.9 7 1 1 FC:A 8600/17612800 8683/17783112 data 7.10 7 1 2 FC:A 8600/17612800 8683/17783112
Here is a copy of sysstat when doing the write to the nfs dir:
malibu> sysstat -x 1 CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk in out read write read write age hit time ty util 2% 0 0 0 0 0 0 0 0 0 0
60 100% 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
60 100% 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
60 - 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
60 100% 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
60 100% 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
60 - 0% - 0%
3% 74 0 0 74 27 15 0 0 0 0
60 100% 0% - 0%
10% 244 0 0 244 341 59 825 973 0 0
60 100% 31% T 23%
8% 252 0 0 252 1145 74 0 0 0 0
60 100% 0% - 0%
6% 240 0 0 240 537 61 8 0 0 0
60 100% 0% - 2%
6% 272 0 0 272 205 62 0 0 0 0
60 100% 0% - 0%
5% 276 0 0 276 154 62 0 0 0 0
60 100% 0% - 0%
6% 280 0 0 280 153 63 8 0 0 0
60 100% 0% - 3%
5% 273 0 0 273 260 63 0 0 0 0
60 100% 0% - 0%
5% 255 0 0 255 163 58 8 0 0 0
60 100% 0% - 2%
5% 270 0 0 270 185 61 8 0 0 0
60 100% 0% - 2%
6% 270 0 0 270 202 61 8 0 0 0
60 100% 0% - 2%
6% 274 0 0 274 362 65 0 0 0 0
60 100% 0% - 0%
13% 271 0 0 271 146 61 272 3860 0 0
60 100% 32% T 22%
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache Cache CP CP Disk in out read write read write age hit time ty util 5% 270 0 0 270 105 60 0 0 0 0
60 100% 0% - 0%
6% 257 0 0 257 278 60 8 0 0 0
60 100% 0% - 2%
7% 267 0 0 267 567 67 0 0 0 0
60 100% 0% - 0%
6% 287 0 0 287 236 66 0 0 0 0
60 100% 0% - 0%
5% 271 0 0 271 180 61 8 0 0 0
60 100% 0% - 2%
5% 240 0 0 240 168 55 0 0 0 0
60 100% 0% - 0%
5% 269 0 0 269 233 62 8 0 0 0
60 100% 0% - 3%
6% 272 0 0 272 208 62 0 0 0 0
60 100% 0% - 0%
5% 275 0 0 275 148 62 8 0 0 0
60 100% 0% - 2%
5% 229 0 0 229 70 51 40 0 0 0
60 99% 0% - 5%
11% 267 0 0 267 89 59 280 2627 0 0
60 100% 29% T 20%
3% 32 0 0 32 12 7 0 0 0 0
60 100% 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
60 100% 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
60 100% 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
60 - 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
60 100% 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
60 100% 0% - 0%
malibu> nfsstat 1 getattr lookup readlink read write create remove readdir{plus} 38878962 17285524 51726 2550544 6300249 2724487 2282134 2642512 0 0 0 0 0 0 13 6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 29 8 0 0 2 4 0 0 68 72 0 0 38 21 1 0 60 60 0 0 61 15 0 0 68 68 0 0 47 17 0 0 72 72 0 0 36 18 0 0 72 72 0 0 36 18 0 0 72 72 0 0 36 18 0 0 70 76 0 0 37 18 0 0 76 76 0 0 38 19 0 0 72 72 0 0 36 18 0 0 72 72 0 0 36 18 0 0 72 72 0 0 40 18 0 0 72 72 0 0 37 18 0 0 72 72 0 0 36 18 0 0 getattr lookup readlink read write create remove readdir{plus} 38879981 17286532 51726 2550544 6300803 2724743 2282148 2642518 66 68 0 0 45 17 0 0 68 72 0 0 36 17 0 0 76 76 0 0 38 19 0 0 71 74 0 0 36 18 0 0 72 72 0 0 36 18 0 0 72 72 0 0 37 18 0 0 57 77 0 0 30 18 1 0 73 83 0 0 34 24 2 0 48 57 0 0 24 15 2 0 0 0 0 0 0 0 0 0
Client settings for nfs:
/nfs/home from malibu:/vol/vol1/home Flags:
vers=3,proto=tcp,sec=sys,hard,intr,link,symlink,rsize=32768,wsize=32 768,retr ans=5
/nfs/dpkg from malibu:/vol/vol1/dpkg Flags:
vers=3,proto=tcp,sec=sys,hard,intr,link,symlink,acl,rsize=32768,wsiz e=32768, retrans=5
The /dev/tcp settings have been left as default, and the /dev/hme settings have been changed to do fdx/non-auto. The filers are on fiber, and the clients are on 100mb, we are using a foundry bigiron 8000 to connect all the clients to the netapps.
I hope you all can help.
cheers,
edwardv-