Hi all,
I hope that someone has gone through this as I have been trying to
figure this out and its making me go nuts. On our solaris(2.8, 2.6) we
usually do small file writes, our users do cvs checkouts to thier home
directories and compile software that they are working on in that
directory also. The problems is not the reads but the writes, I have
changed the client machines to nfs3/tcp with rsize and wsize or 32k. The
avg time todo a cvs checkout has been about 24 seconds, if this is done
to local disk we are seeing the avg. of 4 seconds. The size of the
module they are checking out is 4mb, this is what I have been doing
tests with, but other cvs modules are also slow while being written to
the nfs mount home directory. The filers also have the nfs.tcp.xfersize
of 32k. Here are the configs of the filer:
F760 NetApp Release 6.0.1R3: Fri Feb 16 02:42:49 PST 2001
e0: flags=8042<BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.174.32.67 netmask 0xffffff80 broadcast 10.174.32.127
ether 00:a0:98:00:56:a3 (auto-100tx-down)
e4: flags=900042<BROADCAST,RUNNING,TCPCKSUM> mtu 1500
inet 10.174.32.126 netmask 0xffffff80 broadcast 10.174.32.127
ether 00:60:cf:20:31:fb (1000fx)
e6: flags=848043<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.174.32.7 netmask 0xffffff80 broadcast 10.174.32.127
ether 00:90:27:f4:5b:0e (auto-1000sx-fd-up) flowcontrol full
lo: flags=948049<UP,LOOPBACK,RUNNING,MULTICAST,TCPCKSUM> mtu 9188
inet 127.0.0.1 netmask 0xff000000 broadcast 127.0.0.1
We currently only use e6.
nfs.mount_rootonly on
nfs.per_client_stats.enable off
nfs.tcp.enable on
nfs.tcp.xfersize 32768
nfs.udp.xfersize 8192
nfs.v2.df_2gb_lim off
nfs.v3.enable on
nfs.webnfs.enable off
nfs.webnfs.rootdir XXX
nfs.webnfs.rootdir.set off
malibu> sysconfig -r
Volume vol0 (root, zoned checksums)
RAID group 0
RAID Disk HA.ID HA SHELF BAY CHAN Used (MB/blks) Phys (MB/blks)
--------- ----- ------------ ---- -------------- --------------
parity 7.13 7 1 5 FC:A 8600/17612800 8683/17783112
data 7.14 7 1 6 FC:A 8600/17612800 8683/17783112
data 7.1 7 0 1 FC:A 8600/17612800 8683/17783112
data 7.12 7 1 4 FC:A 8600/17612800 8683/17783112
Volume vol1 (zoned checksums)
RAID group 0
RAID Disk HA.ID HA SHELF BAY CHAN Used (MB/blks) Phys (MB/blks)
--------- ----- ------------ ---- -------------- --------------
parity 7.3 7 0 3 FC:A 8600/17612800 8683/17783112
data 7.2 7 0 2 FC:A 8600/17612800 8683/17783112
data 7.4 7 0 4 FC:A 8600/17612800 8683/17783112
data 7.5 7 0 5 FC:A 8600/17612800 8683/17783112
data 7.6 7 0 6 FC:A 8600/17612800 8683/17783112
data 7.8 7 1 0 FC:A 8600/17612800 8683/17783112
data 7.9 7 1 1 FC:A 8600/17612800 8683/17783112
data 7.10 7 1 2 FC:A 8600/17612800 8683/17783112
Here is a copy of sysstat when doing the write to the nfs dir:
malibu> sysstat -x 1
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache
Cache CP CP Disk
in out read write read write
age hit time ty util
2% 0 0 0 0 0 0 0 0 0 0
>60 100% 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
>60 100% 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
>60 - 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
>60 100% 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
>60 100% 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
>60 - 0% - 0%
3% 74 0 0 74 27 15 0 0 0 0
>60 100% 0% - 0%
10% 244 0 0 244 341 59 825 973 0 0
>60 100% 31% T 23%
8% 252 0 0 252 1145 74 0 0 0 0
>60 100% 0% - 0%
6% 240 0 0 240 537 61 8 0 0 0
>60 100% 0% - 2%
6% 272 0 0 272 205 62 0 0 0 0
>60 100% 0% - 0%
5% 276 0 0 276 154 62 0 0 0 0
>60 100% 0% - 0%
6% 280 0 0 280 153 63 8 0 0 0
>60 100% 0% - 3%
5% 273 0 0 273 260 63 0 0 0 0
>60 100% 0% - 0%
5% 255 0 0 255 163 58 8 0 0 0
>60 100% 0% - 2%
5% 270 0 0 270 185 61 8 0 0 0
>60 100% 0% - 2%
6% 270 0 0 270 202 61 8 0 0 0
>60 100% 0% - 2%
6% 274 0 0 274 362 65 0 0 0 0
>60 100% 0% - 0%
13% 271 0 0 271 146 61 272 3860 0 0
>60 100% 32% T 22%
CPU NFS CIFS HTTP Total Net kB/s Disk kB/s Tape kB/s Cache
Cache CP CP Disk
in out read write read write
age hit time ty util
5% 270 0 0 270 105 60 0 0 0 0
>60 100% 0% - 0%
6% 257 0 0 257 278 60 8 0 0 0
>60 100% 0% - 2%
7% 267 0 0 267 567 67 0 0 0 0
>60 100% 0% - 0%
6% 287 0 0 287 236 66 0 0 0 0
>60 100% 0% - 0%
5% 271 0 0 271 180 61 8 0 0 0
>60 100% 0% - 2%
5% 240 0 0 240 168 55 0 0 0 0
>60 100% 0% - 0%
5% 269 0 0 269 233 62 8 0 0 0
>60 100% 0% - 3%
6% 272 0 0 272 208 62 0 0 0 0
>60 100% 0% - 0%
5% 275 0 0 275 148 62 8 0 0 0
>60 100% 0% - 2%
5% 229 0 0 229 70 51 40 0 0 0
>60 99% 0% - 5%
11% 267 0 0 267 89 59 280 2627 0 0
>60 100% 29% T 20%
3% 32 0 0 32 12 7 0 0 0 0
>60 100% 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
>60 100% 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
>60 100% 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
>60 - 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
>60 100% 0% - 0%
2% 0 0 0 0 0 0 0 0 0 0
>60 100% 0% - 0%
malibu> nfsstat 1
getattr lookup readlink read write create remove
readdir{plus}
38878962 17285524 51726 2550544 6300249 2724487 2282134
2642512
0 0 0 0 0 0
13 6
0 0 0 0 0 0
0 0
0 0 0 0 0 0
0 0
0 0 0 0 0 0
0 0
0 0 0 0 0 0
0 0
29 8 0 0 2 4
0 0
68 72 0 0 38 21
1 0
60 60 0 0 61 15
0 0
68 68 0 0 47 17
0 0
72 72 0 0 36 18
0 0
72 72 0 0 36 18
0 0
72 72 0 0 36 18
0 0
70 76 0 0 37 18
0 0
76 76 0 0 38 19
0 0
72 72 0 0 36 18
0 0
72 72 0 0 36 18
0 0
72 72 0 0 40 18
0 0
72 72 0 0 37 18
0 0
72 72 0 0 36 18
0 0
getattr lookup readlink read write create remove
readdir{plus}
38879981 17286532 51726 2550544 6300803 2724743 2282148
2642518
66 68 0 0 45 17
0 0
68 72 0 0 36 17
0 0
76 76 0 0 38 19
0 0
71 74 0 0 36 18
0 0
72 72 0 0 36 18
0 0
72 72 0 0 37 18
0 0
57 77 0 0 30 18
1 0
73 83 0 0 34 24
2 0
48 57 0 0 24 15
2 0
0 0 0 0 0 0
0 0
Client settings for nfs:
/nfs/home from malibu:/vol/vol1/home
Flags:
vers=3,proto=tcp,sec=sys,hard,intr,link,symlink,rsize=32768,wsize=32768,retrans=5
/nfs/dpkg from malibu:/vol/vol1/dpkg
Flags:
vers=3,proto=tcp,sec=sys,hard,intr,link,symlink,acl,rsize=32768,wsize=32768,retrans=5
The /dev/tcp settings have been left as default, and the /dev/hme
settings have been changed to do fdx/non-auto. The filers are on fiber,
and the clients are on 100mb, we are using a foundry bigiron 8000 to
connect all the clients to the netapps.
I hope you all can help.
cheers,
edwardv-