toasters

toasters@lists.teaparty.net

13520 discussions

Problem to integrate F740 into a Windows 2000 domain
by benny.wistrom＠cn.saab.se 03 Jan '01

03 Jan '01

Hi! I'm having trouble to integrate a F740 into a Windows 2000 domain. When I should set the domain name when doing cifs setup it turns out that it only allows 15 characters. The problem is that my fully qualified domainname is 17 characters. I have tried to use the netbios name which only contains 11 characters but that won't work. Any suggestions? -- Regards: Benny Wiström CSC Phone: +46 13 4653647 Systemadministrator 581 88 Linköping Fax: +46 13 185771 Benny.Wistrom(a)cn.saab.se Sweden Mobile: +46 13 4653647 Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us. -Calvin & Hobbes (Bill Watterson) Talk sense to a fool and he calls you foolish. -- Euripides

1 0

Bad Solaris performance over time
by hill＠communitytelephone.com 03 Jan '01

03 Jan '01

Executive summary: Our core problem appears to be that an NFS load consisting of many file creation/removal operations eventually cause our Solaris 2.6 and Solaris 8 servers to perform very poorly. The culprit seems to be kernel R/W lock contention by the caller 'nfs_access_check' for the lock 'nfs_acache_lock'. What is unclear is what causes the lock contention. It seems to build over time eventually killing the performance of the server. Our current work around is to reboot the server every morning or when the problem occurs. See details below for further information. We recently moved /var/mail from local storage on a Solaris 2.6 server to our F760 and ever since we are suffering from serious performance problems on our Solaris server. We have about 16,000 users and deliver about 100,000 - 150,000 message a day. It is important to note that the we have a few other solaris servers that communicate with our filer and they do not experience any problems. It really seems to be related to the high volume of traffic generated by moving /var/mail over to the filer. Note that we use dot-locking only and therefore are not troubled by the well-known problems with fcntl-based locking over NFS. It is worth noting that we experienced a similar problem a few months ago with a web based email package we wrote. The email package uses form based authentication (as opposed to browser based) and kept relevant session data in a flat file (about 200 bytes big). Originally we had all 18,000 files in a single directory. We tried hashing the directory so that we had about 300 files per directory, but it did not make a difference. We then moved to using a Mysql database to store the session data, and the problem went away. Before moving to the database implementation we tried a two different physical machines (both with the characteristics defined below) as well as Solaris 2.6 and Solaris 8, none of which made a difference. It seems very clear that the symptom is brought on by a heavy load of file creation and removal. We are not sure if file size is an issue, but it could be given the size of our session files and the fact that dot-lock files are 0 bytes. We do have an open ticket with Network Appliance case# 176124 but so far we haven't made much progress with it. On to the details ... Filer: F760 Release 5.3.6R1 Solaris 2.6 Box: Hardware is AXmp, 4 360MHz UltraSPARC-II CPUs w/ 4MB cache, and 2Gig RAM mount options: vers=2/proto=udp/rsize=32768/wsize=32768/noac/remote Our server tends to perform well for about 24-30hrs depending on load. At some point after that we start seeing bad performance. We have gathered the following statistics From the Solaris Box iostat, mpstat, netstat -k hme0, netstat -k, netstat -s, nfsstat -m, nfsstat -cnr, vmstat, vmstat -i, vmstat -s, top -n -dinfinity -s60 0 From the Filer netstat -m, netstat -s, nfsstat -d, nfsstat -l, statit, sysstat The only place I saw anything out of the ordinary from the above stats when the system was working properly vs. having problems was from the output of top and mpstat. I have made all stats available at http://static-content-01.communitytelephone.net/~hill/server_stats In case anyone wanted to see them. Good performance last pid: 505; load averages: 2.26, 2.24, 2.29 14:30:04 407 processes: 393 sleeping, 3 running, 9 zombie, 2 on cpu CPU states: 32.1% idle, 40.0% user, 27.9% kernel, 0.0% iowait, 0.0% swap Memory: 2048M real, 444M free, 1000M swap free CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 516 0 505 281 159 615 23 55 17 1 1970 17 12 4 67 1 298 0 8056 125 0 810 21 63 31 1 1974 13 21 3 62 2 579 0 2056 313 100 717 15 61 33 0 6216 12 12 2 74 3 991 0 832 871 721 547 31 64 40 1 4804 18 24 5 53 Bad performance last pid: 8130; load averages: 21.64, 18.25, 13.56 18:13:05 315 processes: 301 sleeping, 6 running, 6 zombie, 2 on cpu CPU states: 1.4% idle, 28.3% user, 69.9% kernel, 0.4% iowait, 0.0% swap Memory: 2048M real, 93M free, 18M swap in use, 982M swap free CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 608 0 1943 703 366 898 180 131 24 309 2861 24 75 0 2 1 512 0 3161 336 0 936 172 140 18 330 3033 26 72 0 2 2 633 0 10496 515 100 963 168 125 34 330 2712 23 76 0 2 3 607 0 1442 1262 953 883 164 139 31 291 5426 30 68 0 2 You can see from the mpstat output that icsw, srw, usr, and sys have somewhat different characteristics between the two machine states. I found the high srw numbers particularly interesting. I believe they are responsible for the change in the icsw, usr, and sys numbers. (i.e. when there is a lot of lock contention the sys cpu utilization goes up, which could also cause the involuntary context switches (icsw) to increase.) I ran 'lockstat sleep 30' during a bad performance period to try and determine the cause of the high srw numbers. Here is what I saw R/W writer blocked by writer: 135 events Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------- 103 76% 76% 1.00 271500476 nfs_acache_lock nfs_access_cache+0x1c 23 17% 93% 1.00 311148519 nfs_acache_lock nfs_access_purge_rp+0x24 3 2% 96% 1.00 7275646 0x61a87758 ufs_remove+0x11c 2 1% 97% 1.00 17863392 0x61a87758 ufs_rename+0x2c4 2 1% 99% 1.00 19876013 0x61a87758 ufs_create+0x1a8 1 1% 99% 1.00 46461 tim_list_rwlock tim_addlink+0x64 1 1% 100% 1.00 8152217 0x61a87758 ufs_link+0x13c ------------------------------------------------------------------------- R/W writer blocked by readers: 1356 events Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------- 1135 84% 84% 13.47 142070011 nfs_acache_lock nfs_access_cache+0x1c 218 16% 100% 10.52 100639598 nfs_acache_lock nfs_access_purge_rp+0x24 2 0% 100% 1.00 603387 0x61a87758 ufs_rename+0x2c4 1 0% 100% 1.00 121137 0x61a87758 ufs_rename+0x25c ------------------------------------------------------------------------- R/W reader blocked by writer: 1608 events Count indv cuml rcnt nsec Lock Caller ---------------------------------------------------------------------- 1600 100% 100% 1.00 103571147 nfs_acache_lock nfs_access_check+0x18 7 0% 100% 1.00 13694146 0x61a87758 ufs_dirlook+0xc0 1 0% 100% 1.00 5391663 0x61a86e28 ufs_dirlook+0xc0 ---------------------------------------------------------------------- R/W reader blocked by write wanted: 36530 events Count indv cuml rcnt nsec Lock Caller ---------------------------------------------------------------------- 36530 100% 100% 17.19 55578316 nfs_acache_lock nfs_access_check+0x18 ---------------------------------------------------------------------- For grins I did the same while the machine was performing okay R/W writer blocked by writer: 11 events Count indv cuml rcnt nsec Lock Caller ---------------------------------------------------------------------- 4 36% 36% 1.00 34507 nfs_acache_lock nfs_access_cache+0x1c 3 27% 64% 1.00 41125 kas+0x1c as_map+0x6c 2 18% 82% 1.00 94435 kas+0x1c as_unmap+0x48 1 9% 91% 1.00 238723 0x61b7f720 write+0x84 1 9% 100% 1.00 8324704 0x618283a0 ufs_create+0x1a8 ---------------------------------------------------------------------- R/W writer blocked by readers: 37 events Count indv cuml rcnt nsec Lock Caller ---------------------------------------------------------------------- 24 65% 65% 1.04 114407 nfs_acache_lock nfs_access_cache+0x1c 10 27% 92% 1.00 93271 kas+0x1c as_unmap+0x48 2 5% 97% 1.00 101398 kas+0x1c as_map+0x6c 1 3% 100% 1.00 171407 0x618283a0 ufs_rename+0x2c4 ---------------------------------------------------------------------- R/W reader blocked by writer: 61 events Count indv cuml rcnt nsec Lock Caller ---------------------------------------------------------------------- 53 87% 87% 1.00 295492 nfs_acache_lock nfs_access_check+0x18 4 7% 93% 1.00 78438 kas+0x1c as_fault+0x300 2 3% 97% 1.00 8945909 0x61a45214 ufs_getattr+0x50 1 2% 98% 1.00 35317881 0x61a45208 ufs_dirlook+0xc0 1 2% 100% 1.00 3329355 0x618283a0 ufs_dirlook+0xc0 ---------------------------------------------------------------------- R/W reader blocked by write wanted: 3 events Count indv cuml rcnt nsec Lock Caller ---------------------------------------------------------------------- 3 100% 100% 1.00 164763 nfs_acache_lock nfs_access_check+0x18 ---------------------------------------------------------------------- From the percentages it seems that nfs_access_check/nfs_acache_lock is the dominant player in the srw lock contention. What I don't understand, is what causes the drastic change in lock contention over time. I've spent a few hours on sunsolve.sun.com and docs.sun.com looking for information on nfs_access_check/nfs_acache_lock and have come up empty. I have also gone through Sun Performance and Tuning Second Edition, and the toasters mailling list with no luck. At this point I'm not sure if there is a netapp incompatiliby with Solaris, or if there is a bug in Solaris NFS client implementation. If anyone out there has any ideas I would be extrememly grateful. btw: we have made the following changes to /etc/system with no noticeable change in system behaviour. * a few kernel params are related to maxusers which is by default a few * less than amount of ram in system in MB with a default limit of 1024 * by adjusting this up certain other values are also adjusted, which is the * goal here. pp. 359 Sun performance tuning, but the decision to increase * came from the Sun System Tuning white page pp. 69 set maxusers=1536 * These settings are to help tune nfs performance to the netapp -JPH * 12.19.00 per toaster article subject: NFS client tuning, problem and * resolution / Date: Tue, 10 Aug 1999 / From: Nick Christenson * npc(a)sendmail.com * set ncsize=8192 set ncsize=6144 * It appears that the increase in this number has resulted in mutex * contention viewable by mpstat (smtx) and lockstat *set nfs:nfs_nra=10 *set nfs:nfs3_nra=10 * Tweaks from SE and Sun Performance Tuning JPH 12.25.00 * http://www.sun.com/sun-on-net/performance/priority_paging.html set priority_paging=1 * Connection hash table size default 256 increased to make hash lookups * more efficient set tcp:tcp_conn_hash_size=8192 * inode cache size - can view with 'netstat -k inode_cache' * ideally this should be 4/3 the size of ncsize set ufs_ninode=8192 * Things tweaked by virtual_adrian from the se toolkit * There is also precedent for this in Sun Performance Tuning and * Solaris Internals pp. 599 set slowscan=500 set maxpgio=25468 set fastscan=64306 set handspreadpages=30960 --Jamie Hill

3 3

NFS server timeouts on the clients- on new 840's
by Premanshu Jain 03 Jan '01

03 Jan '01

Hi mike, I just migrated from 740 to 840 and face this problem on the two Sun ultra 4500 NFS clients. Here are the error messages... _________________________________ 19:48:01 mthood last message repeated 10 times Jan 2 19:49:29 mthood unix: NFS server filebot not responding still trying Jan 2 19:49:29 mthood unix: NFS server filebot ok Jan 2 20:01:43 mthood xntpd[184]: time reset (step) 0.133869 s Jan 2 20:02:46 mthood unix: NFS server filebot not responding still trying Jan 2 20:02:46 mthood unix: NFS server filebot ok Jan 2 20:05:49 mthood automountd[135]: filebot:/vol/sw/.view.dat: No such file or directory Jan 2 20:05:49 mthood last message repeated 1 time Jan 2 20:07:42 mthood unix: NFS server filebot not responding still trying Jan 2 20:07:42 mthood unix: NFS server filebot ok Jan 2 20:08:42 mthood unix: NFS server filebot not responding still trying Jan 2 20:08:42 mthood unix: NFS server filebot ok _______________________ Can you bail me out of this issues. Both Sun systems are now running very-very slow . Thanks prem -----Original Message----- From: Mike Smith [mailto:mikesmit@netapp.com] Sent: Sunday, December 31, 2000 4:15 PM To: arr(a)oceanwave.com; toasters(a)mathworks.com Subject: Re: NFS server timeouts on the clients Based upon the description there is no way to tell exactly what the problem is. First off if you are using 10baseT then the 2M transfer will take 2 seconds. If you are using 100baseT (full duplex) then it will take 0.2 seconds. If you are using 10baseT then the transfer time is correct. If you are using 100baseT (full duplex) then I would take a look at the duplex settings on the filer side and then the switch side. If either side is not set exactly as the other side then it is possible you have a duplex mismatch. In which case I would simply manually set the port that the filer is connected to to full duplex. Then set the filer to full duplex manually: ifconfig e0 mediatype 100tx-fd Try the transfer again. You should see favorable results. Note: sometimes autonegotiation works ok but for the most part a lot of us stick to using manually set 100tx-fd on both the switch side and the filer side. Hope that helps. If not you might try contacting NetApp customer support and opening a case Mike Smith NetApp Escalations Engineer. mikesmit(a)netapp.com ----- Original Message ----- From: <arr(a)oceanwave.com> To: <toasters(a)mathworks.com> Sent: Sunday, December 31, 2000 3:10 PM Subject: NFS server timeouts on the clients > I have an brand new F720 running 5.3.6R2. We're exporting two qtrees via NFS > and CIFS. When the system is almost completely quiet, I try to tar up about > 2M of stuff and it takes FIVE minutes. I also see NFS timeout errors on all > NFS clients (the one I'm doing the tar from is specifically solaris 8, but I > see the timeouts on linux as well). > > I've tried NFS v2 and v3, both TCP and UDP, and nothing seems to help. netstat > doesn't seem to show packet collisions (the netowrk ispretty much dead), not a > lot of errors (<10%). I'm seeing writes about every 10 seconds (which should > mean good performance), and the CPU is mostly idle. Why the hell does a tar > of 2M take so bloody long? I can understand if I was moving a gig or so, but > 2M?! > > I've included important info (this was after I had changed to v2 UDP). Is > there something else I should look at? > > > >nfsstat -c > > Server rpc: > TCP: > calls badcalls nullrecv badlen xdrcall > 0 0 0 0 0 > > UDP: > calls badcalls nullrecv badlen xdrcall > 1354 0 0 0 0 > > Server nfs: > calls badcalls > 1306 0 > > Server nfs V2: (1306 calls) > null getattr setattr root lookup readlink read > 1 0% 372 28% 2 0% 0 0% 194 15% 0 0% 293 22% > wrcache write create remove rename link symlink > 0 0% 440 34% 2 0% 1 0% 0 0% 0 0% 0 0% > mkdir rmdir readdir statfs > 0 0% 0 0% 1 0% 0 0% > > Server nfs V3: (0 calls) > null getattr setattr lookup access readlink read > 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% > write create mkdir symlink mknod remove rmdir > 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% > rename link readdir readdir+ fsstat fsinfo pathconf > 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% > commit > 0 0% > > NFS V2 non-blocking request statistics: > null getattr setattr root lookup readlink read > 1 100% 372 100% 2 100% 0 0% 194 100% 0 0% 141 48% > wrcache write create remove rename link symlink > 0 0% 440 100% 1 50% 1 100% 0 0% 0 0% 0 0% > mkdir rmdir readdir statfs > 0 0% 0 0% 1 100% 0 0% > > NFS V3 non-blocking request statistics: > null getattr setattr lookup access readlink read > 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% > write create mkdir symlink mknod remove rmdir > 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% > rename link readdir readdir+ fsstat fsinfo pathconf > 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% > > NFS reply cache statistics: > TCP: > In progress Delay hits Misses Idempotent Non-idempotent > 0 0 0 0 0 > UDP: > In progress Delay hits Misses Idempotent Non-idempotent > 0 5 732 7 42 > > > >sysstat 1 > > CPU NFS CIFS HTTP Net kB/s Disk kB/s Tape kB/s Cache > in out read write read write age > 2% 0 0 0 0 0 0 0 0 0 19 > 3% 1 0 0 0 1 0 0 0 0 19 > 2% 0 0 0 0 0 0 0 0 0 19 > 3% 32 0 0 5 3 0 0 0 0 19 > 2% 0 0 0 0 0 0 0 0 0 19 > 6% 0 0 0 0 0 48 128 0 0 19 > 2% 0 0 0 31 0 0 0 0 0 19 > 2% 0 0 0 0 0 0 0 0 0 19 > 2% 0 0 0 0 0 0 0 0 0 19 > 3% 4 0 0 2 1 0 0 0 0 19 > 2% 0 0 0 0 0 0 0 0 0 19 > 2% 0 0 0 0 0 0 0 0 0 19 > 2% 1 0 0 16 0 0 0 0 0 19 > 2% 0 0 0 0 0 0 0 0 0 19 > 2% 0 0 0 0 0 0 0 0 0 19 > 3% 2 0 0 1 1 0 0 0 0 19 > 5% 0 0 0 0 0 32 152 0 0 19 > 2% 0 0 0 0 0 0 0 0 0 19 > > > > >sysconfig -r > > Volume vol0 (root) > > RAID group 0 > > RAID Disk HA.ID HA SHELF BAY CHAN Used (MB/blks) Phys (MB/blks) > --------- ----- ------------ ---- -------------- ---------- ---- > parity 1.3 1 0 3 FC:A 34500/70656000 35003/71687368 > data 1.2 1 0 2 FC:A 34500/70656000 35003/71687368 > data 1.0 1 0 0 FC:A 34500/70656000 35003/71687368 > data 1.5 1 0 5 FC:A 34500/70656000 35003/71687368 > data 1.6 1 0 6 FC:A 34500/70656000 35003/71687368 > data 1.1 1 0 1 FC:A 34500/70656000 35003/71687368 > > Spare disks > > RAID Disk HA.ID HA SHELF BAY CHAN Used (MB/blks) Phys (MB/blks) > --------- ----- ------------ ---- -------------- ---------- ---- > spare 1.4 1 0 4 FC:A 0 35003/71687368 > > > > netstat -s > > tcp: > 19180 packets sent > 16501 data packets (3116859 bytes) > 647 data packets (355343 bytes) retransmitted > 804 ack-only packets (114 delayed) > 0 URG only packets > 0 window probe packets > 1156 window update packets > 76 control packets > 22021 packets received > 15990 acks (for 3112800 bytes) > 187 duplicate acks > 0 acks for unsent data > 13688 packets (5178316 bytes) received in-sequence > 618 completely duplicate packets (890780 bytes) > 0 old duplicate packets > 10 packets with some dup. data (10 bytes duped) > 50 out-of-order packets (61320 bytes) > 0 packets (0 bytes) of data after window > 0 window probes > 0 window update packets > 1 packet received after close > 0 discarded for bad checksums > 0 discarded for bad header offset fields > 0 discarded because packet too short > 13 connection requests > 27 connection accepts > 3 bad connection attempts > 0 listen queue overflows > 32 connections established (including accepts) > 40 connections closed (including 1 drop) > 8 embryonic connections dropped > 15825 segments updated rtt (of 16329 attempts) > 531 retransmit timeouts > 0 connections dropped by rexmit timeout > 0 persist timeouts > 0 connections timed out in persist > 8 keepalive timeouts > 0 keepalive probes sent > 8 connections dropped by keepalive > 3594 correct ACK header predictions > 3474 correct data packet header predictions > 1758 PCB cache misses > 0 segments dropped at untrusted interface > 0 connection requests closed at filter > udp: > 3671023 datagrams received > 0 with incomplete header > 0 with bad data length field > 0 with bad checksum > 0 dropped due to no socket > 137 broadcast/multicast datagrams dropped due to no socket > 0 dropped due to full socket buffers > 3670886 delivered > 3665023 datagrams output > ip: > 7921585 total packets received > 0 bad header checksums > 0 with size smaller than minimum > 0 with size larger than maximum > 0 with data size < data length > 0 with header length < data size > 0 with data length < header length > 0 with bad options > 0 with incorrect version number > 0 packets with spoofed source address > 0 packets arrived on wrong port > 6499598 fragments received > 0 fragments dropped (dup or out of space) > 0 malformed fragments dropped > 0 overlapping fragments discarded > 3966 fragments dropped after timeout > 2272723 packets reassembled ok > 3693053 packets for this host > 790 packets for unknown/unsupported protocol > 0 packets forwarded > 867 packets not forwardable > 0 redirects sent > 3685059 packets sent from this host > 0 packets sent with fabricated ip header > 0 output packets dropped due to no bufs, etc. > 0 output packets discarded due to no route > 412593 output datagrams fragmented > 862312 fragments created > 0 datagrams that can't be fragmented > icmp: > 0 calls to icmp_error > 0 errors not generated 'cuz old message was icmp > Output histogram: > echo reply: 7 > 0 messages with bad code fields > 0 messages < minimum length > 0 bad checksums > 0 messages with bad length > Input histogram: > echo reply: 2 > echo: 7 > time exceeded: 790 > 7 message responses generated > > > > netstat -i > > Name Mtu Network Address Ipkts Ierrs Opkts > Oerrs Collis Queue > e8* 1500 none none 0 0 0 > 0 0 0 > e0 1500 192.168.1 myhost 7929745 5917 > 4 0 with incorrect version number > 0 packets with spoofed source address > 0 packets arrived on wrong port > 6499598 fragments received > 0 fragments dropped (dup or out of space) > 0 malformed fragments dropped > 0 overlapping fragments discarded > 3966 fragments dropped after timeout > 2272723 packets reassembled ok > 3693053 packets for this host > 790 packets for unknown/unsupported protocol > 0 packets forwarded > 867 packets not forwardable > 0 redirects sent > 3685059 packets sent from this host > 0 packets sent with fabricated ip header > 0 output packets dropped due to no bufs, etc. > 0 output packets discarded due to no route > 412593 output datagrams fragmented > 862312 fragments created > 0 datagrams that can't be fragmented > icmp: > 0 calls to icmp_error > 0 errors not generated 'cuz old message was icmp > Output histogram: > echo reply: 7 > 0 messages with bad code fields > 0 messages < minimum length > 0 bad checksums > 0 messages with bad length > Input histogram: > echo reply: 2 > echo: 7 > time exceeded: 790 > 7 message responses generated > > > >

1 0

RE: Bad Solaris performance over time
by Premanshu Jain 03 Jan '01

03 Jan '01

Hi toasters, I just migrated from 740to 840 and ran into NFS time out problems on Sun 4500's with 2.7 on them. Severe issues. Slowing down the server to crap... Any inputs.. prem -----Original Message----- From: hill(a)communitytelephone.com [mailto:hill@communitytelephone.com] Sent: Tuesday, January 02, 2001 3:00 PM To: toasters(a)mathworks.com Cc: access-systems(a)communitytelephone.net Subject: Bad Solaris performance over time Executive summary: Our core problem appears to be that an NFS load consisting of many file creation/removal operations eventually cause our Solaris 2.6 and Solaris 8 servers to perform very poorly. The culprit seems to be kernel R/W lock contention by the caller 'nfs_access_check' for the lock 'nfs_acache_lock'. What is unclear is what causes the lock contention. It seems to build over time eventually killing the performance of the server. Our current work around is to reboot the server every morning or when the problem occurs. See details below for further information. We recently moved /var/mail from local storage on a Solaris 2.6 server to our F760 and ever since we are suffering from serious performance problems on our Solaris server. We have about 16,000 users and deliver about 100,000 - 150,000 message a day. It is important to note that the we have a few other solaris servers that communicate with our filer and they do not experience any problems. It really seems to be related to the high volume of traffic generated by moving /var/mail over to the filer. Note that we use dot-locking only and therefore are not troubled by the well-known problems with fcntl-based locking over NFS. It is worth noting that we experienced a similar problem a few months ago with a web based email package we wrote. The email package uses form based authentication (as opposed to browser based) and kept relevant session data in a flat file (about 200 bytes big). Originally we had all 18,000 files in a single directory. We tried hashing the directory so that we had about 300 files per directory, but it did not make a difference. We then moved to using a Mysql database to store the session data, and the problem went away. Before moving to the database implementation we tried a two different physical machines (both with the characteristics defined below) as well as Solaris 2.6 and Solaris 8, none of which made a difference. It seems very clear that the symptom is brought on by a heavy load of file creation and removal. We are not sure if file size is an issue, but it could be given the size of our session files and the fact that dot-lock files are 0 bytes. We do have an open ticket with Network Appliance case# 176124 but so far we haven't made much progress with it. On to the details ... Filer: F760 Release 5.3.6R1 Solaris 2.6 Box: Hardware is AXmp, 4 360MHz UltraSPARC-II CPUs w/ 4MB cache, and 2Gig RAM mount options: vers=2/proto=udp/rsize=32768/wsize=32768/noac/remote Our server tends to perform well for about 24-30hrs depending on load. At some point after that we start seeing bad performance. We have gathered the following statistics From the Solaris Box iostat, mpstat, netstat -k hme0, netstat -k, netstat -s, nfsstat -m, nfsstat -cnr, vmstat, vmstat -i, vmstat -s, top -n -dinfinity -s60 0 From the Filer netstat -m, netstat -s, nfsstat -d, nfsstat -l, statit, sysstat The only place I saw anything out of the ordinary from the above stats when the system was working properly vs. having problems was from the output of top and mpstat. I have made all stats available at http://static-content-01.communitytelephone.net/~hill/server_stats In case anyone wanted to see them. Good performance last pid: 505; load averages: 2.26, 2.24, 2.29 14:30:04 407 processes: 393 sleeping, 3 running, 9 zombie, 2 on cpu CPU states: 32.1% idle, 40.0% user, 27.9% kernel, 0.0% iowait, 0.0% swap Memory: 2048M real, 444M free, 1000M swap free CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 516 0 505 281 159 615 23 55 17 1 1970 17 12 4 67 1 298 0 8056 125 0 810 21 63 31 1 1974 13 21 3 62 2 579 0 2056 313 100 717 15 61 33 0 6216 12 12 2 74 3 991 0 832 871 721 547 31 64 40 1 4804 18 24 5 53 Bad performance last pid: 8130; load averages: 21.64, 18.25, 13.56 18:13:05 315 processes: 301 sleeping, 6 running, 6 zombie, 2 on cpu CPU states: 1.4% idle, 28.3% user, 69.9% kernel, 0.4% iowait, 0.0% swap Memory: 2048M real, 93M free, 18M swap in use, 982M swap free CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 608 0 1943 703 366 898 180 131 24 309 2861 24 75 0 2 1 512 0 3161 336 0 936 172 140 18 330 3033 26 72 0 2 2 633 0 10496 515 100 963 168 125 34 330 2712 23 76 0 2 3 607 0 1442 1262 953 883 164 139 31 291 5426 30 68 0 2 You can see from the mpstat output that icsw, srw, usr, and sys have somewhat different characteristics between the two machine states. I found the high srw numbers particularly interesting. I believe they are responsible for the change in the icsw, usr, and sys numbers. (i.e. when there is a lot of lock contention the sys cpu utilization goes up, which could also cause the involuntary context switches (icsw) to increase.) I ran 'lockstat sleep 30' during a bad performance period to try and determine the cause of the high srw numbers. Here is what I saw R/W writer blocked by writer: 135 events Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------- 103 76% 76% 1.00 271500476 nfs_acache_lock nfs_access_cache+0x1c 23 17% 93% 1.00 311148519 nfs_acache_lock nfs_access_purge_rp+0x24 3 2% 96% 1.00 7275646 0x61a87758 ufs_remove+0x11c 2 1% 97% 1.00 17863392 0x61a87758 ufs_rename+0x2c4 2 1% 99% 1.00 19876013 0x61a87758 ufs_create+0x1a8 1 1% 99% 1.00 46461 tim_list_rwlock tim_addlink+0x64 1 1% 100% 1.00 8152217 0x61a87758 ufs_link+0x13c ------------------------------------------------------------------------- R/W writer blocked by readers: 1356 events Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------- 1135 84% 84% 13.47 142070011 nfs_acache_lock nfs_access_cache+0x1c 218 16% 100% 10.52 100639598 nfs_acache_lock nfs_access_purge_rp+0x24 2 0% 100% 1.00 603387 0x61a87758 ufs_rename+0x2c4 1 0% 100% 1.00 121137 0x61a87758 ufs_rename+0x25c ------------------------------------------------------------------------- R/W reader blocked by writer: 1608 events Count indv cuml rcnt nsec Lock Caller ---------------------------------------------------------------------- 1600 100% 100% 1.00 103571147 nfs_acache_lock nfs_access_check+0x18 7 0% 100% 1.00 13694146 0x61a87758 ufs_dirlook+0xc0 1 0% 100% 1.00 5391663 0x61a86e28 ufs_dirlook+0xc0 ---------------------------------------------------------------------- R/W reader blocked by write wanted: 36530 events Count indv cuml rcnt nsec Lock Caller ---------------------------------------------------------------------- 36530 100% 100% 17.19 55578316 nfs_acache_lock nfs_access_check+0x18 ---------------------------------------------------------------------- For grins I did the same while the machine was performing okay R/W writer blocked by writer: 11 events Count indv cuml rcnt nsec Lock Caller ---------------------------------------------------------------------- 4 36% 36% 1.00 34507 nfs_acache_lock nfs_access_cache+0x1c 3 27% 64% 1.00 41125 kas+0x1c as_map+0x6c 2 18% 82% 1.00 94435 kas+0x1c as_unmap+0x48 1 9% 91% 1.00 238723 0x61b7f720 write+0x84 1 9% 100% 1.00 8324704 0x618283a0 ufs_create+0x1a8 ---------------------------------------------------------------------- R/W writer blocked by readers: 37 events Count indv cuml rcnt nsec Lock Caller ---------------------------------------------------------------------- 24 65% 65% 1.04 114407 nfs_acache_lock nfs_access_cache+0x1c 10 27% 92% 1.00 93271 kas+0x1c as_unmap+0x48 2 5% 97% 1.00 101398 kas+0x1c as_map+0x6c 1 3% 100% 1.00 171407 0x618283a0 ufs_rename+0x2c4 ---------------------------------------------------------------------- R/W reader blocked by writer: 61 events Count indv cuml rcnt nsec Lock Caller ---------------------------------------------------------------------- 53 87% 87% 1.00 295492 nfs_acache_lock nfs_access_check+0x18 4 7% 93% 1.00 78438 kas+0x1c as_fault+0x300 2 3% 97% 1.00 8945909 0x61a45214 ufs_getattr+0x50 1 2% 98% 1.00 35317881 0x61a45208 ufs_dirlook+0xc0 1 2% 100% 1.00 3329355 0x618283a0 ufs_dirlook+0xc0 ---------------------------------------------------------------------- R/W reader blocked by write wanted: 3 events Count indv cuml rcnt nsec Lock Caller ---------------------------------------------------------------------- 3 100% 100% 1.00 164763 nfs_acache_lock nfs_access_check+0x18 ---------------------------------------------------------------------- From the percentages it seems that nfs_access_check/nfs_acache_lock is the dominant player in the srw lock contention. What I don't understand, is what causes the drastic change in lock contention over time. I've spent a few hours on sunsolve.sun.com and docs.sun.com looking for information on nfs_access_check/nfs_acache_lock and have come up empty. I have also gone through Sun Performance and Tuning Second Edition, and the toasters mailling list with no luck. At this point I'm not sure if there is a netapp incompatiliby with Solaris, or if there is a bug in Solaris NFS client implementation. If anyone out there has any ideas I would be extrememly grateful. btw: we have made the following changes to /etc/system with no noticeable change in system behaviour. * a few kernel params are related to maxusers which is by default a few * less than amount of ram in system in MB with a default limit of 1024 * by adjusting this up certain other values are also adjusted, which is the * goal here. pp. 359 Sun performance tuning, but the decision to increase * came from the Sun System Tuning white page pp. 69 set maxusers=1536 * These settings are to help tune nfs performance to the netapp -JPH * 12.19.00 per toaster article subject: NFS client tuning, problem and * resolution / Date: Tue, 10 Aug 1999 / From: Nick Christenson * npc(a)sendmail.com * set ncsize=8192 set ncsize=6144 * It appears that the increase in this number has resulted in mutex * contention viewable by mpstat (smtx) and lockstat *set nfs:nfs_nra=10 *set nfs:nfs3_nra=10 * Tweaks from SE and Sun Performance Tuning JPH 12.25.00 * http://www.sun.com/sun-on-net/performance/priority_paging.html set priority_paging=1 * Connection hash table size default 256 increased to make hash lookups * more efficient set tcp:tcp_conn_hash_size=8192 * inode cache size - can view with 'netstat -k inode_cache' * ideally this should be 4/3 the size of ncsize set ufs_ninode=8192 * Things tweaked by virtual_adrian from the se toolkit * There is also precedent for this in Sun Performance Tuning and * Solaris Internals pp. 599 set slowscan=500 set maxpgio=25468 set fastscan=64306 set handspreadpages=30960 --Jamie Hill

1 0

RE: SnapManager
by Tom "Mad Dog" Yergeau 02 Jan '01

02 Jan '01

Linda, I would do the migration like this: 1. Install your new Exchange server, as a new server, into the existing site 2. Move the databases (empty) to the filer 3. Migrate your mailboxes and public folders one at a time to the new Exchange server 4. Follow the procedure for removing the first server in an Exchange site (it's a TechNet article, http://support.microsoft.com/support/kb/articles/Q152/9/59.asp ) 5. Turn off the old server This is also known as the Ed Crowley server move method (why it's named after Ed I don't know, he didn't invent it). You can read more about it at http://www.mailmover.com/resource/exch_faq_appxa.htm It's more transparent to your users than one big switchover, less complicated (though more time consuming) than trying to back up and then restore the databases to the new server, and it lets you test the new server with just a few users on it as opposed to moving all your users at once. If you have any questions hit me off list and I'll try to help. MD -----Original Message----- From: Loux, Linda [mailto:LLoux@rii.com] Sent: Tuesday, January 02, 2001 2:17 PM To: 'toasters(a)mathworks.com' Subject: SnapManager I am upgrading the server hardware for my Exchange server. My current Exchange server is running NT 4.0, and the databases are on local raid disks. My new hardware will be running Windows 2000, and the databases will be on my filer. I am having trouble trying to accomplish this, since the Migrator assumes the server is not changing. My new server will not have enough local disk space to put the Information Store on it first and then do the migration. Does anyone have the correct procedures for doing this operation? Linda Loux Systems Administrator Rosetta Inpharmatics ********************************************************************** This email message is the property of Rosetta Inpharmatics and contains confidential information that may also be privileged. If you are not the intended recipient, any use, disclosure, copying, or communication of the contents of this message is prohibited. If this message was received in error, please forward a copy to mail_admin(a)rii.com and delete the message and any attachments from your computer. Thank you. **********************************************************************

1 0

RE: SnapManager
by Daniel Oxenhandler 02 Jan '01

02 Jan '01

Linda, I don't know if this is the fastest way, but couldn't you first move the existing Information Store to the filer using the Exchange Optimizer program (perfwiz.exe), and then do the Windows/Exchange 2000 upgrade? Sounds like you're trying to do the upgrade and IS move at the same time which may not be supported. Regards, Daniel Oxenhandler Network Engineer Sprint Laboratories 650-375-4281 -----Original Message----- From: Loux, Linda [mailto:LLoux@rii.com] Sent: Tuesday, January 02, 2001 11:17 AM To: 'toasters(a)mathworks.com' Subject: SnapManager I am upgrading the server hardware for my Exchange server. My current Exchange server is running NT 4.0, and the databases are on local raid disks. My new hardware will be running Windows 2000, and the databases will be on my filer. I am having trouble trying to accomplish this, since the Migrator assumes the server is not changing. My new server will not have enough local disk space to put the Information Store on it first and then do the migration. Does anyone have the correct procedures for doing this operation? Linda Loux Systems Administrator Rosetta Inpharmatics ********************************************************************** This email message is the property of Rosetta Inpharmatics and contains confidential information that may also be privileged. If you are not the intended recipient, any use, disclosure, copying, or communication of the contents of this message is prohibited. If this message was received in error, please forward a copy to mail_admin(a)rii.com and delete the message and any attachments from your computer. Thank you. **********************************************************************

1 0

SnapManager
by Loux, Linda 02 Jan '01

02 Jan '01

I am upgrading the server hardware for my Exchange server. My current Exchange server is running NT 4.0, and the databases are on local raid disks. My new hardware will be running Windows 2000, and the databases will be on my filer. I am having trouble trying to accomplish this, since the Migrator assumes the server is not changing. My new server will not have enough local disk space to put the Information Store on it first and then do the migration. Does anyone have the correct procedures for doing this operation? Linda Loux Systems Administrator Rosetta Inpharmatics ********************************************************************** This email message is the property of Rosetta Inpharmatics and contains confidential information that may also be privileged. If you are not the intended recipient, any use, disclosure, copying, or communication of the contents of this message is prohibited. If this message was received in error, please forward a copy to mail_admin(a)rii.com and delete the message and any attachments from your computer. Thank you. **********************************************************************

1 0

NFS server timeouts on the clients
by arr＠oceanwave.com 02 Jan '01

02 Jan '01

I have an brand new F720 running 5.3.6R2. We're exporting two qtrees via NFS and CIFS. When the system is almost completely quiet, I try to tar up about 2M of stuff and it takes FIVE minutes. I also see NFS timeout errors on all NFS clients (the one I'm doing the tar from is specifically solaris 8, but I see the timeouts on linux as well). I've tried NFS v2 and v3, both TCP and UDP, and nothing seems to help. netstat doesn't seem to show packet collisions (the netowrk ispretty much dead), not a lot of errors (<10%). I'm seeing writes about every 10 seconds (which should mean good performance), and the CPU is mostly idle. Why the hell does a tar of 2M take so bloody long? I can understand if I was moving a gig or so, but 2M?! I've included important info (this was after I had changed to v2 UDP). Is there something else I should look at? >nfsstat -c Server rpc: TCP: calls badcalls nullrecv badlen xdrcall 0 0 0 0 0 UDP: calls badcalls nullrecv badlen xdrcall 1354 0 0 0 0 Server nfs: calls badcalls 1306 0 Server nfs V2: (1306 calls) null getattr setattr root lookup readlink read 1 0% 372 28% 2 0% 0 0% 194 15% 0 0% 293 22% wrcache write create remove rename link symlink 0 0% 440 34% 2 0% 1 0% 0 0% 0 0% 0 0% mkdir rmdir readdir statfs 0 0% 0 0% 1 0% 0 0% Server nfs V3: (0 calls) null getattr setattr lookup access readlink read 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% write create mkdir symlink mknod remove rmdir 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% rename link readdir readdir+ fsstat fsinfo pathconf 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% commit 0 0% NFS V2 non-blocking request statistics: null getattr setattr root lookup readlink read 1 100% 372 100% 2 100% 0 0% 194 100% 0 0% 141 48% wrcache write create remove rename link symlink 0 0% 440 100% 1 50% 1 100% 0 0% 0 0% 0 0% mkdir rmdir readdir statfs 0 0% 0 0% 1 100% 0 0% NFS V3 non-blocking request statistics: null getattr setattr lookup access readlink read 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% write create mkdir symlink mknod remove rmdir 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% rename link readdir readdir+ fsstat fsinfo pathconf 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% NFS reply cache statistics: TCP: In progress Delay hits Misses Idempotent Non-idempotent 0 0 0 0 0 UDP: In progress Delay hits Misses Idempotent Non-idempotent 0 5 732 7 42 >sysstat 1 CPU NFS CIFS HTTP Net kB/s Disk kB/s Tape kB/s Cache in out read write read write age 2% 0 0 0 0 0 0 0 0 0 19 3% 1 0 0 0 1 0 0 0 0 19 2% 0 0 0 0 0 0 0 0 0 19 3% 32 0 0 5 3 0 0 0 0 19 2% 0 0 0 0 0 0 0 0 0 19 6% 0 0 0 0 0 48 128 0 0 19 2% 0 0 0 31 0 0 0 0 0 19 2% 0 0 0 0 0 0 0 0 0 19 2% 0 0 0 0 0 0 0 0 0 19 3% 4 0 0 2 1 0 0 0 0 19 2% 0 0 0 0 0 0 0 0 0 19 2% 0 0 0 0 0 0 0 0 0 19 2% 1 0 0 16 0 0 0 0 0 19 2% 0 0 0 0 0 0 0 0 0 19 2% 0 0 0 0 0 0 0 0 0 19 3% 2 0 0 1 1 0 0 0 0 19 5% 0 0 0 0 0 32 152 0 0 19 2% 0 0 0 0 0 0 0 0 0 19 >sysconfig -r Volume vol0 (root) RAID group 0 RAID Disk HA.ID HA SHELF BAY CHAN Used (MB/blks) Phys (MB/blks) --------- ----- ------------ ---- -------------- -------------- parity 1.3 1 0 3 FC:A 34500/70656000 35003/71687368 data 1.2 1 0 2 FC:A 34500/70656000 35003/71687368 data 1.0 1 0 0 FC:A 34500/70656000 35003/71687368 data 1.5 1 0 5 FC:A 34500/70656000 35003/71687368 data 1.6 1 0 6 FC:A 34500/70656000 35003/71687368 data 1.1 1 0 1 FC:A 34500/70656000 35003/71687368 Spare disks RAID Disk HA.ID HA SHELF BAY CHAN Used (MB/blks) Phys (MB/blks) --------- ----- ------------ ---- -------------- -------------- spare 1.4 1 0 4 FC:A 0 35003/71687368 > netstat -s tcp: 19180 packets sent 16501 data packets (3116859 bytes) 647 data packets (355343 bytes) retransmitted 804 ack-only packets (114 delayed) 0 URG only packets 0 window probe packets 1156 window update packets 76 control packets 22021 packets received 15990 acks (for 3112800 bytes) 187 duplicate acks 0 acks for unsent data 13688 packets (5178316 bytes) received in-sequence 618 completely duplicate packets (890780 bytes) 0 old duplicate packets 10 packets with some dup. data (10 bytes duped) 50 out-of-order packets (61320 bytes) 0 packets (0 bytes) of data after window 0 window probes 0 window update packets 1 packet received after close 0 discarded for bad checksums 0 discarded for bad header offset fields 0 discarded because packet too short 13 connection requests 27 connection accepts 3 bad connection attempts 0 listen queue overflows 32 connections established (including accepts) 40 connections closed (including 1 drop) 8 embryonic connections dropped 15825 segments updated rtt (of 16329 attempts) 531 retransmit timeouts 0 connections dropped by rexmit timeout 0 persist timeouts 0 connections timed out in persist 8 keepalive timeouts 0 keepalive probes sent 8 connections dropped by keepalive 3594 correct ACK header predictions 3474 correct data packet header predictions 1758 PCB cache misses 0 segments dropped at untrusted interface 0 connection requests closed at filter udp: 3671023 datagrams received 0 with incomplete header 0 with bad data length field 0 with bad checksum 0 dropped due to no socket 137 broadcast/multicast datagrams dropped due to no socket 0 dropped due to full socket buffers 3670886 delivered 3665023 datagrams output ip: 7921585 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with size larger than maximum 0 with data size < data length 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 packets with spoofed source address 0 packets arrived on wrong port 6499598 fragments received 0 fragments dropped (dup or out of space) 0 malformed fragments dropped 0 overlapping fragments discarded 3966 fragments dropped after timeout 2272723 packets reassembled ok 3693053 packets for this host 790 packets for unknown/unsupported protocol 0 packets forwarded 867 packets not forwardable 0 redirects sent 3685059 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 412593 output datagrams fragmented 862312 fragments created 0 datagrams that can't be fragmented icmp: 0 calls to icmp_error 0 errors not generated 'cuz old message was icmp Output histogram: echo reply: 7 0 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length Input histogram: echo reply: 2 echo: 7 time exceeded: 790 7 message responses generated > netstat -i Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Collis Queue e8* 1500 none none 0 0 0 0 0 0 e0 1500 192.168.1 myhost 7929745 5917 4 0 with incorrect version number 0 packets with spoofed source address 0 packets arrived on wrong port 6499598 fragments received 0 fragments dropped (dup or out of space) 0 malformed fragments dropped 0 overlapping fragments discarded 3966 fragments dropped after timeout 2272723 packets reassembled ok 3693053 packets for this host 790 packets for unknown/unsupported protocol 0 packets forwarded 867 packets not forwardable 0 redirects sent 3685059 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 0 output packets discarded due to no route 412593 output datagrams fragmented 862312 fragments created 0 datagrams that can't be fragmented icmp: 0 calls to icmp_error 0 errors not generated 'cuz old message was icmp Output histogram: echo reply: 7 0 messages with bad code fields 0 messages < minimum length 0 bad checksums 0 messages with bad length Input histogram: echo reply: 2 echo: 7 time exceeded: 790 7 message responses generated

2 3

01-01-2001
by Oldies.Online.Casino＠mathworks.co.uk 02 Jan '01

02 Jan '01

1 0

ESTALE
by Edward Hibbert 31 Dec '00

31 Dec '00

We're using HP machines running HP 11 against a NetApp, using NFS. We are occasionally seeing ESTALE errors on file I/O operations. This may be a problem in the application, but I'd welcome any information any of you have on just what ESTALE means. I've seen it described as occurring when an NFS client uses an invalid file handle, but I've also seen reference to it occurring when machines are rebooted. So, my rather vague question is: what do you all know about ESTALE? Edward Hibbert Internet Applications Group Data Connection Ltd Tel: +44 131 662 1212 Fax: +44 131 662 1345 Email: eh(a)dataconnection.com Web: http://www.dataconnection.com

2 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters