toasters January 2001

toasters@lists.teaparty.net

135 participants
157 discussions

RE: Sun photon (fcal) disks on toaster?
by Traitel, Eyal 04 Jan '01

04 Jan '01

Games like this could be fun indeed, but I doubt if our code supports them... Anyway - you can always try to boot a regular PC with our Intel platform floppies... :) Eyal. ---------------------------------------------------------------------- eTraitel - I'm the new eBuzzword around !!! ---------------------------------------------------------------------- Eyal Traitel - Filer Escalation Engineer CNA, MCSE, CSA, NACA Network Appliance BV Holland Office Center Kruisweg 799b 2132 NG, Hoofddorp The Netherlands Office: +31 23 567 9685 Cellular: +31 6 5497 2568 Email: eyal(a)netapp.com ---------------------------------------------------------------------- Get answers NOW! - NetApp On the Web - http://now.netapp.com ---------------------------------------------------------------------- -----Original Message----- From: Jason S. Cash [mailto:cash@udel.edu] Sent: Thursday, January 04, 2001 2:48 PM To: toasters(a)mathworks.com Subject: Sun photon (fcal) disks on toaster? Has anyone successfully hooked Sun photon disk arrays to a netapp? I have a bunch of these sitting around at the moment a f740 that it might be amusing to see how fast they can go (if at all). Management might be willing to buy us some 840s to act as heads for these arrays if it will work :) TIA, Jason /* Jason S. Cash - Systems Programmer IT/Network and Systems Services University of Delaware, Newark Delaware */

1 0

Sun photon (fcal) disks on toaster?
by Jason S. Cash 04 Jan '01

04 Jan '01

Has anyone successfully hooked Sun photon disk arrays to a netapp? I have a bunch of these sitting around at the moment a f740 that it might be amusing to see how fast they can go (if at all). Management might be willing to buy us some 840s to act as heads for these arrays if it will work :) TIA, Jason /* Jason S. Cash - Systems Programmer IT/Network and Systems Services University of Delaware, Newark Delaware */

1 0

Filer + Netscape Messaging Server
by Brian L. Brush 03 Jan '01

03 Jan '01

We are considering using a filer to provide the message stores for our Netscape (soon to be iPlanet) Messaging Server. Of course, neither Netscape (iPlanet) nor NetApp supports such a configuration. Can anyone offer relevant insight or experience? For a couple of years we have been using a similarly unsupported configuration to provide storage space for Rational ClearCase. While we've had no insuperable problems, I don't relish the prospect of hitting a brick wall on those rare occasions when we request support--for the application, not the configuration. --Brian L. Brush Senior Systems Administrator Paradyne Corporation

1 0

Re: Bad Solaris performance over time
by hill＠communitytelephone.com 03 Jan '01

03 Jan '01

On Tue, 2 Jan 2001, Chris Lamb wrote: ->One box handled incoming SMTP and all local delivery (sendmail+procmail) ->and POP, the other handled some unholy number of Majordomo lists. I mean, ->this was with hardware that nowadays people scoff at, running ancient ->software releases and NFS V2 over a single network interface. I find it ->impossible to believe that a 4-way UltraII and an F760 can't handle the ->load. (Yeah, the average mail message with all the crap "modern" mailers ->add on is probably several Kb larger, but that's almost never the real ->bottleneck.) That was supposed to be part of my point, the machine we have can more the adequetly handle the load. :) The stats http://static-content-01.communitytelephone.net/~hill/server_stats show that the machine doesn't breathe hard all day long. It seems that something happens over time and at some point hits critical mass, which causes the performance to become very bad. -> ->Some questions that may be a bit obvious: Is your 2.6 box patched to ->current levels? Yes. Patch revisions are posted at the above URL. ->Why NFS V2? This was a typo, it is really V3. ->How much NVRAM in the filer? 32Mb. The filer weekly_log is at the above URL. ->Is the network bogged down, or the switch misconfigured? We see no errors on the switch, or the servers network interfaces. ->How many network interfaces, of what type/speed? Filer has a Gigabit interface. Solaris servers has 100Mb. You can see the throughput information at the above URL file: nx.se. ->What does snoop/tcpdump/your network sniffer say? I haven't tried this yet. It looked like there was going to be way too much information to sort through. I will gather it the next time the system begins to act up. ->How many disks in the volume that contains /var/mail? 14 ->Is this filer running CIFS & NFS, or is it purely an NFS box? NFS only ->If /var/mail is on the filer, what about home directories (i.e., is ->your mailer bogging doing ".forward" lookups)? home directories are on the netapp as well. Again I must state it is not a general performance problem, but something that happens over time. -> Are you using NIS and/or DNS on the filer? No. ->What version of sendmail/qmail/whatever? Latest smail. ->Are you using procmail for delivery? Why not? :-) Yes. ->Is disk scrubbing kicking in? Are you taking snapshots too frequently? ->Do you have cron jobs walking the filesystem or other network clients ->hitting the filer or the network when these slowdowns are observed on ->the ->mail server? And really, why NFS V2? I appreciate what you are getting at here, but I don't believe that sort of thing is happening. It you look at the stats for the filer the performance of it stays stable through the trouble period with the one server. Also, all other servers continue to perform well. I believe the server stats show that things like these aren't occuring as well. We are using V3 mounts I typo'd my earlier email. ->(I'm not really a smartass, I just play one on the Internet. :-) These ->are just some of the questions I'd look to answer before poking at ->/etc/system. We did do that. We feel stronghly that there is a bug in Solaris or in the interaction between it and the netapp which cause our performance problems to occur. We tried to massage /etc/system parameters to keep from occuring. Again, I must point out our server ran great when we have /var/mail on local storage. We only began to see this problem after moving it to the filer and only after 24 - 30 hours of operation. I have found a bug ID on sunsolve that described similarities to our problem, but we already have the patch that was supposed to fix that bug. I include it here so you can get a feel for the actual problem from someone elses perspective. Even though our problem is not an exact match we do have similarities with it. We also have similarities to a posting on toasters Date: Tue, 10 Aug 1999 08:05:18 -0700 (PDT) From: Nick Christenson <npc(a)sendmail.com> To: toasters(a)mathworks.com Subject: NFS client tuning, problem and resolution. ------------------------ Sun Bug ID 4034003 --------------------------- Some time ago I got email from someone on a system w/ 37,000 users who had problem using mail from it's mailserver, both CPU were very busy and the system had slowed to a crawl at peak times. The systems gets +/- 70K sendmail calls, 130K pop3 calls a day, mail lives ona NFSv3 capable network appliance server. Mail is delivered to the home directories, not to /var/mail. He ran kernel profiling and found crcmp() topping the list: granularity: each sample hit covers 2 byte(s) for 0.12% of 8.65 seconds % cumulative self self total time seconds seconds calls ms/call ms/call name 39.2 3.39 3.39 crcmp [1] 2.8 3.63 0.24 locked_pgcopy [2] 2.5 3.85 0.22 splx [3] 1.8 4.01 0.16 bcopy [4] 1.7 4.16 0.15 mutex_enter [5] 1.7 4.31 0.15 disp_getwork [6] 1.6 4.45 0.14 dnlc_purge_vp [7] 1.5 4.58 0.13 nfs3_access [8] 1.5 4.71 0.13 ross625_vac_segflush [9] 1.3 4.82 0.11 blkclr [10] 1.3 4.93 0.11 idle [11] 1.0 5.02 0.09 mutex_adaptive_enter [12] While the information is not complete, here's what I think happens, all users access the same rnode ("/export/home", mail is delivered to the home directies) a large access cache is built and here's where we serialize:, in nfs3_access() : [common/fs/nfs/nfs3_vnops.c] ........ mutex_enter(&rp->r_statelock); for (acp = rp->r_acc; acp != NULL; acp = acp->next) { /* * Look for an entry by comparing credentials. */ if (crcmp(acp->cred, cr) == 0) { if ((acp->known & acc) == acc) { #ifdef DEBUG nfs3_access_cache_hits++; #endif if ((acp->allowed & acc) == acc) { mutex_exit(&rp->r_statelock); return (0); } mutex_exit(&rp->r_statelock); return (EACCES); } break; } } mutex_exit(&rp->r_statelock); And further down, if we have a cache miss, we do the same all over again. (As we unlock, we need to recheck whether a new entry was added, so the list is rechecked, again holding the r_statelock) Now, that algorithm works fine in the typical case where users access their own files; or whena few users do. It does *not* work when you have 37K users accessing a single file. As long as the node is cached and the kernel doesn't reap kernel memory, the list will grow out of bounds and access to this node will get increasingly slow. This theory was tested by switching to NFSv2 mounts. This fixed the problem. Attached is a program that shows the behaviour somewhat. It calls "access()" on a file (specified on the command line) as many different users. The file must exist on a NFSv3 filesystem, the program must be run as root and the file must be openable as root on the client. One sample run printed this: 0 18860 500 5680 1000 4798 1500 3963 2000 3071 2500 2425 3000 1552 3500 863 and this tells that after running access(file, F_OK) as 4000 different uids, there's a really high penalty for the first users that called access. In this case for uid(0) searching the cache costs 18ms. It seems that this cache should not be allowed to grow out of bounds, currently it's ownly limited by kernel memory. -Work Around- Use NFSv2 mounts on filesystems that are used by lots of users such as "/var/mail". michael.eisler@Eng 1997-02-24 The above aork around is fine as long as the server is not a Solaris 2.5[.1] or 2.6 system. Since the ACL protocol has an ACCESS procedure for NFS V2, a Solaris 2.5 or alter system will also use the ACL cache system. ---------------------- End Sun Bug ID 4034003 ------------------------- We did try using V2 mounts as a test, and we still experienced the problem. --Jamie Hill -> ->-- Chris -> ->-- ->Chris Lamb, Unix Guy ->MeasureCast, Inc. ->503-241-1469 x247 -><skeezics(a)measurecast.com> -> ->

1 0

Problem to integrate F740 into a Windows 2000 domain (2)
by benny.wistrom＠cn.saab.se 03 Jan '01

03 Jan '01

Hi again! I have tried once again and I get the following messages when I try to integrate my filer into a Windows 2000 domain. CIFS server is beginning domain installation. CIFS server is locating PDC. Wed Jan 3 12:03:31 GMT [rc]: Connection with DC \\ZAMIRA established Wed Jan 3 12:03:31 GMT [rc]: Connection with DC \\ZAMIRA terminated Wed Jan 3 12:03:31 GMT [rc]: CIFS - Filer retrying authentication with backup password. Wed Jan 3 12:03:33 GMT [rc]: Connection with DC \\ZAMIRA established Wed Jan 3 12:03:33 GMT [rc]: Connection with DC \\ZAMIRA terminated Wed Jan 3 12:03:33 GMT [rc]: CIFS - Filer retrying installation. Wed Jan 3 12:03:35 GMT [rc]: Connection with DC \\ZAMIRA established CIFS could not start. See log. CIFS local server is shutting down... Wed Jan 3 12:03:35 GMT [rc]: Connection with DC \\ZAMIRA terminated zam> Wed Jan 3 12:03:35 GMT [rc]: CIFS - Filer's security information differs from domain controller. Wed Jan 3 12:03:36 GMT [nbns_timer]: All CIFS name registrations complete for local server What does the message "CIFS - Filer's security information differs from domain controller." mean? Any suggestions? -- Regards: Benny Wiström CSC Phone: +46 13 4653647 Systemadministrator 581 88 Linköping Fax: +46 13 185771 Benny.Wistrom(a)cn.saab.se Sweden Mobile: +46 13 4653647 Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us. -Calvin & Hobbes (Bill Watterson) Talk sense to a fool and he calls you foolish. -- Euripides

2 1

RE: NFS server timeouts on the clients- on new 840's
by Fox, Adam 03 Jan '01

03 Jan '01

After checking the duplex settings (and that can't be stressed enough), you may also want to check the the rsize and wsize of your NFS connections. There was a change right around 5.3.5 (I believe) that changed the default rsize and wsize on the server from 8192 to 32768. Some networks have trouble with the larger transaction sizes and it causes timeouts. A way to test this would be to do an NFS mount from your sun with the mount options rsize=8192,wsize=8192. If this helps, then that's probably your issue. You can then tell the filer to change the size on the server side with options nfs.udp.xfersize This info can be found at: http://now.netapp.com/Knowledgebase/solutionarea.asp?id=ba668eb2-9bbd-11d4-9 2f7-080020c8339b&resource=&number=5&isExternal=0 Just another thing to check. -- Adam Fox NetApp Professional Services, NC adamfox(a)netapp.com > -----Original Message----- > From: Krause, Oliver [mailto:OKrause@netapp.com] > Sent: Wednesday, January 03, 2001 6:07 AM > To: 'Premanshu Jain'; Smith, (CS) Michael > Cc: 'arr(a)oceanwave.com'; 'toasters(a)mathworks.com' > Subject: RE: NFS server timeouts on the clients- on new 840's > > > Looks like a common half-/full-duplex mismatch. Check your > network. On the > filer check your interface with ifstat <interface> and look > for CRC errors. > Always run your filer AND switchport in 100b-tx (not auto). > Sometimes this > connection is done right and people forget to check between > the switch and > the client ... > > Oliver > > > -----Original Message----- > > From: Premanshu Jain [mailto:PrJain@shastanets.com] > > Sent: Mittwoch, 3. Januar 2001 05:15 > > To: 'Mike Smith' > > Cc: 'arr(a)oceanwave.com'; 'toasters(a)mathworks.com' > > Subject: NFS server timeouts on the clients- on new 840's > > > > > > Hi mike, > > I just migrated from 740 to 840 and face this problem on the > > two Sun ultra > > 4500 NFS clients. Here are the error messages... > > _________________________________ > > 19:48:01 mthood last message repeated 10 times > > Jan 2 19:49:29 mthood unix: NFS server filebot not > > responding still trying > > Jan 2 19:49:29 mthood unix: NFS server filebot ok > > Jan 2 20:01:43 mthood xntpd[184]: time reset (step) 0.133869 s > > Jan 2 20:02:46 mthood unix: NFS server filebot not > > responding still trying > > Jan 2 20:02:46 mthood unix: NFS server filebot ok > > Jan 2 20:05:49 mthood automountd[135]: > > filebot:/vol/sw/.view.dat: No such > > file or directory > > Jan 2 20:05:49 mthood last message repeated 1 time > > Jan 2 20:07:42 mthood unix: NFS server filebot not > > responding still trying > > Jan 2 20:07:42 mthood unix: NFS server filebot ok > > Jan 2 20:08:42 mthood unix: NFS server filebot not > > responding still trying > > Jan 2 20:08:42 mthood unix: NFS server filebot ok > > _______________________ > > Can you bail me out of this issues. Both Sun systems are now running > > very-very slow . > > Thanks > > prem > > > > -----Original Message----- > > From: Mike Smith [mailto:mikesmit@netapp.com] > > Sent: Sunday, December 31, 2000 4:15 PM > > To: arr(a)oceanwave.com; toasters(a)mathworks.com > > Subject: Re: NFS server timeouts on the clients > > > > > > Based upon the description there is no way to tell exactly > > what the problem is. First off if you are using 10baseT > > then the 2M transfer will take 2 seconds. If you are using > > 100baseT (full duplex) then it will take 0.2 seconds. > > > > If you are using 10baseT then the transfer time is correct. > > If you are using 100baseT (full duplex) then I would take > > a look at the duplex settings on the filer side and then the > > switch side. If either side is not set exactly as the other side > > then it is possible you have a duplex mismatch. In which case > > I would simply manually set the port that the filer is connected > > to to full duplex. Then set the filer to full duplex manually: > > > > ifconfig e0 mediatype 100tx-fd > > > > Try the transfer again. You should see favorable results. > > Note: sometimes autonegotiation works ok but for the most > > part a lot of us stick to using manually set 100tx-fd on both the > > switch side and the filer side. Hope that helps. > > If not you might try contacting NetApp customer support and opening > > a case > > > > Mike Smith > > NetApp Escalations Engineer. > > mikesmit(a)netapp.com > > > > ----- Original Message ----- > > From: <arr(a)oceanwave.com> > > To: <toasters(a)mathworks.com> > > Sent: Sunday, December 31, 2000 3:10 PM > > Subject: NFS server timeouts on the clients > > > > > > > I have an brand new F720 running 5.3.6R2. We're exporting > > two qtrees via > > NFS > > > and CIFS. When the system is almost completely quiet, I > > try to tar up > > about > > > 2M of stuff and it takes FIVE minutes. I also see NFS > > timeout errors on > > all > > > NFS clients (the one I'm doing the tar from is specifically > > solaris 8, but > > I > > > see the timeouts on linux as well). > > > > > > I've tried NFS v2 and v3, both TCP and UDP, and nothing > > seems to help. > > netstat > > > doesn't seem to show packet collisions (the netowrk > > ispretty much dead), > > not a > > > lot of errors (<10%). I'm seeing writes about every 10 > > seconds (which > > should > > > mean good performance), and the CPU is mostly idle. Why > > the hell does a > > tar > > > of 2M take so bloody long? I can understand if I was > > moving a gig or so, > > but > > > 2M?! > > > > > > I've included important info (this was after I had changed > > to v2 UDP). Is > > > there something else I should look at? > > > > > > > > > >nfsstat -c > > > > > > Server rpc: > > > TCP: > > > calls badcalls nullrecv badlen xdrcall > > > 0 0 0 0 0 > > > > > > UDP: > > > calls badcalls nullrecv badlen xdrcall > > > 1354 0 0 0 0 > > > > > > Server nfs: > > > calls badcalls > > > 1306 0 > > > > > > Server nfs V2: (1306 calls) > > > null getattr setattr root lookup > > readlink read > > > 1 0% 372 28% 2 0% 0 0% 194 15% 0 0% > > 293 22% > > > wrcache write create remove rename link > > symlink > > > 0 0% 440 34% 2 0% 1 0% 0 0% 0 0% > > 0 0% > > > mkdir rmdir readdir statfs > > > 0 0% 0 0% 1 0% 0 0% > > > > > > Server nfs V3: (0 calls) > > > null getattr setattr lookup access > > readlink read > > > 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% > > 0 0% > > > write create mkdir symlink mknod > > remove rmdir > > > 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% > > 0 0% > > > rename link readdir readdir+ fsstat > > fsinfo pathconf > > > 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% > > 0 0% > > > commit > > > 0 0% > > > > > > NFS V2 non-blocking request statistics: > > > null getattr setattr root lookup > > readlink read > > > 1 100% 372 100% 2 100% 0 0% 194 100% 0 0% > > 141 48% > > > wrcache write create remove rename link > > symlink > > > 0 0% 440 100% 1 50% 1 100% 0 0% 0 0% > > 0 0% > > > mkdir rmdir readdir statfs > > > 0 0% 0 0% 1 100% 0 0% > > > > > > NFS V3 non-blocking request statistics: > > > null getattr setattr lookup access > > readlink read > > > 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% > > 0 0% > > > write create mkdir symlink mknod > > remove rmdir > > > 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% > > 0 0% > > > rename link readdir readdir+ fsstat > > fsinfo pathconf > > > 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% > > 0 0% > > > > > > NFS reply cache statistics: > > > TCP: > > > In progress Delay hits Misses Idempotent > > Non-idempotent > > > 0 0 0 0 0 > > > UDP: > > > In progress Delay hits Misses Idempotent > > Non-idempotent > > > 0 5 732 7 42 > > > > > > > > > >sysstat 1 > > > > > > CPU NFS CIFS HTTP Net kB/s Disk kB/s Tape kB/s > > Cache > > > in out read write > read write > > age > > > 2% 0 0 0 0 0 0 0 > 0 0 > > 19 > > > 3% 1 0 0 0 1 0 0 > 0 0 > > 19 > > > 2% 0 0 0 0 0 0 0 > 0 0 > > 19 > > > 3% 32 0 0 5 3 0 0 > 0 0 > > 19 > > > 2% 0 0 0 0 0 0 0 > 0 0 > > 19 > > > 6% 0 0 0 0 0 48 128 > 0 0 > > 19 > > > 2% 0 0 0 31 0 0 0 > 0 0 > > 19 > > > 2% 0 0 0 0 0 0 0 > 0 0 > > 19 > > > 2% 0 0 0 0 0 0 0 > 0 0 > > 19 > > > 3% 4 0 0 2 1 0 0 > 0 0 > > 19 > > > 2% 0 0 0 0 0 0 0 > 0 0 > > 19 > > > 2% 0 0 0 0 0 0 0 > 0 0 > > 19 > > > 2% 1 0 0 16 0 0 0 > 0 0 > > 19 > > > 2% 0 0 0 0 0 0 0 > 0 0 > > 19 > > > 2% 0 0 0 0 0 0 0 > 0 0 > > 19 > > > 3% 2 0 0 1 1 0 0 > 0 0 > > 19 > > > 5% 0 0 0 0 0 32 152 > 0 0 > > 19 > > > 2% 0 0 0 0 0 0 0 > 0 0 > > 19 > > > > > > > > > > > > >sysconfig -r > > > > > > Volume vol0 (root) > > > > > > RAID group 0 > > > > > > RAID Disk HA.ID HA SHELF BAY CHAN Used > (MB/blks) Phys > > (MB/blks) > > > --------- ----- ------------ ---- > > -------------- ---------- > > ---- > > > parity 1.3 1 0 3 FC:A 34500/70656000 > > 35003/71687368 > > > data 1.2 1 0 2 FC:A 34500/70656000 > > 35003/71687368 > > > data 1.0 1 0 0 FC:A 34500/70656000 > > 35003/71687368 > > > data 1.5 1 0 5 FC:A 34500/70656000 > > 35003/71687368 > > > data 1.6 1 0 6 FC:A 34500/70656000 > > 35003/71687368 > > > data 1.1 1 0 1 FC:A 34500/70656000 > > 35003/71687368 > > > > > > Spare disks > > > > > > RAID Disk HA.ID HA SHELF BAY CHAN Used > (MB/blks) Phys > > (MB/blks) > > > --------- ----- ------------ ---- > > -------------- ---------- > > ---- > > > spare 1.4 1 0 4 FC:A 0 > > 35003/71687368 > > > > > > > > > > netstat -s > > > > > > tcp: > > > 19180 packets sent > > > 16501 data packets (3116859 bytes) > > > 647 data packets (355343 bytes) retransmitted > > > 804 ack-only packets (114 delayed) > > > 0 URG only packets > > > 0 window probe packets > > > 1156 window update packets > > > 76 control packets > > > 22021 packets received > > > 15990 acks (for 3112800 bytes) > > > 187 duplicate acks > > > 0 acks for unsent data > > > 13688 packets (5178316 bytes) received in-sequence > > > 618 completely duplicate packets (890780 bytes) > > > 0 old duplicate packets > > > 10 packets with some dup. data (10 bytes duped) > > > 50 out-of-order packets (61320 bytes) > > > 0 packets (0 bytes) of data after window > > > 0 window probes > > > 0 window update packets > > > 1 packet received after close > > > 0 discarded for bad checksums > > > 0 discarded for bad header offset fields > > > 0 discarded because packet too short > > > 13 connection requests > > > 27 connection accepts > > > 3 bad connection attempts > > > 0 listen queue overflows > > > 32 connections established (including accepts) > > > 40 connections closed (including 1 drop) > > > 8 embryonic connections dropped > > > 15825 segments updated rtt (of 16329 attempts) > > > 531 retransmit timeouts > > > 0 connections dropped by rexmit timeout > > > 0 persist timeouts > > > 0 connections timed out in persist > > > 8 keepalive timeouts > > > 0 keepalive probes sent > > > 8 connections dropped by keepalive > > > 3594 correct ACK header predictions > > > 3474 correct data packet header predictions > > > 1758 PCB cache misses > > > 0 segments dropped at untrusted interface > > > 0 connection requests closed at filter > > > udp: > > > 3671023 datagrams received > > > 0 with incomplete header > > > 0 with bad data length field > > > 0 with bad checksum > > > 0 dropped due to no socket > > > 137 broadcast/multicast datagrams dropped due to no socket > > > 0 dropped due to full socket buffers > > > 3670886 delivered > > > 3665023 datagrams output > > > ip: > > > 7921585 total packets received > > > 0 bad header checksums > > > 0 with size smaller than minimum > > > 0 with size larger than maximum > > > 0 with data size < data length > > > 0 with header length < data size > > > 0 with data length < header length > > > 0 with bad options > > > 0 with incorrect version number > > > 0 packets with spoofed source address > > > 0 packets arrived on wrong port > > > 6499598 fragments received > > > 0 fragments dropped (dup or out of space) > > > 0 malformed fragments dropped > > > 0 overlapping fragments discarded > > > 3966 fragments dropped after timeout > > > 2272723 packets reassembled ok > > > 3693053 packets for this host > > > 790 packets for unknown/unsupported protocol > > > 0 packets forwarded > > > 867 packets not forwardable > > > 0 redirects sent > > > 3685059 packets sent from this host > > > 0 packets sent with fabricated ip header > > > 0 output packets dropped due to no bufs, etc. > > > 0 output packets discarded due to no route > > > 412593 output datagrams fragmented > > > 862312 fragments created > > > 0 datagrams that can't be fragmented > > > icmp: > > > 0 calls to icmp_error > > > 0 errors not generated 'cuz old message was icmp > > > Output histogram: > > > echo reply: 7 > > > 0 messages with bad code fields > > > 0 messages < minimum length > > > 0 bad checksums > > > 0 messages with bad length > > > Input histogram: > > > echo reply: 2 > > > echo: 7 > > > time exceeded: 790 > > > 7 message responses generated > > > > > > > > > > netstat -i > > > > > > Name Mtu Network Address > > Ipkts Ierrs > > Opkts > > > Oerrs Collis Queue > > > e8* 1500 none none 0 > > 0 0 > > > 0 0 0 > > > e0 1500 192.168.1 myhost 7929745 > > 5917 > > > 4 0 with incorrect version number > > > 0 packets with spoofed source address > > > 0 packets arrived on wrong port > > > 6499598 fragments received > > > 0 fragments dropped (dup or out of space) > > > 0 malformed fragments dropped > > > 0 overlapping fragments discarded > > > 3966 fragments dropped after timeout > > > 2272723 packets reassembled ok > > > 3693053 packets for this host > > > 790 packets for unknown/unsupported protocol > > > 0 packets forwarded > > > 867 packets not forwardable > > > 0 redirects sent > > > 3685059 packets sent from this host > > > 0 packets sent with fabricated ip header > > > 0 output packets dropped due to no bufs, etc. > > > 0 output packets discarded due to no route > > > 412593 output datagrams fragmented > > > 862312 fragments created > > > 0 datagrams that can't be fragmented > > > icmp: > > > 0 calls to icmp_error > > > 0 errors not generated 'cuz old message was icmp > > > Output histogram: > > > echo reply: 7 > > > 0 messages with bad code fields > > > 0 messages < minimum length > > > 0 bad checksums > > > 0 messages with bad length > > > Input histogram: > > > echo reply: 2 > > > echo: 7 > > > time exceeded: 790 > > > 7 message responses generated > > > > > > > > > > > > > > >

1 0

Re: Problem to integrate F740 into a Windows 2000 domain (2)
by benny.wistrom＠cn.saab.se 03 Jan '01

03 Jan '01

Problem solved! Thanks everyone for the input. It worked fine. It turned out that I had missed checking the "Allow pre-Windows 2000 authentication" box when I created the computer account. I also had to remove some files in the /etc directory on the filer because the filer had been part of another domain earlier. -- Regards: Benny Wiström CSC Phone: +46 13 4653647 Systemadministrator 581 88 Linköping Fax: +46 13 185771 Benny.Wistrom(a)cn.saab.se Sweden Mobile: +46 13 4653647 Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us. -Calvin & Hobbes (Bill Watterson) Talk sense to a fool and he calls you foolish. -- Euripides

1 0

RE: NFS server timeouts on the clients- on new 840's
by Krause, Oliver 03 Jan '01

03 Jan '01

Looks like a common half-/full-duplex mismatch. Check your network. On the filer check your interface with ifstat <interface> and look for CRC errors. Always run your filer AND switchport in 100b-tx (not auto). Sometimes this connection is done right and people forget to check between the switch and the client ... Oliver > -----Original Message----- > From: Premanshu Jain [mailto:PrJain@shastanets.com] > Sent: Mittwoch, 3. Januar 2001 05:15 > To: 'Mike Smith' > Cc: 'arr(a)oceanwave.com'; 'toasters(a)mathworks.com' > Subject: NFS server timeouts on the clients- on new 840's > > > Hi mike, > I just migrated from 740 to 840 and face this problem on the > two Sun ultra > 4500 NFS clients. Here are the error messages... > _________________________________ > 19:48:01 mthood last message repeated 10 times > Jan 2 19:49:29 mthood unix: NFS server filebot not > responding still trying > Jan 2 19:49:29 mthood unix: NFS server filebot ok > Jan 2 20:01:43 mthood xntpd[184]: time reset (step) 0.133869 s > Jan 2 20:02:46 mthood unix: NFS server filebot not > responding still trying > Jan 2 20:02:46 mthood unix: NFS server filebot ok > Jan 2 20:05:49 mthood automountd[135]: > filebot:/vol/sw/.view.dat: No such > file or directory > Jan 2 20:05:49 mthood last message repeated 1 time > Jan 2 20:07:42 mthood unix: NFS server filebot not > responding still trying > Jan 2 20:07:42 mthood unix: NFS server filebot ok > Jan 2 20:08:42 mthood unix: NFS server filebot not > responding still trying > Jan 2 20:08:42 mthood unix: NFS server filebot ok > _______________________ > Can you bail me out of this issues. Both Sun systems are now running > very-very slow . > Thanks > prem > > -----Original Message----- > From: Mike Smith [mailto:mikesmit@netapp.com] > Sent: Sunday, December 31, 2000 4:15 PM > To: arr(a)oceanwave.com; toasters(a)mathworks.com > Subject: Re: NFS server timeouts on the clients > > > Based upon the description there is no way to tell exactly > what the problem is. First off if you are using 10baseT > then the 2M transfer will take 2 seconds. If you are using > 100baseT (full duplex) then it will take 0.2 seconds. > > If you are using 10baseT then the transfer time is correct. > If you are using 100baseT (full duplex) then I would take > a look at the duplex settings on the filer side and then the > switch side. If either side is not set exactly as the other side > then it is possible you have a duplex mismatch. In which case > I would simply manually set the port that the filer is connected > to to full duplex. Then set the filer to full duplex manually: > > ifconfig e0 mediatype 100tx-fd > > Try the transfer again. You should see favorable results. > Note: sometimes autonegotiation works ok but for the most > part a lot of us stick to using manually set 100tx-fd on both the > switch side and the filer side. Hope that helps. > If not you might try contacting NetApp customer support and opening > a case > > Mike Smith > NetApp Escalations Engineer. > mikesmit(a)netapp.com > > ----- Original Message ----- > From: <arr(a)oceanwave.com> > To: <toasters(a)mathworks.com> > Sent: Sunday, December 31, 2000 3:10 PM > Subject: NFS server timeouts on the clients > > > > I have an brand new F720 running 5.3.6R2. We're exporting > two qtrees via > NFS > > and CIFS. When the system is almost completely quiet, I > try to tar up > about > > 2M of stuff and it takes FIVE minutes. I also see NFS > timeout errors on > all > > NFS clients (the one I'm doing the tar from is specifically > solaris 8, but > I > > see the timeouts on linux as well). > > > > I've tried NFS v2 and v3, both TCP and UDP, and nothing > seems to help. > netstat > > doesn't seem to show packet collisions (the netowrk > ispretty much dead), > not a > > lot of errors (<10%). I'm seeing writes about every 10 > seconds (which > should > > mean good performance), and the CPU is mostly idle. Why > the hell does a > tar > > of 2M take so bloody long? I can understand if I was > moving a gig or so, > but > > 2M?! > > > > I've included important info (this was after I had changed > to v2 UDP). Is > > there something else I should look at? > > > > > > >nfsstat -c > > > > Server rpc: > > TCP: > > calls badcalls nullrecv badlen xdrcall > > 0 0 0 0 0 > > > > UDP: > > calls badcalls nullrecv badlen xdrcall > > 1354 0 0 0 0 > > > > Server nfs: > > calls badcalls > > 1306 0 > > > > Server nfs V2: (1306 calls) > > null getattr setattr root lookup > readlink read > > 1 0% 372 28% 2 0% 0 0% 194 15% 0 0% > 293 22% > > wrcache write create remove rename link > symlink > > 0 0% 440 34% 2 0% 1 0% 0 0% 0 0% > 0 0% > > mkdir rmdir readdir statfs > > 0 0% 0 0% 1 0% 0 0% > > > > Server nfs V3: (0 calls) > > null getattr setattr lookup access > readlink read > > 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% > 0 0% > > write create mkdir symlink mknod > remove rmdir > > 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% > 0 0% > > rename link readdir readdir+ fsstat > fsinfo pathconf > > 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% > 0 0% > > commit > > 0 0% > > > > NFS V2 non-blocking request statistics: > > null getattr setattr root lookup > readlink read > > 1 100% 372 100% 2 100% 0 0% 194 100% 0 0% > 141 48% > > wrcache write create remove rename link > symlink > > 0 0% 440 100% 1 50% 1 100% 0 0% 0 0% > 0 0% > > mkdir rmdir readdir statfs > > 0 0% 0 0% 1 100% 0 0% > > > > NFS V3 non-blocking request statistics: > > null getattr setattr lookup access > readlink read > > 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% > 0 0% > > write create mkdir symlink mknod > remove rmdir > > 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% > 0 0% > > rename link readdir readdir+ fsstat > fsinfo pathconf > > 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% > 0 0% > > > > NFS reply cache statistics: > > TCP: > > In progress Delay hits Misses Idempotent > Non-idempotent > > 0 0 0 0 0 > > UDP: > > In progress Delay hits Misses Idempotent > Non-idempotent > > 0 5 732 7 42 > > > > > > >sysstat 1 > > > > CPU NFS CIFS HTTP Net kB/s Disk kB/s Tape kB/s > Cache > > in out read write read write > age > > 2% 0 0 0 0 0 0 0 0 0 > 19 > > 3% 1 0 0 0 1 0 0 0 0 > 19 > > 2% 0 0 0 0 0 0 0 0 0 > 19 > > 3% 32 0 0 5 3 0 0 0 0 > 19 > > 2% 0 0 0 0 0 0 0 0 0 > 19 > > 6% 0 0 0 0 0 48 128 0 0 > 19 > > 2% 0 0 0 31 0 0 0 0 0 > 19 > > 2% 0 0 0 0 0 0 0 0 0 > 19 > > 2% 0 0 0 0 0 0 0 0 0 > 19 > > 3% 4 0 0 2 1 0 0 0 0 > 19 > > 2% 0 0 0 0 0 0 0 0 0 > 19 > > 2% 0 0 0 0 0 0 0 0 0 > 19 > > 2% 1 0 0 16 0 0 0 0 0 > 19 > > 2% 0 0 0 0 0 0 0 0 0 > 19 > > 2% 0 0 0 0 0 0 0 0 0 > 19 > > 3% 2 0 0 1 1 0 0 0 0 > 19 > > 5% 0 0 0 0 0 32 152 0 0 > 19 > > 2% 0 0 0 0 0 0 0 0 0 > 19 > > > > > > > > >sysconfig -r > > > > Volume vol0 (root) > > > > RAID group 0 > > > > RAID Disk HA.ID HA SHELF BAY CHAN Used (MB/blks) Phys > (MB/blks) > > --------- ----- ------------ ---- > -------------- ---------- > ---- > > parity 1.3 1 0 3 FC:A 34500/70656000 > 35003/71687368 > > data 1.2 1 0 2 FC:A 34500/70656000 > 35003/71687368 > > data 1.0 1 0 0 FC:A 34500/70656000 > 35003/71687368 > > data 1.5 1 0 5 FC:A 34500/70656000 > 35003/71687368 > > data 1.6 1 0 6 FC:A 34500/70656000 > 35003/71687368 > > data 1.1 1 0 1 FC:A 34500/70656000 > 35003/71687368 > > > > Spare disks > > > > RAID Disk HA.ID HA SHELF BAY CHAN Used (MB/blks) Phys > (MB/blks) > > --------- ----- ------------ ---- > -------------- ---------- > ---- > > spare 1.4 1 0 4 FC:A 0 > 35003/71687368 > > > > > > > netstat -s > > > > tcp: > > 19180 packets sent > > 16501 data packets (3116859 bytes) > > 647 data packets (355343 bytes) retransmitted > > 804 ack-only packets (114 delayed) > > 0 URG only packets > > 0 window probe packets > > 1156 window update packets > > 76 control packets > > 22021 packets received > > 15990 acks (for 3112800 bytes) > > 187 duplicate acks > > 0 acks for unsent data > > 13688 packets (5178316 bytes) received in-sequence > > 618 completely duplicate packets (890780 bytes) > > 0 old duplicate packets > > 10 packets with some dup. data (10 bytes duped) > > 50 out-of-order packets (61320 bytes) > > 0 packets (0 bytes) of data after window > > 0 window probes > > 0 window update packets > > 1 packet received after close > > 0 discarded for bad checksums > > 0 discarded for bad header offset fields > > 0 discarded because packet too short > > 13 connection requests > > 27 connection accepts > > 3 bad connection attempts > > 0 listen queue overflows > > 32 connections established (including accepts) > > 40 connections closed (including 1 drop) > > 8 embryonic connections dropped > > 15825 segments updated rtt (of 16329 attempts) > > 531 retransmit timeouts > > 0 connections dropped by rexmit timeout > > 0 persist timeouts > > 0 connections timed out in persist > > 8 keepalive timeouts > > 0 keepalive probes sent > > 8 connections dropped by keepalive > > 3594 correct ACK header predictions > > 3474 correct data packet header predictions > > 1758 PCB cache misses > > 0 segments dropped at untrusted interface > > 0 connection requests closed at filter > > udp: > > 3671023 datagrams received > > 0 with incomplete header > > 0 with bad data length field > > 0 with bad checksum > > 0 dropped due to no socket > > 137 broadcast/multicast datagrams dropped due to no socket > > 0 dropped due to full socket buffers > > 3670886 delivered > > 3665023 datagrams output > > ip: > > 7921585 total packets received > > 0 bad header checksums > > 0 with size smaller than minimum > > 0 with size larger than maximum > > 0 with data size < data length > > 0 with header length < data size > > 0 with data length < header length > > 0 with bad options > > 0 with incorrect version number > > 0 packets with spoofed source address > > 0 packets arrived on wrong port > > 6499598 fragments received > > 0 fragments dropped (dup or out of space) > > 0 malformed fragments dropped > > 0 overlapping fragments discarded > > 3966 fragments dropped after timeout > > 2272723 packets reassembled ok > > 3693053 packets for this host > > 790 packets for unknown/unsupported protocol > > 0 packets forwarded > > 867 packets not forwardable > > 0 redirects sent > > 3685059 packets sent from this host > > 0 packets sent with fabricated ip header > > 0 output packets dropped due to no bufs, etc. > > 0 output packets discarded due to no route > > 412593 output datagrams fragmented > > 862312 fragments created > > 0 datagrams that can't be fragmented > > icmp: > > 0 calls to icmp_error > > 0 errors not generated 'cuz old message was icmp > > Output histogram: > > echo reply: 7 > > 0 messages with bad code fields > > 0 messages < minimum length > > 0 bad checksums > > 0 messages with bad length > > Input histogram: > > echo reply: 2 > > echo: 7 > > time exceeded: 790 > > 7 message responses generated > > > > > > > netstat -i > > > > Name Mtu Network Address > Ipkts Ierrs > Opkts > > Oerrs Collis Queue > > e8* 1500 none none 0 > 0 0 > > 0 0 0 > > e0 1500 192.168.1 myhost 7929745 > 5917 > > 4 0 with incorrect version number > > 0 packets with spoofed source address > > 0 packets arrived on wrong port > > 6499598 fragments received > > 0 fragments dropped (dup or out of space) > > 0 malformed fragments dropped > > 0 overlapping fragments discarded > > 3966 fragments dropped after timeout > > 2272723 packets reassembled ok > > 3693053 packets for this host > > 790 packets for unknown/unsupported protocol > > 0 packets forwarded > > 867 packets not forwardable > > 0 redirects sent > > 3685059 packets sent from this host > > 0 packets sent with fabricated ip header > > 0 output packets dropped due to no bufs, etc. > > 0 output packets discarded due to no route > > 412593 output datagrams fragmented > > 862312 fragments created > > 0 datagrams that can't be fragmented > > icmp: > > 0 calls to icmp_error > > 0 errors not generated 'cuz old message was icmp > > Output histogram: > > echo reply: 7 > > 0 messages with bad code fields > > 0 messages < minimum length > > 0 bad checksums > > 0 messages with bad length > > Input histogram: > > echo reply: 2 > > echo: 7 > > time exceeded: 790 > > 7 message responses generated > > > > > > > > >

1 0

Problem to integrate F740 into a Windows 2000 domain
by benny.wistrom＠cn.saab.se 03 Jan '01

03 Jan '01

Hi! I'm having trouble to integrate a F740 into a Windows 2000 domain. When I should set the domain name when doing cifs setup it turns out that it only allows 15 characters. The problem is that my fully qualified domainname is 17 characters. I have tried to use the netbios name which only contains 11 characters but that won't work. Any suggestions? -- Regards: Benny Wiström CSC Phone: +46 13 4653647 Systemadministrator 581 88 Linköping Fax: +46 13 185771 Benny.Wistrom(a)cn.saab.se Sweden Mobile: +46 13 4653647 Sometimes I think the surest sign that intelligent life exists elsewhere in the universe is that none of it has tried to contact us. -Calvin & Hobbes (Bill Watterson) Talk sense to a fool and he calls you foolish. -- Euripides

1 0

Bad Solaris performance over time
by hill＠communitytelephone.com 03 Jan '01

03 Jan '01

Executive summary: Our core problem appears to be that an NFS load consisting of many file creation/removal operations eventually cause our Solaris 2.6 and Solaris 8 servers to perform very poorly. The culprit seems to be kernel R/W lock contention by the caller 'nfs_access_check' for the lock 'nfs_acache_lock'. What is unclear is what causes the lock contention. It seems to build over time eventually killing the performance of the server. Our current work around is to reboot the server every morning or when the problem occurs. See details below for further information. We recently moved /var/mail from local storage on a Solaris 2.6 server to our F760 and ever since we are suffering from serious performance problems on our Solaris server. We have about 16,000 users and deliver about 100,000 - 150,000 message a day. It is important to note that the we have a few other solaris servers that communicate with our filer and they do not experience any problems. It really seems to be related to the high volume of traffic generated by moving /var/mail over to the filer. Note that we use dot-locking only and therefore are not troubled by the well-known problems with fcntl-based locking over NFS. It is worth noting that we experienced a similar problem a few months ago with a web based email package we wrote. The email package uses form based authentication (as opposed to browser based) and kept relevant session data in a flat file (about 200 bytes big). Originally we had all 18,000 files in a single directory. We tried hashing the directory so that we had about 300 files per directory, but it did not make a difference. We then moved to using a Mysql database to store the session data, and the problem went away. Before moving to the database implementation we tried a two different physical machines (both with the characteristics defined below) as well as Solaris 2.6 and Solaris 8, none of which made a difference. It seems very clear that the symptom is brought on by a heavy load of file creation and removal. We are not sure if file size is an issue, but it could be given the size of our session files and the fact that dot-lock files are 0 bytes. We do have an open ticket with Network Appliance case# 176124 but so far we haven't made much progress with it. On to the details ... Filer: F760 Release 5.3.6R1 Solaris 2.6 Box: Hardware is AXmp, 4 360MHz UltraSPARC-II CPUs w/ 4MB cache, and 2Gig RAM mount options: vers=2/proto=udp/rsize=32768/wsize=32768/noac/remote Our server tends to perform well for about 24-30hrs depending on load. At some point after that we start seeing bad performance. We have gathered the following statistics From the Solaris Box iostat, mpstat, netstat -k hme0, netstat -k, netstat -s, nfsstat -m, nfsstat -cnr, vmstat, vmstat -i, vmstat -s, top -n -dinfinity -s60 0 From the Filer netstat -m, netstat -s, nfsstat -d, nfsstat -l, statit, sysstat The only place I saw anything out of the ordinary from the above stats when the system was working properly vs. having problems was from the output of top and mpstat. I have made all stats available at http://static-content-01.communitytelephone.net/~hill/server_stats In case anyone wanted to see them. Good performance last pid: 505; load averages: 2.26, 2.24, 2.29 14:30:04 407 processes: 393 sleeping, 3 running, 9 zombie, 2 on cpu CPU states: 32.1% idle, 40.0% user, 27.9% kernel, 0.0% iowait, 0.0% swap Memory: 2048M real, 444M free, 1000M swap free CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 516 0 505 281 159 615 23 55 17 1 1970 17 12 4 67 1 298 0 8056 125 0 810 21 63 31 1 1974 13 21 3 62 2 579 0 2056 313 100 717 15 61 33 0 6216 12 12 2 74 3 991 0 832 871 721 547 31 64 40 1 4804 18 24 5 53 Bad performance last pid: 8130; load averages: 21.64, 18.25, 13.56 18:13:05 315 processes: 301 sleeping, 6 running, 6 zombie, 2 on cpu CPU states: 1.4% idle, 28.3% user, 69.9% kernel, 0.4% iowait, 0.0% swap Memory: 2048M real, 93M free, 18M swap in use, 982M swap free CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 608 0 1943 703 366 898 180 131 24 309 2861 24 75 0 2 1 512 0 3161 336 0 936 172 140 18 330 3033 26 72 0 2 2 633 0 10496 515 100 963 168 125 34 330 2712 23 76 0 2 3 607 0 1442 1262 953 883 164 139 31 291 5426 30 68 0 2 You can see from the mpstat output that icsw, srw, usr, and sys have somewhat different characteristics between the two machine states. I found the high srw numbers particularly interesting. I believe they are responsible for the change in the icsw, usr, and sys numbers. (i.e. when there is a lot of lock contention the sys cpu utilization goes up, which could also cause the involuntary context switches (icsw) to increase.) I ran 'lockstat sleep 30' during a bad performance period to try and determine the cause of the high srw numbers. Here is what I saw R/W writer blocked by writer: 135 events Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------- 103 76% 76% 1.00 271500476 nfs_acache_lock nfs_access_cache+0x1c 23 17% 93% 1.00 311148519 nfs_acache_lock nfs_access_purge_rp+0x24 3 2% 96% 1.00 7275646 0x61a87758 ufs_remove+0x11c 2 1% 97% 1.00 17863392 0x61a87758 ufs_rename+0x2c4 2 1% 99% 1.00 19876013 0x61a87758 ufs_create+0x1a8 1 1% 99% 1.00 46461 tim_list_rwlock tim_addlink+0x64 1 1% 100% 1.00 8152217 0x61a87758 ufs_link+0x13c ------------------------------------------------------------------------- R/W writer blocked by readers: 1356 events Count indv cuml rcnt nsec Lock Caller ------------------------------------------------------------------------- 1135 84% 84% 13.47 142070011 nfs_acache_lock nfs_access_cache+0x1c 218 16% 100% 10.52 100639598 nfs_acache_lock nfs_access_purge_rp+0x24 2 0% 100% 1.00 603387 0x61a87758 ufs_rename+0x2c4 1 0% 100% 1.00 121137 0x61a87758 ufs_rename+0x25c ------------------------------------------------------------------------- R/W reader blocked by writer: 1608 events Count indv cuml rcnt nsec Lock Caller ---------------------------------------------------------------------- 1600 100% 100% 1.00 103571147 nfs_acache_lock nfs_access_check+0x18 7 0% 100% 1.00 13694146 0x61a87758 ufs_dirlook+0xc0 1 0% 100% 1.00 5391663 0x61a86e28 ufs_dirlook+0xc0 ---------------------------------------------------------------------- R/W reader blocked by write wanted: 36530 events Count indv cuml rcnt nsec Lock Caller ---------------------------------------------------------------------- 36530 100% 100% 17.19 55578316 nfs_acache_lock nfs_access_check+0x18 ---------------------------------------------------------------------- For grins I did the same while the machine was performing okay R/W writer blocked by writer: 11 events Count indv cuml rcnt nsec Lock Caller ---------------------------------------------------------------------- 4 36% 36% 1.00 34507 nfs_acache_lock nfs_access_cache+0x1c 3 27% 64% 1.00 41125 kas+0x1c as_map+0x6c 2 18% 82% 1.00 94435 kas+0x1c as_unmap+0x48 1 9% 91% 1.00 238723 0x61b7f720 write+0x84 1 9% 100% 1.00 8324704 0x618283a0 ufs_create+0x1a8 ---------------------------------------------------------------------- R/W writer blocked by readers: 37 events Count indv cuml rcnt nsec Lock Caller ---------------------------------------------------------------------- 24 65% 65% 1.04 114407 nfs_acache_lock nfs_access_cache+0x1c 10 27% 92% 1.00 93271 kas+0x1c as_unmap+0x48 2 5% 97% 1.00 101398 kas+0x1c as_map+0x6c 1 3% 100% 1.00 171407 0x618283a0 ufs_rename+0x2c4 ---------------------------------------------------------------------- R/W reader blocked by writer: 61 events Count indv cuml rcnt nsec Lock Caller ---------------------------------------------------------------------- 53 87% 87% 1.00 295492 nfs_acache_lock nfs_access_check+0x18 4 7% 93% 1.00 78438 kas+0x1c as_fault+0x300 2 3% 97% 1.00 8945909 0x61a45214 ufs_getattr+0x50 1 2% 98% 1.00 35317881 0x61a45208 ufs_dirlook+0xc0 1 2% 100% 1.00 3329355 0x618283a0 ufs_dirlook+0xc0 ---------------------------------------------------------------------- R/W reader blocked by write wanted: 3 events Count indv cuml rcnt nsec Lock Caller ---------------------------------------------------------------------- 3 100% 100% 1.00 164763 nfs_acache_lock nfs_access_check+0x18 ---------------------------------------------------------------------- From the percentages it seems that nfs_access_check/nfs_acache_lock is the dominant player in the srw lock contention. What I don't understand, is what causes the drastic change in lock contention over time. I've spent a few hours on sunsolve.sun.com and docs.sun.com looking for information on nfs_access_check/nfs_acache_lock and have come up empty. I have also gone through Sun Performance and Tuning Second Edition, and the toasters mailling list with no luck. At this point I'm not sure if there is a netapp incompatiliby with Solaris, or if there is a bug in Solaris NFS client implementation. If anyone out there has any ideas I would be extrememly grateful. btw: we have made the following changes to /etc/system with no noticeable change in system behaviour. * a few kernel params are related to maxusers which is by default a few * less than amount of ram in system in MB with a default limit of 1024 * by adjusting this up certain other values are also adjusted, which is the * goal here. pp. 359 Sun performance tuning, but the decision to increase * came from the Sun System Tuning white page pp. 69 set maxusers=1536 * These settings are to help tune nfs performance to the netapp -JPH * 12.19.00 per toaster article subject: NFS client tuning, problem and * resolution / Date: Tue, 10 Aug 1999 / From: Nick Christenson * npc(a)sendmail.com * set ncsize=8192 set ncsize=6144 * It appears that the increase in this number has resulted in mutex * contention viewable by mpstat (smtx) and lockstat *set nfs:nfs_nra=10 *set nfs:nfs3_nra=10 * Tweaks from SE and Sun Performance Tuning JPH 12.25.00 * http://www.sun.com/sun-on-net/performance/priority_paging.html set priority_paging=1 * Connection hash table size default 256 increased to make hash lookups * more efficient set tcp:tcp_conn_hash_size=8192 * inode cache size - can view with 'netstat -k inode_cache' * ideally this should be 4/3 the size of ncsize set ufs_ninode=8192 * Things tweaked by virtual_adrian from the se toolkit * There is also precedent for this in Sun Performance Tuning and * Solaris Internals pp. 599 set slowscan=500 set maxpgio=25468 set fastscan=64306 set handspreadpages=30960 --Jamie Hill

3 3

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters January 2001