toasters April 1998

toasters@lists.teaparty.net

66 participants
93 discussions

Re: stale NFS file handles
by Daniel Quinlan 14 Apr '98

14 Apr '98

guy(a)netapp.com (Guy Harris) writes: > That still doesn't describe the full scenario. If somebody merely > writes to a file - in the sense of "does a 'write()' to the file" - that > will *NOT* cause ESTALE to occur on other clients - there's no reason > for it to occur. > > If the full scenario has "writes to a file" meaning "unlinks the file, > creates a new file with the same name, and writes to *that* file", > that's the scenario I described in earlier mail. Well, since the file hadn't been recently modified, I wasn't able to come up with any scenario that could apply. It looks more likely that the NetApp was sent an incorrect file handle (or didn't receive the correct one). What level of consistency checking does NFS v2 over UDP provide? > > It may be generally useful for NetApp to publish all of the known > > cases where different (legitimate to the NFS specification) errors can > > be returned from Network Appliance servers. > > "Different" in the sense of "different from, say, a Sun server"? "Different" as in "various" or "different types of errors". > I presume "overwritten" here means "removed and rewritten", as per > the above. Yes it does, please forgive my imprecise language. > If the client is assuming that a file with inode number XXXX (as > returned by an NFS "getattr" call) is the same file as a file that had > inode number XXXX on the same mount from the same client at some point > in the past, the client is buggy - it cannot validly assume that, given > the scenario I outlined in earlier mail. I'll ask the local experts about that. Linux 2.1.x (x > 42) also has a directory/file name cache that hashes names to "dentries" that include a pointer to the inode structure. - Dan

2 1

Re: stale NFS file handles
by Daniel Quinlan 14 Apr '98

14 Apr '98

guy(a)netapp.com (Guy Harris) writes: > You're not describing the full scenario here - the only operations you > mention are a write from one client and an unspecified operation from a > different client that would get an error. What's the rest of the > scenario? The most common scenario where this occurs here is for ELF executables. It may happen other places, but escape notice. It may be generally useful for NetApp to publish all of the known cases where different (legitimate to the NFS specification) errors can be returned from Network Appliance servers. For example, what set of circumstances (or client bugs) can lead to ESTALE. Another time, we had a problem running "ls -l" from client "A" on a binary that had been overwritten from client "B" while one copy of the older version was already running on "A". It was still in the directory cache, but half of the stat would fail (the directory cache lookup would give a nonexistent inode number, and the getattr would return an I/O error). > If a file that one client has open is removed while that client still > has it open, and a new file is created on the server with the same inode > number as the file that was removed, and the client that has the file > open tries to perform some operation on it via the file descriptor it > has opened for that file, it will get ESTALE (if the operation goes over > the wire to the server, which e.g. a "read()" or "stat()" might not if > it can be satisfied from a cache on the client) on most if not all UNIX > NFS servers, as well as on the filer. Yes. > The line > > fatal_error("Execv failed", strerror(errno)); > > should probably be changed to something such as > > fatal_error("Execv failed: ", strerror(errno)); > > or "fatal_error()" should be changed to add the ": ", to make the error > message look more reasonable. I agree. It's not my code, but I said about the same thing to the author. :-) > > The Linux kernel on that host also logged the error, showing: > > > > Apr 11 00:00:49 kernel: nfs_revalidate_inode: bin/xx getattr failed, > > ino=4072292, error=-116 > > It'd be interesting to see a network trace of the NFS traffic between > the client and the server, to see > > 1) what file handle was used to refer to the file on a > successful call; > > 2) what file handle was used to refer to the file on the > unsuccessful call if the latter call went over the wire. Newer versions of the Linux kernel log this information. Unfortunately, the sheer volume of NFS traffic and relative infrequency of errors makes logging this information unfeasible. > It may just be that the client is sending a bad file handle over the > wire. A possibility. Thanks for all the info. Dan

2 1

Re: stale NFS file handles
by Daniel Quinlan 14 Apr '98

14 Apr '98

hitz(a)netapp.com (Dave Hitz) writes: > The ESTALE error simply means that the server couldn't find any file > for a given file handle. If a client writes to a file, but the inode number does not change, would a different client receive ESTALE or a different error? Anyway, the reason I'm asking all this is because I'm at a loss to explain the following sequence of events. First, a daemon "xx" failed as follows: Sat Apr 11 00:00:48 1998 xx: (INFO) Removing old directories Sat Apr 11 00:00:49 1998 xx: (INFO) Done removing old directories Sat Apr 11 00:00:49 1998 xx: (INFO) Daemon is using 7024 Kb (limit is 7000) - restarting Sat Apr 11 00:00:49 1998 xx: (INFO) next instruction is execv(...self...) Sat Apr 11 00:00:49 1998 xx: (SEVERE ERROR) Execv failedStale NFS file handle Sat Apr 11 00:00:49 1998 xx: (INFO) Dropping core in directory /xxx/yyy The Linux kernel on that host also logged the error, showing: Apr 11 00:00:49 kernel: nfs_revalidate_inode: bin/xx getattr failed, ino=4072292, error=-116 Unfortunately, this version of the kernel doesn't show the "before" and "after" NFS filehandles, but the server did return ESTALE (error=-116 in Linux) according to the kernel. (The NetApp server logs don't show anything intereresting around this time.) The weird part is that all three timestamps for "xx" precede April 11 00:00:49 by a wide margin. $ ls -ali /foo/bar/bin/xx* 4072292 -rwx------ [...] Apr 10 13:11 /foo/bar/bin/xx 1161604 -rwx------ [...] Apr 10 11:22 /foo/bar/bin/xx.old Here is the code sequence that was running on the "xx" daemon when it received ESTALE when it tried to re-execute itself (it had been running since April 8). ------- start of cut text -------------- void re_exec_daemon(void) { char buf[TMP_BUF_SIZE]; disconnect(); if (close(Admin_sock) == -1) { sprintf(buf, "Can't close admin socket (%s)", strerror(errno)); log_mesg(ERROR, buf); } if (close(User_sock) == -1) { sprintf(buf, "Can't close user socket (%s)", strerror(errno)); log_mesg(ERROR, buf); } /* jump to directory we started from */ if (chdir(Invoke_dir) == -1) { sprintf(buf, "Can't change to directory %s", Invoke_dir); fatal_error(buf, strerror(errno)); } /* make sure sockets are closed */ (void) close(Admin_sock); (void) close(User_sock); log_mesg(INFO, "next instruction is execv(...self...)"); /* Re-execute us */ execv(Invoke_command[0], Invoke_command); /* should not get here */ fatal_error("Execv failed", strerror(errno)); } ------- end ---------------------------- Dan

2 1

stale NFS file handles
by Daniel Quinlan 13 Apr '98

13 Apr '98

Could someone from NetApp list the circumstances under which a Network Appliance server will return the NFS RPC ESTALE error? I'm specifically interested in NFS v2, but a v3 listing would be nice if it's different. Thanks. - Dan

3 3

snmpwalk on NetApp 5.0 problem
by Daniel Quinlan 13 Apr '98

13 Apr '98

We seem to be unable to use snmpwalk on NetApp 5.0 and 5.0R1. We're using UCD-snmp version 3.3.1 to retrieve data as well as the latest out-of-date NetApp MIB. SNMP works fine for us on 4.3.1 and earlier versions. The problem is that we can't walk the entire SNMP tree -- it fails after barely starting. It does seem possible to access later portions of the tree, but only if you know they exist and ask specifically for a smaller part of the tree. Here's the UCD-snmp environment variables we're using: export MIBFILE=/usr/local/share/snmp/mibs/netapp.mib export PREFIX=.iso.org.dod.internet.private.enterprises I was wondering if anyone else does or does not have this problem. Thanks for any info. - Dan

1 0

RE: 4GB/9GB shelf migration from toaster A to toaster B.
by grant＠netapp.com 10 Apr '98

10 Apr '98

Hi Thomas, > [Weeks, Thomas] > What's this multivolume thing? > From 5.0 Data ONTAP has support for multiple filesystems on each filer. Each filesystem is termed a volume and each volume has support for multiple raid groups, i.e. more than one (parity + data) disk set. So now (well soon) you can take your TB+ filer and configure it with 5+ x 200GB volumes, etc And you can download 5.0R1 from NOW today! The major features in Release 5.0R1 are support for Multiple Raid Groups, Multiple Volume File systems and Fibre Channel disks. > > [Weeks, Thomas] > When we bought a new filer (FAS1400-->FS230) we just used > NDMPcopy to go from the old filer to the new filer... > > We got around 5Mbps xfer rates, powered down the old server, > powered up the new server and ran.. keeping the old server in the rack > for a week just in case. > Cool! Just so you know with 4.3 releases you can migrate a whole filer even faster by asking your S.E. to turn up with his Bare Metal Migrate floppies and it supports FAS1400 hardware! Cheers, Grant

1 0

450 and a network switch
by Mark Richards 10 Apr '98

10 Apr '98

Hello I'm having a problem with my 450 (running 4.3.3) when it is connected to a LanCast 10mbp switch. Put simply. performance is terrible when switch is in my network and clients are acessing the 450 (which is attached to the switch with a crossover cable). If I remove the the switch from the network, all is well. I've sent the switch back to be tested and it is fine. Other UNIX servers in my network give great performance with or with the switch as do the clients. I've thought about half/full duplex issues on the 450's ethernet but its set correctly on the netapp and I've read that the 450 does support full-duplex anyway. Any ideas as to why my 450 doesn't like the switch? Regards Mark Richards

3 2

Re: 4GB/9GB shelf migration from toaster A to toaster B.
by sirbruce＠ix.netcom.com 10 Apr '98

10 Apr '98

On 04/09/98 16:50:02 you wrote: >Here's my theory as to the cause (translation: guess): > >The netapp mirrors the OS on the header of all the disks. > >At least half of the disks must have the same OS copy/version/checksum/magic-number >in order for the netapp to boot correctly (guess #1) Well, if true, this would have to be a new thing with respect to the OS version... it would just boot whatever OS it found "first". The "first" disk is usually the same every reboot, but circumstances (as well as simple chance) can change this. You're right about the magic-number stuff, but I think there is more than one magic-number to consider. >The "vol destroy" uses a hardware-quick-erase feature which blows away the whole >disk, not just the data portion. This blows away the OS portion of the disk. > >Viola! After a vol destroy on 22 of my 24 disks, I now have a valid OS on less >than half the disks. Non-floppy boots fail until I get it up with a floppy, and >run download (but no data is lost). The only thing that doesn't make sense to me in this theory is that if the header is invalid, then the disk will just show up as such and have to be reinitialized. But there are various types of headers on the disks and they are looked at in different ways, so the matter is probably much more complicated. My guess is the vol destroy just isn't handling the header correctly, not that the software isn't getting downloaded to it. (The latter might also be true.) Bruce

1 0

Re: 4GB/9GB shelf migration from toaster A to toaster B.
by grant＠netapp.com 09 Apr '98

09 Apr '98

Hi Jim, Jim Davis <jdavis(a)CS.Arizona.EDU> writes: > Potential plan: buy a shelf of 9GB disks, put it on toaster A, move a > shelf of 4GB disks to toaster B, keeping all of the toaster A data on > toaster A. Toaster B ends up with an empty file system on its 4GB disk > shelf. > > Now the mechanics of the move are a little unclear to me. Presumably we > hook up the 9GB shelf to toaster A and 'raid fail' each 4GB disk in the > shelf we want to move? But there will be 5 9GB data disks, since we have > to burn one as a hot spare and one becomes the new parity disk. Will > there be a problem with having 7 disks to move data from, but only 5 disks > to move data to? There would be enough GBs on the 5 9GB disks, but... I'd propose this plan of action: Buy enough 9GB disks to support all data resident on 4GB disks. Install Data ONTAP 5.0. Attach 9GB disks and create second volume. Run NDMPcopy to transfer vol0 -> vol1 (special care required if you use incrementals). Make vol1 the root volume Offline & destroy vol0 after vol1 deemed okay! Break up 4GB disks between filer A & B. Create second volume on Filer A using leftover 4GB disks. Alternative is: dump data to tape (quota trees would be useful) Break up 4GB disks between filer A & B Add 2 volumes to Filer A using 4GB & 9GB disks restore from tape to any volume (again quota trees would be useful) You have to get *all* the data off the 4GB disks before you can free any of them up (as Dan already pointed out). We don't have an EVACUATE command. Note for the future: We aim to have a volume copy command in a future release that will allow a larger volume (measured by disk capacity) to be transferred to a smaller volume (measured by disk capacity) as long as the resident data will fit. You will need to use NDMPcopy to transfer the volume if you do it with one filer and multiple volumes since BareMetal Migrate is only available in the 4.3. software release. With two (single volume) filers in play BareMetal Migrate is just the ticket. Cheers, Grant

2 1

Re: Fun with NDMP
by grant＠netapp.com 09 Apr '98

09 Apr '98

Hi Tripp, > > I have successfully (I think) ported ndmpcopy to Digital Unix, > Cool! Any chance you can contribute your changes back to ftp.ndmp.org. If you leave the code in /incoming. I'll make sure it makes its way to the contrib directory. > > but I am running into some problems trying to initiate a copy. > Boo. Hiss. > > adminhost# ./ndmpcopy srcfiler:/ms/ftp destfiler:/ftp -sa > root:passwd -da root:passwd -level 0 -v > > srcfiler> Thu Apr 9 17:09:12 EDT [ndmp_session_0]: > ../common/ndmpd/mover.c:1169 ndmpdMoverConnect: connect error: > No route to host > > destfiler> Thu Apr 9 21:09:12 GMT [ndmp_session_0]: > ../common/ndmpd/mover.c:1169 ndmpdMoverConnect: connect error: > Can't assign requested address > > I can ping destfiler from srcfiler and vice-versa. > The output below shows me that the mover on the destination is still listening for a NDMP data connection to be initiated by the source machine. Obviously the DEC client can see both filers, but perhaps the filers can't see each other on these addresses without a route being added. You can check that all ip addresses are consistent on each machine. I know that I will change the local hosts file on a filer here to talk to a server that the filer knows as server1 (111.222.333.12) whereas the yp map on all other clients will be server1 (222.333.111.70) and server1-333 (111.222.333.12). Alternatively, you could use the -dhost option with NDMPcopy and provide the ip address of the destination filer. You get this value from running ping -s destfiler from srcfiler. BTW, you can generate a bunch of (pretty cryptic) debug by running: rc_toggle_basic ; dbg level ndmpd=0xffff ; rc_toggle_basic. To turn off again run: rc_toggle_basic ; dbg level ndmpd=0x0 ; rc_toggle_basic Hope this helps. Let me know how you get on or if you have any more questions. > > Here's some interesting information: > > srcfiler> ps > [...] > 3687 BR 0% 2160 13% ndmpd > 3688 ?? 0% 3888 11% ndmp_session_0 > 3689 BR 0% 256 0% ndmp_session_1 > 3690 BR 0% 256 0% ndmp_session_2 > 3691 BR 0% 256 0% ndmp_session_3 > 3692 BR 0% 528 1% ndmp_session_4 > [...] > > destfiler> ps > [...] > 3692 ?? 0% 4112 12% ndmp_session_0 > 3693 BR 0% 208 0% ndmp_session_1 > 3694 BR 0% 208 0% ndmp_session_2 > 3695 BR 0% 208 0% ndmp_session_3 > 3696 BR 0% 208 0% ndmp_session_4 > [...] > > srcfiler> ndmpd status > ndmpd ON. > Session: 0 > Active > tape device: not open > data state: Idle > data operation: None > mover state: Idle > > destfiler> ndmpd status > ndmpd ON. > Session: 0 > Active > tape device: not open > data state: Idle > data operation: None > mover state: Listen > > srcfiler> ndmpd probe 0 > ndmpd ON. > Session: 0 > pid: 0xfffffc00006432f0 > isActive: TRUE > eof: FALSE > authorized: TRUE > spt.device_id: -1 > spt.ha: -1 > spt.scsi_id: -1 > spt.scsi_lun: -1 > tape.id: -1 > tape.device: not open > recordCount: 0 > tape.mode: Read only > mover.state: Idle > mover.mode: Read > mover.pauseReason: N/A > mover.haltReason: N/A > mover.dataWritten: 0 > mover.seekPosition: 0 > mover.bytesLeftToRead: 0 > mover.windowOffset: 0 > mover.windowLength: 18446744073709551615 > mover.position: 0 > mover.discardLength: 0 > mover.recordSize: 64512 > mover.recordNum: 0 > mover.rIndex: 0 > mover.wIndex: 0 > data.operation: None > data.state: Idle > data.haltReason: N/A > data.pauseReason: N/A > data.abort: FALSE > data.mover.addr_type TCP > data.mover.addr 0xc0003f0d > data.mover.port 1046 > data.readOffset: 0 > data.readLength: 0 > > destfiler> ndmpd probe 0 > ndmpd ON. > Session: 0 > pid: 0xfffffc00007299f0 > isActive: TRUE > eof: FALSE > authorized: TRUE > spt.device_id: -1 > spt.ha: -1 > spt.scsi_id: -1 > spt.scsi_lun: -1 > tape.id: -1 > tape.device: not open > recordCount: 0 > tape.mode: Read only > mover.state: Listen > mover.mode: Write > mover.pauseReason: N/A > mover.haltReason: N/A > mover.dataWritten: 0 > mover.seekPosition: 0 > mover.bytesLeftToRead: 0 > mover.windowOffset: 0 > mover.windowLength: 18446744073709551615 > mover.position: 0 > mover.discardLength: 0 > mover.recordSize: 64512 > mover.recordNum: 0 > mover.rIndex: 0 > mover.wIndex: 0 > data.operation: None > data.state: Idle > data.haltReason: N/A > data.pauseReason: N/A > data.abort: FALSE > data.mover.addr_type TCP > data.mover.addr 0x0 > data.mover.port 0 > data.readOffset: 0 > data.readLength: 0 > > Any help or ideas would be greatly appreciated. > > Tripp > > Cheers, Grant

1 0

← Newer
1
2
3
4
5
6
7
8
9
10
Older →

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

toasters April 1998