Jay,
I've unpicked these things on occasions before, but your posting of Guy Harris' description was a useful memory jog! I don't think there's anything wrong with it except that there are three "unused bytes", not one.
NFS write error on host hardrock: Stale NFS file handle. (file handle: 2ae0500 51356400 20000000 aa5c8 54417810 5c140000 2ae0500 51356400)
The snapshot id of 32 means the active filing system.
The mount and export inode number is 0x05ae02 (read little-endian) = 372226. You should be able to identify it from that unless you have unreasonably many export/mount points.
The file inode number is 0xaa5c8 (read big-endian) = 697800. Of course it probably isn't there any longer... But the inode number might have been re-used in the same directory, or some adjacent ones may exist there (because of the way inodes are allocated based on directory), or you might find a snapshot still containing that inode number. A bit of "find ... -inum +697791 -inum -697824 -ls" (the right block of 32 inodes) may be what you need.
They all have the same File handle, and the toaster has no error messages in its etc/messages files. I tried rebooting a few systems, but they still generate these error messages.
It's theoretically possible for a system to remember an NFS file handle over a reboot, but it's such an evil thing to do that I am reluctant to believe that is your problem! It's more likely that some client has taken to regularly removing a file that is in use by others.
Your error messages look suspiciously like those from Solaris 2+, though you don't say. I have found that it is possible for Solaris to get dirty blocks into its "buffer cache" (really, all of RAM) that it goes on for hours periodically trying to write to the server and generating the same "stale NFS file handle" message: it does *eventually* give up! I think this happens when the process that wrote the data never fsync'd, and so isn't there any longer to report the error to.
Chris Thompson University of Cambridge Computing Service, Email: cet1@ucs.cam.ac.uk New Museums Site, Cambridge CB2 3QG, Phone: +44 1223 334715 United Kingdom.