We installed the latest kernel update for Red Hat FC5 (forgot the kernel version number) and we noticed a nifty new NFS client feature. The kernel is "snapshot aware" which means that when you "stat()" something in a snapshot, the device number is different from the live filesystem. Furthermore, different snapshots have different device numbers.
I can see where this is a very nice feature, because it solves some difficulties with snapshots.
o You can now run "find /mountpoint -mount" (or -xdev) to avoid dropping into the .snapshot tree.
o You can now "diff" a live file and a snapshot copy of the file because many versions of diff simply return no differences when two files have the same device and inode number.
o You can now "cp" a snapshot copy of a file on top of the live file. This used to fail because cp checks the device and inode number to prevent you from copying a file on top of itself.
However, we have some locally written software that expects the live filesystem and snapshot to have the same device number. The person who wrote the software is wondering if this change in NFS behavior is a bug, or fluke, or experimental, or if the Linux developers intend for the Linux kernel to behave this way from now on. Can this behavior be enabled/disabled ? If no one knows offhand, where might I go to find out?
Steve Losen scl@virginia.edu phone: 434-924-0640
University of Virginia ITC Unix Support
My suggestion would be to ask nfs@lists.sourceforge.net. Trond and Chuck Lever both seem to hang out there, and discuss changes to the linux nfs client. Alternately, the source might have some interesting comments. This might be an interesting start. I'll admit I did a quick check, and didn't find anything specific regarding snapshot support.
http://lxr.linux.no/source/fs/nfs/?v=2.6.18
-Blake
On 11/6/06, Stephen C. Losen scl@sasha.acc.virginia.edu wrote:
We installed the latest kernel update for Red Hat FC5 (forgot the kernel version number) and we noticed a nifty new NFS client feature. The kernel is "snapshot aware" which means that when you "stat()" something in a snapshot, the device number is different from the live filesystem. Furthermore, different snapshots have different device numbers.
I can see where this is a very nice feature, because it solves some difficulties with snapshots.
o You can now run "find /mountpoint -mount" (or -xdev) to avoid dropping into the .snapshot tree.
o You can now "diff" a live file and a snapshot copy of the file because many versions of diff simply return no differences when two files have the same device and inode number.
o You can now "cp" a snapshot copy of a file on top of the live file. This used to fail because cp checks the device and inode number to prevent you from copying a file on top of itself.
However, we have some locally written software that expects the live filesystem and snapshot to have the same device number. The person who wrote the software is wondering if this change in NFS behavior is a bug, or fluke, or experimental, or if the Linux developers intend for the Linux kernel to behave this way from now on. Can this behavior be enabled/disabled ? If no one knows offhand, where might I go to find out?
Steve Losen scl@virginia.edu phone: 434-924-0640
University of Virginia ITC Unix Support
My suggestion would be to ask nfs@lists.sourceforge.net. Trond and Chuck Lever both seem to hang out there, and discuss changes to the linux nfs client. Alternately, the source might have some interesting comments. This might be an interesting start. I'll admit I did a quick check, and didn't find anything specific regarding snapshot support.
http://lxr.linux.no/source/fs/nfs/?v=2.6.18
-Blake
On 11/6/06, Stephen C. Losen scl@sasha.acc.virginia.edu wrote:
We installed the latest kernel update for Red Hat FC5 (forgot the kernel version number) and we noticed a nifty new NFS client feature. The kernel is "snapshot aware" which means that when you "stat()" something in a snapshot, the device number is different from the live filesystem. Furthermore, different snapshots have different device numbers.
I looked at the Linux source code in the above URL and found what may explain the behavior. The getattr() call receives a struct with a "rdev" (remove device?) field. I think this is a remote device number that is supplied by the NFS server (filer). The Linux NFS client code uses the rdev field to set a "fsys" field, which I think is a local (client) filesystem identifier. I suspect the "fsys" field is what stat() returns in the st_dev field of a "struct stat".
I suspect that when a NFS client does a getattr() on an object in a snapshot that the filer returns a different "rdev" value than it returns for objects in the live filesystem. Does anyone know if this is indeed how a filer behaves?
Perhaps the "fsys" field is traditionally determined by the NFS client at mount time and everything below a mount point (including snapshots) gets the same "fsys". Perhaps Linux is now deriving the fsys field from the rdev field returned by the filer.
Are there any NFS experts out there who can comment further?
I did some more experiments with a Linux NFS client running an earlier kernel. According to stat each mount point has a unique device number and all objects (including those in snapshots) have this device number.
With the current kernel (2.6.18-1.2200.fc5smp) things are different. As I said before, the device number changes between the live filesystem and snapshots. Furthermore, if I mount two subtrees of the same netapp volume on two different mount points such as this:
mount filer:/vol/vol0/x /mnt/x
mount filer:/vol/vol0/y /mnt/y
then stat shows the same device number for both /mnt/x and /mnt/y.
On the old kernel the device numbers are different.
Steve Losen scl@virginia.edu phone: 434-924-0640
University of Virginia ITC Unix Support
Trond Myklebust of Network Appliance, who is also a developer of the Linux Kernel NFS client, told me that the new Linux NFS client behavior is intentional and supported.
Starting with the 2.6.18 Linux kernel, Netapp snapshots have a different device number than the corresponding live volume when viewed with NFS.
The change was made to better support NFS 4, which allows a NFS client to cross mount points on the NFS server provided that the server has exported everything appropriately.
One of the file attributes that a NFS server passes is the FSID (file system ID). On a netapp filer the FSID of a snapshot is different from the FSID of the corresponding live volume. Most NFS clients ignore the FSID attribute.
Steve Losen scl@virginia.edu phone: 434-924-0640
University of Virginia ITC Unix Support
Trond Myklebust of Network Appliance, who is also a developer of the Linux Kernel NFS client, told me that the new Linux NFS client behavior is intentional and supported.
Starting with the 2.6.18 Linux kernel, Netapp snapshots have a different device number than the corresponding live volume when viewed with NFS.
The change was made to better support NFS 4, which allows a NFS client to cross mount points on the NFS server provided that the server has exported everything appropriately.
So just to confirm, this change in behavior does not require that NFS 4 is actually in use? We'll still see it in a v3 mount as well?
So just to confirm, this change in behavior does not require that NFS 4 is actually in use? We'll still see it in a v3 mount as well?
Yes. We use NFS 3 and we see the behavior. The code in the kernel that was changed is shared by both NFS 3 and 4 (and maybe even 2).
Steve Losen scl@virginia.edu phone: 434-924-0640
University of Virginia ITC Unix Support
I would try searching on the internet or maybe contact Red Hat.
-----Original Message----- From: owner-toasters@mathworks.com [mailto:owner-toasters@mathworks.com] On Behalf Of Stephen C. Losen Sent: Monday, November 06, 2006 8:17 AM To: toasters@mathworks.com Subject: New Linux Kernel NFS client and netapp snapshot behavior
We installed the latest kernel update for Red Hat FC5 (forgot the kernel version number) and we noticed a nifty new NFS client feature. The kernel is "snapshot aware" which means that when you "stat()" something in a snapshot, the device number is different from the live filesystem. Furthermore, different snapshots have different device numbers.
I can see where this is a very nice feature, because it solves some difficulties with snapshots.
o You can now run "find /mountpoint -mount" (or -xdev) to avoid dropping into the .snapshot tree.
o You can now "diff" a live file and a snapshot copy of the file because many versions of diff simply return no differences when two files have the same device and inode number.
o You can now "cp" a snapshot copy of a file on top of the live file. This used to fail because cp checks the device and inode number to prevent you from copying a file on top of itself.
However, we have some locally written software that expects the live filesystem and snapshot to have the same device number. The person who wrote the software is wondering if this change in NFS behavior is a bug, or fluke, or experimental, or if the Linux developers intend for the Linux kernel to behave this way from now on. Can this behavior be enabled/disabled ? If no one knows offhand, where might I go to find out?
Steve Losen scl@virginia.edu phone: 434-924-0640
University of Virginia ITC Unix Support
-
Blake Golliher -
Darren Dunham -
Johnson, James A -
Stephen C. Losen