Trond Myklebust wrote:
On Thu, 2007-04-19 at 15:37 -0500, Gregory Baker wrote:
---+ BAD RHEL 5 64 system
...
[root@adcgar04 mnt2]# cat /proc/mounts | grep mnt2 eng:/vol/vol4/pandora/pandora-k26_g25_64-2 /mnt2 nfs
ro,vers=3,rsize=65536,wsize=65536,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,addr=eng
0 0
That would be your problem right there. Is the same volume perhaps mounted read-only somewhere else on the same client? That is no longer allowed.
Cheers Trond
Ah, news to me!
So if you mount a volume ro at one mount point /mnt1 and then try to mount the same volume rw at a second mount point /mnt2 you'll run into problems? Apologies, didn't catch this in release notes.
--Greg
---+ A New Beginning
[root@adcgar04 /]# umount -a -t nfs
[root@adcgar04 /]# mount -v | grep pandora
[root@adcgar04 /]# cat /proc/mounts | grep pandora
[root@adcgar04 /]# mount eng:/vol/vol4/pandora/pandora-k26_g25_64-2 /mnt2
[root@adcgar04 /]# mount -v | grep pandora eng:/vol/vol4/pandora/pandora-k26_g25_64-2 on /mnt2 type nfs (rw,addr=163.181.34.137)
[root@adcgar04 /]# cat /proc/mounts | grep pandora eng:/vol/vol4/pandora/pandora-k26_g25_64-2 /mnt2 nfs rw,vers=3,rsize=65536,wsize=65536,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=eng 0 0
[root@adcgar04 /]# touch /mnt2/asdf
---+ The ro Strikes Back
[root@adcgar04 /]# mount -o ro eng:/vol/vol4/pandora/pandora-k26_g25_64-2 /mnt1
[root@adcgar04 /]# mount -o rw eng:/vol/vol4/pandora/pandora-k26_g25_64-2 /mnt2
[root@adcgar04 /]# mount -v | grep mnt eng:/vol/vol4/pandora/pandora-k26_g25_64-2 on /mnt1 type nfs (ro,addr=163.181.34.137) eng:/vol/vol4/pandora/pandora-k26_g25_64-2 on /mnt2 type nfs (rw,addr=163.181.34.137)
[root@adcgar04 /]# cat /proc/mounts | grep mnt eng:/vol/vol4/pandora/pandora-k26_g25_64-2 /mnt1 nfs ro,vers=3,rsize=65536,wsize=65536,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=eng 0 0 eng:/vol/vol4/pandora/pandora-k26_g25_64-2 /mnt2 nfs ro,vers=3,rsize=65536,wsize=65536,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=eng 0 0
[root@adcgar04 /]# touch /mnt2/asdf touch: cannot touch `/mnt2/asdf': Read-only file system
Choudhury, Bikash wrote:
There are two different things happening here.
The ro/rw rule in the filer exports file
The mount -v and cat/proc/mounts
Regarding the ro/rw rule that you have in the export file, I run a
small test in my lab. I have tried to replicate the problem with the same rule as you have sent in your exports but I have not used the netgroups instead used the ip subnets. I ran these tests on 7.2.2 filer.
fas960c-svl04*> exportfs
/vol/test_vol -sec=sys,ro=172.17.39.0/22,rw=172.17.39.0/29,root=172.17.39.72,nosuid
/vol/vol0/home -sec=sys,rw,nosuid
/vol/vol0 -sec=sys,rw,anon=0,nosuid
This is what you noticed on the Linux client:
[root@ibmx335-svl14 mnt]# mount fas960c-svl04:/vol/test_vol /mnt/b1
[root@ibmx335-svl14 mnt]# ls -al
total 20
drwxr-xr-x 3 root root 4096 Feb 28 03:12 .
drwxr-xr-x 25 root root 4096 Feb 24 09:27 ..
drwxr-xr-x 3 root root 4096 Nov 20 11:11 b1
[root@ibmx335-svl14 mnt]# cd /etc
[root@ibmx335-svl14 etc]# touch /mnt/b1/testfile1
touch: cannot touch `/mnt/bik/testfile1': Permission denied
On the filer I checked the permission for that file system. You can probably run the same command on your filer to check the exact permissions for /vol/vol4/Pandora. Replace the ip address with the one for your RHEl5 client.
fas960c-svl04*> exportfs -c 172.17.39.72 /vol/test_vol rw
exportfs: 172.17.39.72 does not have rw access to /vol/test_vol (Access reduced)
fas960c-svl04*> exportfs -c 172.17.39.72 /vol/test_vol ro
exportfs: 172.17.39.72 has ro access to /vol/test_vol
This is definitely going to happen BECAUSE; take a look at the /etc/exports file. The –ro is set on /22 subnet. This means 22 bits out 32 are used as network address and the remaining 10 bits determine the number of host i.e: 2^10. This indicates that the /22 with 255.255.252.0 subnet has more number of clients that /29 with subnet 255.255.255.248. When –ro is specified on a bigger network (/22) and –rw is listed with the (/29) network, the more restricted one takes precedence, i.e ‘-ro’. In my example the linux client had ip address 172.17.39.72 which falls in the bigger network (/22) and is not in the (/29) network. Therefore the client gets ‘-ro’ access instead of ‘rw’. When you try you a ‘touch’ a file as ‘root’ from this client, you get ‘permission denied’ which is legitimate because this client has ‘ro’ access.
In your case you have -ro set on /vol/vol4/Pandora to every one and then have -rw set to @volexport-pandora,root=@volexport-pandora which is part of the wide spectrum that has '-ro' set.
Then I switched the ‘ro’ to the (/29) network and ‘rw’ to (/22) network. This is what it looks now.
On the filer after the switch
fas960c-svl04*> exportfs
/vol/test_vol -sec=sys,ro=172.17.39.0/29,rw=172.17.39.0/22,root=172.17.39.72,nosuid
/vol/vol0/home -sec=sys,rw,nosuid
/vol/vol0 -sec=sys,rw,anon=0,nosuid
On the client after the switch:
[root@ibmx335-svl14 etc]# touch /mnt/b1/testfile1
[root@ibmx335-svl14 etc]# ls -l /mnt/b1
total 0
-rw-r--r-- 1 root root 0 Feb 27 10:46 testfile1
fas960c-svl04*> exportfs -c 172.17.39.72 /vol/test_vol ro
exportfs: 172.17.39.72 has ro access to /vol/test_vol
fas960c-svl04*> exportfs -c 172.17.39.72 /vol/test_vol rw
exportfs: 172.17.39.72 has rw access to /vol/test_vol
Now I am able to touch the file w/o any problems. In your case adcgar04.amd.com is part of the bigger network that has ‘ro’ access. As suggested in the man page ‘ro’ will take precedence over ‘rw’ and touching a file from a host in the /22 network will end up with a ‘permission denied’. Currently this is by design. However, there is an RFE 237892 open for this configuration to work. If you decide to upgrade then you may consider 7.2.2 but this release does not fix this RFE.
- I was currently running the above test from a RHEL5 client. But I did
not notice any difference in the mount options between 'mount -v' and 'cat /proc/mounts'.
[root@ibmx335-svl47 etc]# uname -a
Linux ibmx335-svl47 2.6.18-8.el5 #1 SMP Fri Jan 26 14:15:21 EST 2007 i686 i686 i386 GNU/Linux
[root@ibmx335-svl47 ~]# rpm -qa |grep nfs-utils
nfs-utils-1.0.9-16.el5
nfs-utils-lib-1.0.8-7.2
[root@ibmx335-svl47 ~]# mount -v|grep b1
172.17.44.44:/vol/test_vol on /mnt/b1 type nfs4 (*rw*,vers=4,rsize=65536,wsize=65536,hard,intr,proto=tcp,timeo=600,retrans=3,sec=sys,addr=172.17.44.44)
[root@ibmx335-svl47 ~]# cat /proc/mounts|grep b1
172.17.44.44:/vol/test_vol /mnt/b1 nfs4* rw*,vers=4,rsize=65536,wsize=65536,hard,intr,proto=tcp,timeo=600,retrans=3,sec=sys,add
Bikash R.Choudhury
-----Original Message----- From: Gregory Baker [mailto:gregory.baker@amd.com] Sent: Wednesday, April 18, 2007 3:23 PM To: toasters@mathworks.com Cc: julia bauer Subject: inconsistent mount attributes (ro/rw), RHEL5 / Netapp]
Note I have cases open with Redhat and Netapp, but was curious if other
people have also seen inconsistent mount attributes (ro/rw) when
mounting RHEL5 client vs. Netapp 7.2 Ontap.
---+ System environment
[greg@adcgar04 greg]$ uname -a
Linux adcgar04 2.6.18-8.el5 #1 SMP Fri Jan 26 14:15:14 EST 2007 x86_64
unknown unknown GNU/Linux
[greg@adcgar04 greg]$ rpm -qa | grep nfs-utils
nfs-utils-1.0.9-16.el5
nfs-utils-lib-1.0.8-7.2
nfs-utils-lib-devel-1.0.8-7.2
nfs-utils-lib-1.0.8-7.2
nfs-utils-lib-devel-1.0.8-7.2
[greg@adcgar04 greg]$ mount -V
mount (util-linux 2.13-pre7)
---+ NFS server (Netapp) environment
[greg@apathy greg]$ sudo rsh eng version
NetApp Release 7.2P4: Tue Nov 28 02:55:54 PST 2006
---++ NFS export file entry
[greg@apathy greg]$ sudo rsh eng exportfs | grep pandora
/vol/vol4/pandora
-sec=sys,ro,rw=@volexport-pandora,root=@volexport-pandora,anon=4058
---++ netgroup member (for export file entry above)
[greg@apathy greg]$ show_netgroup volexport-pandora | grep adcgar
adcgar04.amd.com
---+ Demonstration of inconsistent ro/rw mount reporting
[root@adcgar04 /]# mount eng:/vol/vol4/pandora/pandora-k26_g25_64-2 /mnt2
[root@adcgar04 /]# mount -v | grep mnt2
eng:/vol/vol4/pandora/pandora-k26_g25_64-2 on /mnt2 type nfs
(rw,addr=163.181.34.137)
[root@adcgar04 /]# cat /proc/mounts | grep mnt2
eng:/vol/vol4/pandora/pandora-k26_g25_64-2 /mnt2 nfs
ro,vers=3,rsize=65536,wsize=65536,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,addr=eng
0 0
[root@adcgar04 /]# cd /mnt2/
[root@adcgar04 mnt2]# touch asdf
touch: cannot touch `asdf': Read-only file system
---+ Discussion
---++Linux (RHEL3/4) NFS servers that have similiar exportfs options
/tmp *(ro,anonuid=4058) @volexport-pandora(rw,no_root_squash)
do not cause the inconsistent behavior between mount -v and /proc/mounts
(and it is mounted rw as expected on the client).
---++ A reply from NetApp had this info:
Starting with ONTAP 7.2.1 onward, ONTAP will display the "most
pessimistic" permissions to NFSv3 and NFSv4 clients. NFSv2 clients will
see permissions the same way as in previous releases of ONTAP, i.e. the
"most optimistic" permissions.
And mounting using NFS v2 (instead of v3) does give us the expected
rw/rw consistency and ability.
---++ So now what?
Should the linux mount -v and cat /proc/mounts be consistent with what
is actually happening?
Should netapp exports syntax handle a wildcard ro and a netgroup rw?
Comments and feedback welcome.
Thanks,
--Greg
--
Greg Baker 512-602-3287 (work)
gregory.baker@amd.com 512-602-6970 (fax)
5204 E. Ben White Blvd MS 625 512-555-1212 (info)
Austin, TX 78741
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
--
Greg Baker 512-602-3287 (work)
gregory.baker@amd.com 512-602-6970 (fax)
5204 E. Ben White Blvd MS 625 512-555-1212 (info)
Austin, TX 78741
...closing the loop on what we've found out... it doesn't appear that a convenient workaround (at the mount level) exists. We'll play around with local executable automount maps to see if that will work for us.
Thanks,
--Greg
***** Indeed, now that I read that comment by Trond and went to the source code, I found out about upstream commit 54ceac4515986030c2502960be620198dd8fe25b.
The idea is that having a single super_block structure per server per FSID prevents corner cases that can lead to corrupt dentry cache trees, prevents conflicting buffer cache contents to what ends up being the same file, and some other scary situations.
So the deal is, the mount flags (and NFS options) are set only the first time that a given combination of server and filesystem are mounted. If you ever mount the same filesystem from the same server on another mountpoint, you'll get the flags and options that were passed on to the first mount. There's no working around that. *****
Gregory Baker wrote:
Trond Myklebust wrote:
On Thu, 2007-04-19 at 15:37 -0500, Gregory Baker wrote:
---+ BAD RHEL 5 64 system
...
[root@adcgar04 mnt2]# cat /proc/mounts | grep mnt2 eng:/vol/vol4/pandora/pandora-k26_g25_64-2 /mnt2 nfs
ro,vers=3,rsize=65536,wsize=65536,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,addr=eng
0 0
That would be your problem right there. Is the same volume perhaps mounted read-only somewhere else on the same client? That is no longer allowed.
Cheers Trond
Ah, news to me!
So if you mount a volume ro at one mount point /mnt1 and then try to mount the same volume rw at a second mount point /mnt2 you'll run into problems? Apologies, didn't catch this in release notes.
--Greg
---+ A New Beginning
[root@adcgar04 /]# umount -a -t nfs
[root@adcgar04 /]# mount -v | grep pandora
[root@adcgar04 /]# cat /proc/mounts | grep pandora
[root@adcgar04 /]# mount eng:/vol/vol4/pandora/pandora-k26_g25_64-2 /mnt2
[root@adcgar04 /]# mount -v | grep pandora eng:/vol/vol4/pandora/pandora-k26_g25_64-2 on /mnt2 type nfs (rw,addr=163.181.34.137)
[root@adcgar04 /]# cat /proc/mounts | grep pandora eng:/vol/vol4/pandora/pandora-k26_g25_64-2 /mnt2 nfs rw,vers=3,rsize=65536,wsize=65536,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=eng 0 0
[root@adcgar04 /]# touch /mnt2/asdf
---+ The ro Strikes Back
[root@adcgar04 /]# mount -o ro eng:/vol/vol4/pandora/pandora-k26_g25_64-2 /mnt1
[root@adcgar04 /]# mount -o rw eng:/vol/vol4/pandora/pandora-k26_g25_64-2 /mnt2
[root@adcgar04 /]# mount -v | grep mnt eng:/vol/vol4/pandora/pandora-k26_g25_64-2 on /mnt1 type nfs (ro,addr=163.181.34.137) eng:/vol/vol4/pandora/pandora-k26_g25_64-2 on /mnt2 type nfs (rw,addr=163.181.34.137)
[root@adcgar04 /]# cat /proc/mounts | grep mnt eng:/vol/vol4/pandora/pandora-k26_g25_64-2 /mnt1 nfs ro,vers=3,rsize=65536,wsize=65536,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=eng 0 0 eng:/vol/vol4/pandora/pandora-k26_g25_64-2 /mnt2 nfs ro,vers=3,rsize=65536,wsize=65536,hard,proto=tcp,timeo=600,retrans=2,sec=sys,addr=eng 0 0
[root@adcgar04 /]# touch /mnt2/asdf touch: cannot touch `/mnt2/asdf': Read-only file system
Choudhury, Bikash wrote:
There are two different things happening here.
The ro/rw rule in the filer exports file
The mount -v and cat/proc/mounts
Regarding the ro/rw rule that you have in the export file, I run a
small test in my lab. I have tried to replicate the problem with the same rule as you have sent in your exports but I have not used the netgroups instead used the ip subnets. I ran these tests on 7.2.2 filer.
fas960c-svl04*> exportfs
/vol/test_vol -sec=sys,ro=172.17.39.0/22,rw=172.17.39.0/29,root=172.17.39.72,nosuid
/vol/vol0/home -sec=sys,rw,nosuid
/vol/vol0 -sec=sys,rw,anon=0,nosuid
This is what you noticed on the Linux client:
[root@ibmx335-svl14 mnt]# mount fas960c-svl04:/vol/test_vol /mnt/b1
[root@ibmx335-svl14 mnt]# ls -al
total 20
drwxr-xr-x 3 root root 4096 Feb 28 03:12 .
drwxr-xr-x 25 root root 4096 Feb 24 09:27 ..
drwxr-xr-x 3 root root 4096 Nov 20 11:11 b1
[root@ibmx335-svl14 mnt]# cd /etc
[root@ibmx335-svl14 etc]# touch /mnt/b1/testfile1
touch: cannot touch `/mnt/bik/testfile1': Permission denied
On the filer I checked the permission for that file system. You can probably run the same command on your filer to check the exact permissions for /vol/vol4/Pandora. Replace the ip address with the one for your RHEl5 client.
fas960c-svl04*> exportfs -c 172.17.39.72 /vol/test_vol rw
exportfs: 172.17.39.72 does not have rw access to /vol/test_vol (Access reduced)
fas960c-svl04*> exportfs -c 172.17.39.72 /vol/test_vol ro
exportfs: 172.17.39.72 has ro access to /vol/test_vol
This is definitely going to happen BECAUSE; take a look at the /etc/exports file. The –ro is set on /22 subnet. This means 22 bits out 32 are used as network address and the remaining 10 bits determine the number of host i.e: 2^10. This indicates that the /22 with 255.255.252.0 subnet has more number of clients that /29 with subnet 255.255.255.248. When –ro is specified on a bigger network (/22) and –rw is listed with the (/29) network, the more restricted one takes precedence, i.e ‘-ro’. In my example the linux client had ip address 172.17.39.72 which falls in the bigger network (/22) and is not in the (/29) network. Therefore the client gets ‘-ro’ access instead of ‘rw’. When you try you a ‘touch’ a file as ‘root’ from this client, you get ‘permission denied’ which is legitimate because this client has ‘ro’ access.
In your case you have -ro set on /vol/vol4/Pandora to every one and then have -rw set to @volexport-pandora,root=@volexport-pandora which is part of the wide spectrum that has '-ro' set.
Then I switched the ‘ro’ to the (/29) network and ‘rw’ to (/22) network. This is what it looks now.
On the filer after the switch
fas960c-svl04*> exportfs
/vol/test_vol -sec=sys,ro=172.17.39.0/29,rw=172.17.39.0/22,root=172.17.39.72,nosuid
/vol/vol0/home -sec=sys,rw,nosuid
/vol/vol0 -sec=sys,rw,anon=0,nosuid
On the client after the switch:
[root@ibmx335-svl14 etc]# touch /mnt/b1/testfile1
[root@ibmx335-svl14 etc]# ls -l /mnt/b1
total 0
-rw-r--r-- 1 root root 0 Feb 27 10:46 testfile1
fas960c-svl04*> exportfs -c 172.17.39.72 /vol/test_vol ro
exportfs: 172.17.39.72 has ro access to /vol/test_vol
fas960c-svl04*> exportfs -c 172.17.39.72 /vol/test_vol rw
exportfs: 172.17.39.72 has rw access to /vol/test_vol
Now I am able to touch the file w/o any problems. In your case adcgar04.amd.com is part of the bigger network that has ‘ro’ access. As suggested in the man page ‘ro’ will take precedence over ‘rw’ and touching a file from a host in the /22 network will end up with a ‘permission denied’. Currently this is by design. However, there is an RFE 237892 open for this configuration to work. If you decide to upgrade then you may consider 7.2.2 but this release does not fix this RFE.
- I was currently running the above test from a RHEL5 client. But I
did not notice any difference in the mount options between 'mount -v' and 'cat /proc/mounts'.
[root@ibmx335-svl47 etc]# uname -a
Linux ibmx335-svl47 2.6.18-8.el5 #1 SMP Fri Jan 26 14:15:21 EST 2007 i686 i686 i386 GNU/Linux
[root@ibmx335-svl47 ~]# rpm -qa |grep nfs-utils
nfs-utils-1.0.9-16.el5
nfs-utils-lib-1.0.8-7.2
[root@ibmx335-svl47 ~]# mount -v|grep b1
172.17.44.44:/vol/test_vol on /mnt/b1 type nfs4 (*rw*,vers=4,rsize=65536,wsize=65536,hard,intr,proto=tcp,timeo=600,retrans=3,sec=sys,addr=172.17.44.44)
[root@ibmx335-svl47 ~]# cat /proc/mounts|grep b1
172.17.44.44:/vol/test_vol /mnt/b1 nfs4* rw*,vers=4,rsize=65536,wsize=65536,hard,intr,proto=tcp,timeo=600,retrans=3,sec=sys,add
Bikash R.Choudhury
-----Original Message----- From: Gregory Baker [mailto:gregory.baker@amd.com] Sent: Wednesday, April 18, 2007 3:23 PM To: toasters@mathworks.com Cc: julia bauer Subject: inconsistent mount attributes (ro/rw), RHEL5 / Netapp]
Note I have cases open with Redhat and Netapp, but was curious if other
people have also seen inconsistent mount attributes (ro/rw) when
mounting RHEL5 client vs. Netapp 7.2 Ontap.
---+ System environment
[greg@adcgar04 greg]$ uname -a
Linux adcgar04 2.6.18-8.el5 #1 SMP Fri Jan 26 14:15:14 EST 2007 x86_64
unknown unknown GNU/Linux
[greg@adcgar04 greg]$ rpm -qa | grep nfs-utils
nfs-utils-1.0.9-16.el5
nfs-utils-lib-1.0.8-7.2
nfs-utils-lib-devel-1.0.8-7.2
nfs-utils-lib-1.0.8-7.2
nfs-utils-lib-devel-1.0.8-7.2
[greg@adcgar04 greg]$ mount -V
mount (util-linux 2.13-pre7)
---+ NFS server (Netapp) environment
[greg@apathy greg]$ sudo rsh eng version
NetApp Release 7.2P4: Tue Nov 28 02:55:54 PST 2006
---++ NFS export file entry
[greg@apathy greg]$ sudo rsh eng exportfs | grep pandora
/vol/vol4/pandora
-sec=sys,ro,rw=@volexport-pandora,root=@volexport-pandora,anon=4058
---++ netgroup member (for export file entry above)
[greg@apathy greg]$ show_netgroup volexport-pandora | grep adcgar
adcgar04.amd.com
---+ Demonstration of inconsistent ro/rw mount reporting
[root@adcgar04 /]# mount eng:/vol/vol4/pandora/pandora-k26_g25_64-2 /mnt2
[root@adcgar04 /]# mount -v | grep mnt2
eng:/vol/vol4/pandora/pandora-k26_g25_64-2 on /mnt2 type nfs
(rw,addr=163.181.34.137)
[root@adcgar04 /]# cat /proc/mounts | grep mnt2
eng:/vol/vol4/pandora/pandora-k26_g25_64-2 /mnt2 nfs
ro,vers=3,rsize=65536,wsize=65536,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,addr=eng
0 0
[root@adcgar04 /]# cd /mnt2/
[root@adcgar04 mnt2]# touch asdf
touch: cannot touch `asdf': Read-only file system
---+ Discussion
---++Linux (RHEL3/4) NFS servers that have similiar exportfs options
/tmp *(ro,anonuid=4058) @volexport-pandora(rw,no_root_squash)
do not cause the inconsistent behavior between mount -v and /proc/mounts
(and it is mounted rw as expected on the client).
---++ A reply from NetApp had this info:
Starting with ONTAP 7.2.1 onward, ONTAP will display the "most
pessimistic" permissions to NFSv3 and NFSv4 clients. NFSv2 clients will
see permissions the same way as in previous releases of ONTAP, i.e. the
"most optimistic" permissions.
And mounting using NFS v2 (instead of v3) does give us the expected
rw/rw consistency and ability.
---++ So now what?
Should the linux mount -v and cat /proc/mounts be consistent with what
is actually happening?
Should netapp exports syntax handle a wildcard ro and a netgroup rw?
Comments and feedback welcome.
Thanks,
--Greg
--
Greg Baker 512-602-3287 (work)
gregory.baker@amd.com 512-602-6970 (fax)
5204 E. Ben White Blvd MS 625 512-555-1212 (info)
Austin, TX 78741
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
NFS maillist - NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs
--
Greg Baker 512-602-3287 (work)
gregory.baker@amd.com 512-602-6970 (fax)
5204 E. Ben White Blvd MS 625 512-555-1212 (info)
Austin, TX 78741
-
Gregory Baker