Hi All,
We upgraded our 6 nodes cluster to ONTAP9 yesterday. Today I discovered some freshly installed CentOS 6.8 hosts not being able to mount file shares via NFS v4 protocol. The mount command just kind of hangs. I verified it's not a connectivity issue, ie connections to the filer's port 2049 are possible.
Also NFS v3 works just fine.
Did anyone experience anything like that?
Cheers, Vladimir
Should check the IMT... refers to http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=1035160 and that you should be on GA 6.8 and Supported Kernel version:- kernel-2.6.32-642.el6.x86_64 Supported NFS-utils version:- nfs-utils-1.2.3-70.el6.x86_64 Supported libtirpc version:- libtirpc-0.2.1-11.el6.x86_64
This is for NFS4.1...which you may or may not be using....worth checking into.
--tmac
*Tim McCarthy, **Principal Consultant*
*Proud Member of the #NetAppATeam https://twitter.com/NetAppATeam*
*I Blog at TMACsRack https://tmacsrack.wordpress.com/*
On Wed, Nov 2, 2016 at 6:06 AM, Momonth momonth@gmail.com wrote:
Hi All,
We upgraded our 6 nodes cluster to ONTAP9 yesterday. Today I discovered some freshly installed CentOS 6.8 hosts not being able to mount file shares via NFS v4 protocol. The mount command just kind of hangs. I verified it's not a connectivity issue, ie connections to the filer's port 2049 are possible.
Also NFS v3 works just fine.
Did anyone experience anything like that?
Cheers, Vladimir _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
That wouldn’t cause hangs though.
Official NetApp NFS support statement is that we support any client that follows the RFC specification. No custom kernels. RHEL/CentOS 6.8 should be fine.
I’d look into a getting a packet capture of the hang to see where they are occurring in the process.
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of tmac Sent: Wednesday, November 02, 2016 7:08 AM To: Vladimir Zhigulin Cc: toasters@teaparty.net Subject: Re: ONTAP9 NFSv4 + CentOS 6.8 does not work
Should check the IMT... refers to http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=1035160 and that you should be on GA 6.8 and Supported Kernel version:- kernel-2.6.32-642.el6.x86_64 Supported NFS-utils version:- nfs-utils-1.2.3-70.el6.x86_64 Supported libtirpc version:- libtirpc-0.2.1-11.el6.x86_64
This is for NFS4.1...which you may or may not be using....worth checking into.
--tmac
Tim McCarthy, Principal Consultant
Proud Member of the #NetAppATeamhttps://twitter.com/NetAppATeam
I Blog at TMACsRackhttps://tmacsrack.wordpress.com/
On Wed, Nov 2, 2016 at 6:06 AM, Momonth <momonth@gmail.commailto:momonth@gmail.com> wrote: Hi All,
We upgraded our 6 nodes cluster to ONTAP9 yesterday. Today I discovered some freshly installed CentOS 6.8 hosts not being able to mount file shares via NFS v4 protocol. The mount command just kind of hangs. I verified it's not a connectivity issue, ie connections to the filer's port 2049 are possible.
Also NFS v3 works just fine.
Did anyone experience anything like that?
Cheers, Vladimir _______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
What version did you upgrade from?
Sent from Yahoo Mail on Android
On Wed, Nov 2, 2016 at 11:02 AM, Parisi, JustinJustin.Parisi@netapp.com wrote: That wouldn’t cause hangs though.
Official NetApp NFS support statement is that we support any client that follows the RFC specification. No custom kernels. RHEL/CentOS 6.8 should be fine.
I’d look into a getting a packet capture of the hang to see where they are occurring in the process.
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net]On Behalf Of tmac Sent: Wednesday, November 02, 2016 7:08 AM To: Vladimir Zhigulin Cc: toasters@teaparty.net Subject: Re: ONTAP9 NFSv4 + CentOS 6.8 does not work
Should check the IMT...
refers to http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=1035160
and that you should be on GA 6.8 and
Supported Kernel version:- kernel-2.6.32-642.el6.x86_64 Supported NFS-utils version:- nfs-utils-1.2.3-70.el6.x86_64 Supported libtirpc version:- libtirpc-0.2.1-11.el6.x86_64
This is for NFS4.1...which you may or may not be using....worth checking into.
--tmac
Tim McCarthy, Principal Consultant
Proud Member of the #NetAppATeam
I Blog at TMACsRack
On Wed, Nov 2, 2016 at 6:06 AM, Momonth momonth@gmail.com wrote:
Hi All,
We upgraded our 6 nodes cluster to ONTAP9 yesterday. Today I discovered some freshly installed CentOS 6.8 hosts not being able to mount file shares via NFS v4 protocol. The mount command just kind of hangs. I verified it's not a connectivity issue, ie connections to the filer's port 2049 are possible.
Also NFS v3 works just fine.
Did anyone experience anything like that?
Cheers, Vladimir _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
We upgraded from cDTO 8.3.2
Something to add: existing v4 clients (ie those that mounted the share before ONTAP 9 upgrade) are just fine after the upgrade, only brand new NFS clients are having issues.
On Wed, Nov 2, 2016 at 6:25 PM, Fred Grieco fredgrieco@yahoo.com wrote:
What version did you upgrade from?
Sent from Yahoo Mail on Android
On Wed, Nov 2, 2016 at 11:02 AM, Parisi, Justin Justin.Parisi@netapp.com wrote:
That wouldn’t cause hangs though.
Official NetApp NFS support statement is that we support any client that follows the RFC specification. No custom kernels. RHEL/CentOS 6.8 should be fine.
I’d look into a getting a packet capture of the hang to see where they are occurring in the process.
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of tmac Sent: Wednesday, November 02, 2016 7:08 AM To: Vladimir Zhigulin Cc: toasters@teaparty.net Subject: Re: ONTAP9 NFSv4 + CentOS 6.8 does not work
Should check the IMT...
refers to http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=1035160
and that you should be on GA 6.8 and
Supported Kernel version:- kernel-2.6.32-642.el6.x86_64 Supported NFS-utils version:- nfs-utils-1.2.3-70.el6.x86_64 Supported libtirpc version:- libtirpc-0.2.1-11.el6.x86_64
This is for NFS4.1...which you may or may not be using....worth checking into.
--tmac
Tim McCarthy, Principal Consultant
Proud Member of the #NetAppATeam
I Blog at TMACsRack
On Wed, Nov 2, 2016 at 6:06 AM, Momonth momonth@gmail.com wrote:
Hi All,
We upgraded our 6 nodes cluster to ONTAP9 yesterday. Today I discovered some freshly installed CentOS 6.8 hosts not being able to mount file shares via NFS v4 protocol. The mount command just kind of hangs. I verified it's not a connectivity issue, ie connections to the filer's port 2049 are possible.
Also NFS v3 works just fine.
Did anyone experience anything like that?
Cheers, Vladimir _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Do new installs of older RHEL 6.x work with NFSv4 on the Ontap9 setup? Just curious... and what does an 'strace' of the NFS process show? Or a tcpdump trace? Can you share the options and full command line you're using for the mounts? Is your Authentication working properly on both sides?
And have you disabled selinux as well? That's always bit me in the ass at times.
John
Momonth> We upgraded from cDTO 8.3.2 Momonth> Something to add: existing v4 clients (ie those that mounted the share Momonth> before ONTAP 9 upgrade) are just fine after the upgrade, only brand Momonth> new NFS clients are having issues.
Momonth> On Wed, Nov 2, 2016 at 6:25 PM, Fred Grieco fredgrieco@yahoo.com wrote:
What version did you upgrade from?
Sent from Yahoo Mail on Android
On Wed, Nov 2, 2016 at 11:02 AM, Parisi, Justin Justin.Parisi@netapp.com wrote:
That wouldn’t cause hangs though.
Official NetApp NFS support statement is that we support any client that follows the RFC specification. No custom kernels. RHEL/CentOS 6.8 should be fine.
I’d look into a getting a packet capture of the hang to see where they are occurring in the process.
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of tmac Sent: Wednesday, November 02, 2016 7:08 AM To: Vladimir Zhigulin Cc: toasters@teaparty.net Subject: Re: ONTAP9 NFSv4 + CentOS 6.8 does not work
Should check the IMT...
refers to http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=1035160
and that you should be on GA 6.8 and
Supported Kernel version:- kernel-2.6.32-642.el6.x86_64 Supported NFS-utils version:- nfs-utils-1.2.3-70.el6.x86_64 Supported libtirpc version:- libtirpc-0.2.1-11.el6.x86_64
This is for NFS4.1...which you may or may not be using....worth checking into.
--tmac
Tim McCarthy, Principal Consultant
Proud Member of the #NetAppATeam
I Blog at TMACsRack
On Wed, Nov 2, 2016 at 6:06 AM, Momonth momonth@gmail.com wrote:
Hi All,
We upgraded our 6 nodes cluster to ONTAP9 yesterday. Today I discovered some freshly installed CentOS 6.8 hosts not being able to mount file shares via NFS v4 protocol. The mount command just kind of hangs. I verified it's not a connectivity issue, ie connections to the filer's port 2049 are possible.
Also NFS v3 works just fine.
Did anyone experience anything like that?
Cheers, Vladimir _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
Momonth> _______________________________________________ Momonth> Toasters mailing list Momonth> Toasters@teaparty.net Momonth> http://www.teaparty.net/mailman/listinfo/toasters
On Wed, Nov 2, 2016 at 10:49 PM, John Stoffel john@stoffel.org wrote:
Do new installs of older RHEL 6.x work with NFSv4 on the Ontap9 setup?
I haven't tried anything else yet.
Just curious... and what does an 'strace' of the NFS process show? Or a tcpdump trace? Can you share the options and full command line you're using for the mounts? Is your Authentication working properly on both sides?
I'm going to collect tcpdump and strace.
We have provisioning system and puppetized configs, so all servers are supposed to have the same settings etc. However servers can land in different VLANs.
And have you disabled selinux as well? That's always bit me in the ass at times.
SELinux is disabled.
Just a thought - I ran into a similar problem once where what happened was I upgraded and ended up with a system enabled for 4.1 when previously only 4.0 was enabled. A few clients glitched because they detected the change and tried to connect with NFSv4.1. I had to explicitly force the mount to v4.0 only.
-----Original Message----- From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Momonth Sent: Thursday, November 03, 2016 12:51 PM To: John Stoffel john@stoffel.org Cc: toasters@teaparty.net Subject: Re: ONTAP9 NFSv4 + CentOS 6.8 does not work
On Wed, Nov 2, 2016 at 10:49 PM, John Stoffel john@stoffel.org wrote:
Do new installs of older RHEL 6.x work with NFSv4 on the Ontap9 setup?
I haven't tried anything else yet.
Just curious... and what does an 'strace' of the NFS process show? Or a tcpdump trace? Can you share the options and full command line you're using for the mounts? Is your Authentication working properly on both sides?
I'm going to collect tcpdump and strace.
We have provisioning system and puppetized configs, so all servers are supposed to have the same settings etc. However servers can land in different VLANs.
And have you disabled selinux as well? That's always bit me in the ass at times.
SELinux is disabled. _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
I would like to just suggest... I'm aware that some ppl migh want to start a long debate about this (which I will then just have to dodge)
If you really really want to deploy v4.x in an ubiquitous manner, scale it up/out a lot so to speak for some good reason you have thought trough really well.
Do it with NFS clients that can talk 4.1 (i.e. minorversion=1.) NFS clients which do not understand v4.1 but only 4.0, deny them any v4 access, this can be done by simply disabling v4.0 in the vservers configs, but keep 4.1 enabled. Just make those older NFS clients fall back to v3 and all will work, the users won't notice any difference and there's no real loss of anything that you will miss
(AUTH_GSS, aka RPCSEC_GSS, with sec=krb5{,i,p} is supported and works just fine with NFSv3 in ONTAP and in almost all if not all, wide-spread Linux distros as well as Oracle Solaris. Don't know about *BSD).
/M
On 2016-11-03 19:41, Steiner, Jeffrey wrote:
Just a thought - I ran into a similar problem once where what happened was I upgraded and ended up with a system enabled for 4.1 when previously only 4.0 was enabled. A few clients glitched because they detected the change and tried to connect with NFSv4.1. I had to explicitly force the mount to v4.0 only.
Are those clients set up the same...i.e the nfs domain is identical? Network/netmask is correct?
Per others, a packet trace from the client and the netapp may prove useful.
--tmac
*Tim McCarthy, **Principal Consultant*
*Proud Member of the #NetAppATeam https://twitter.com/NetAppATeam*
*I Blog at TMACsRack https://tmacsrack.wordpress.com/*
RHCE6 110-107-141 https://www.redhat.com/wapps/training/certification/verify.html?certNumber=110-107-141&isSearch=False&verify=Verify
On Wed, Nov 2, 2016 at 3:37 PM, Momonth momonth@gmail.com wrote:
We upgraded from cDTO 8.3.2
Something to add: existing v4 clients (ie those that mounted the share before ONTAP 9 upgrade) are just fine after the upgrade, only brand new NFS clients are having issues.
On Wed, Nov 2, 2016 at 6:25 PM, Fred Grieco fredgrieco@yahoo.com wrote:
What version did you upgrade from?
Sent from Yahoo Mail on Android
On Wed, Nov 2, 2016 at 11:02 AM, Parisi, Justin Justin.Parisi@netapp.com wrote:
That wouldn’t cause hangs though.
Official NetApp NFS support statement is that we support any client that follows the RFC specification. No custom kernels. RHEL/CentOS 6.8 should
be
fine.
I’d look into a getting a packet capture of the hang to see where they
are
occurring in the process.
From: toasters-bounces@teaparty.net [mailto:toasters-bounces@
teaparty.net]
On Behalf Of tmac Sent: Wednesday, November 02, 2016 7:08 AM To: Vladimir Zhigulin Cc: toasters@teaparty.net Subject: Re: ONTAP9 NFSv4 + CentOS 6.8 does not work
Should check the IMT...
refers to http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=1035160
and that you should be on GA 6.8 and
Supported Kernel version:- kernel-2.6.32-642.el6.x86_64 Supported
NFS-utils
version:- nfs-utils-1.2.3-70.el6.x86_64 Supported libtirpc version:- libtirpc-0.2.1-11.el6.x86_64
This is for NFS4.1...which you may or may not be using....worth checking into.
--tmac
Tim McCarthy, Principal Consultant
Proud Member of the #NetAppATeam
I Blog at TMACsRack
On Wed, Nov 2, 2016 at 6:06 AM, Momonth momonth@gmail.com wrote:
Hi All,
We upgraded our 6 nodes cluster to ONTAP9 yesterday. Today I discovered some freshly installed CentOS 6.8 hosts not being able to mount file shares via NFS v4 protocol. The mount command just kind of hangs. I verified it's not a connectivity issue, ie connections to the filer's port 2049 are possible.
Also NFS v3 works just fine.
Did anyone experience anything like that?
Cheers, Vladimir _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters
I'm not 100% sure what has happened, most likely in-house "server provisioning tool" failed in such a way that some NFS related packages were not installed. As after re-installing the server in question from scratch everything runs fine. Unfortunately I was not able to collect enough debug info to have better conclusions.
On Wed, Nov 2, 2016 at 11:06 AM, Momonth momonth@gmail.com wrote:
Hi All,
We upgraded our 6 nodes cluster to ONTAP9 yesterday. Today I discovered some freshly installed CentOS 6.8 hosts not being able to mount file shares via NFS v4 protocol. The mount command just kind of hangs. I verified it's not a connectivity issue, ie connections to the filer's port 2049 are possible.
Also NFS v3 works just fine.
Did anyone experience anything like that?
Cheers, Vladimir
This issues re-occurred on already running (ie it's not freshly provisioned servers) CentOS 6.6, and also on CentOS 7.2.1511. What I observed is that NFS clients could not mount the share on ONTAP 9 SVM after "lif migrate .." command.
This time I was able to collect tcpdump while running mount command. It was as follows:
1. Client sends (decoded by wireshark):
Remote Procedure Call, Type:Call XID:0x9fc37eb7 Network File System, Ops(3): SETCLIENTID_CONFIRM, PUTROOTFH, GETATTR [Program Version: 4] [V4 Procedure: COMPOUND (1)] Tag: <EMPTY> minorversion: 0 Operations (count: 3): SETCLIENTID_CONFIRM, PUTROOTFH, GETATTR Opcode: SETCLIENTID_CONFIRM (36) clientid: 0x36c1330000000076 verifier: 0x0000000000000000 Opcode: PUTROOTFH (24) Opcode: GETATTR (9) Attr mask: 0x00000400 (Lease_Time) reqd_attr: Lease_Time (10) [Main Opcode: SETCLIENTID_CONFIRM (36)]
2. SVM responds with:
Network File System, Ops(2): SETCLIENTID_CONFIRM PUTROOTFH(NFS4ERR_DELAY) [Program Version: 4] [V4 Procedure: COMPOUND (1)] Status: NFS4ERR_DELAY (10008) Tag: <EMPTY> length: 0 contents: <EMPTY> Operations (count: 2) Opcode: SETCLIENTID_CONFIRM (36) Status: NFS4_OK (0) Opcode: PUTROOTFH (24) Status: NFS4ERR_DELAY (10008) [Main Opcode: SETCLIENTID_CONFIRM (36)]
IMO, it was in an endless loop until I CTRL+C'ed the mount command.
On Tue, Nov 8, 2016 at 5:23 PM, Momonth momonth@gmail.com wrote:
I'm not 100% sure what has happened, most likely in-house "server provisioning tool" failed in such a way that some NFS related packages were not installed. As after re-installing the server in question from scratch everything runs fine. Unfortunately I was not able to collect enough debug info to have better conclusions.
On Wed, Nov 2, 2016 at 11:06 AM, Momonth momonth@gmail.com wrote:
Hi All,
We upgraded our 6 nodes cluster to ONTAP9 yesterday. Today I discovered some freshly installed CentOS 6.8 hosts not being able to mount file shares via NFS v4 protocol. The mount command just kind of hangs. I verified it's not a connectivity issue, ie connections to the filer's port 2049 are possible.
Also NFS v3 works just fine.
Did anyone experience anything like that?
Cheers, Vladimir
-
Fred Grieco -
John Stoffel -
Michael Bergman -
Momonth -
Parisi, Justin -
Steiner, Jeffrey -
tmac