We get lots of questions about NFSv4, especially for the security and locking aspects.

The main issues people see with it are performance (especially with high metadata workloads) and complexity in setup.

It’s a stateful protocol, so it also sees disruption during failovers.

Mostly, people use it for the following reasons:

* Their business makes it a requirement for the security * They have an application that requires it for the locking * They want ACLs * They only want to open up a single port over a firewall

If you just want it for ACLs, ONTAP allows you to set v4 ACLs and still use NFSv3:

https://whyistheinternetbroken.wordpress.com/2017/08/11/nfsv4acls-nfsv3mount...

From: Toasters toasters-bounces@teaparty.net On Behalf Of Sebastian P. Goetze Sent: Friday, October 2, 2020 3:26 AM To: Douglas Siggins siggins@gmail.com Cc: Toasters Toasters@teaparty.net Subject: Re:

NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe.

I teach NetApp and do have students once in a while using NFS v4, but usually only when necessary, e.g. routing through firewalls (which in my experience seems to be the #1 reason using v4). I did have people using pNFS in production, but that's rare.

Some are evaluating, though. Maybe when multipathing is supported (regular like VMware, not pNFS), there will be an uptick in adoption...

My 2c

Greetings from Northfriesland

Sebastian

sent from my mobile, spellchecker might have messed up...

On Thu, 1 Oct 2020, 02:33 Douglas Siggins <siggins@gmail.commailto:siggins@gmail.com> wrote: Yes, thanks folks. It appears there's not much v4 out there as suspected.

On Wed, Sep 30, 2020, 20:12 Jeff Cleverley via Toasters <toasters@teaparty.netmailto:toasters@teaparty.net> wrote:

---------- Forwarded message ---------- From: Jeff Cleverley <jeff.cleverley@broadcom.commailto:jeff.cleverley@broadcom.com> To: Wayne McCormick <Wayne.McCormick@sjrb.camailto:Wayne.McCormick@sjrb.ca> Cc: Toasters <toasters@teaparty.netmailto:toasters@teaparty.net> Bcc: Date: Wed, 30 Sep 2020 18:03:42 -0600 Subject: Re: Any Openstack/Netapp FAS deployments w cinder NFS, v3 or v4 Wayne,

Just so you know your posts are working, I'll reply. I have no experience with that at this point in time. Personally I don't see an issue posting to ensure the NetApp configurations are correct or compatible with the configurations.

Jeff

On Fri, Sep 25, 2020 at 9:09 AM Wayne McCormick <Wayne.McCormick@sjrb.camailto:Wayne.McCormick@sjrb.ca> wrote: I hope I’m doing this right… (posting to the list I mean).

I have four MySQL servers using NFSv4, but not with krb5. The rest of the environment is v3.

We did do a pilot with OpenStack Container Platform and we used v3 (v4 was enabled, so it may have used it) provisioning via Trident with success. We also used Trident to provision iSCSI volumes. Worked quite well.

Hope that helps a little.

Wayne

From: Toasters <toasters-bounces@teaparty.netmailto:toasters-bounces@teaparty.net> On Behalf Of Douglas Siggins Sent: Friday, September 25, 2020 9:01 AM To: Toasters <toasters@teaparty.netmailto:toasters@teaparty.net> Subject: Re: Any Openstack/Netapp FAS deployments w cinder NFS, v3 or v4

CAUTION: This email is from an external source. Do not click links or open attachments unless you recognize the sender and know the content is safe. Crickets ? :)

Alright let us remove openstack qualifier. Does anyone use v4 or know anyone that does with Netapp FAS?

On Tue, Sep 22, 2020 at 6:53 PM Douglas Siggins <siggins@gmail.commailto:siggins@gmail.com> wrote: Greetings, Just reaching out for experiences on Openstack deployments with Netapp NFS. Are you using the default v4 or have switched the mount options to v3? We have a pretty good base of ISCSI, but based on performance and manageability are considering making the jump.

Speaking of NFSv4, outside of openstack has anyone made any large deployments? Is it as solid as v3? using uid/gid, is anyone using sec=krb5?

Regards, Douglas _______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net https://www.teaparty.net/mailman/listinfo/toasters

-- ** Please note my email address has changed to jeff.cleverley@broadcom.commailto:jeff.cleverley@broadcom.com

Jeff Cleverley Factory Systems Engineer 4380 Ziegler Road Building 1, Dock 1 Fort Collins, Colorado 80525 970-288-4611

---------- Forwarded message ---------- From: Jeff Cleverley via Toasters <toasters@teaparty.netmailto:toasters@teaparty.net> To: Wayne McCormick <Wayne.McCormick@sjrb.camailto:Wayne.McCormick@sjrb.ca> Cc: Toasters <toasters@teaparty.netmailto:toasters@teaparty.net> Bcc: Date: Thu, 1 Oct 2020 01:04:02 +0100 Subject: _______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net https://www.teaparty.net/mailman/listinfo/toasters _______________________________________________ Toasters mailing list Toasters@teaparty.netmailto:Toasters@teaparty.net https://www.teaparty.net/mailman/listinfo/toasters

Re: v3 vs. v4.x security...

With NFSv3, the sideband protocols are not Kerberized when with krb5*. So all that gets kerberized is the NFS protocol; mount, portmap, nlm, etc are all plaintext. That's one aspect of security that v4.x provides with everything encapsulated. Those sideband protocols can contain some information about the mount that you don't want to be readable.

For example, MOUNT shows:

- machine account name - uid/gid mounting - auxiliary groups - mount path - filehandle info

NFSv4.x stores all that info with PUTROOTFH calls in the NFS packets, so that all gets GSS wrapped.

Plus, you can configure v4.x to force domain string ID auth only, whereas v3 is numeric ID only. That offers some flexibility (such as being able to have different UIDs on storage and client, provided they can both resolve name@domain.com to a user), but also offers some security benefits when you tack on Kerberos UPN.

V4.x does offer some security benefits that v3 does not, but you're right that you can accomplish most of the v4 security stuff in v3.

-----Original Message----- From: Michael Bergman michael.bergman@ericsson.com Sent: Monday, October 5, 2020 6:18 AM To: Toasters toasters@teaparty.net Subject: Re: Any Openstack/Netapp FAS deployments w cinder NFS, v3 or v4

While the locking mechanism is understandable, the side band way (NLM protocol) in v3 really is a bummer (very very bad performance for example) this surprises me:

"Their business makes it a requirement for the security"

There's really nothing w.r.t. "security" in v4.1 and up that is crucial or important that cannot, with the same effort (client side effort...), be implemented with v3. In practice. There's the sec=krb5{,i,p} stuff (old 7-mode syntax there I know). All is supported in all sensible Linux distros I'm aware of (and Oracle Solaris if anyone cares about that anymore I'm not sure). Other NFS client implementations, sure, YMMV. But it should be residual by now.

Another dimension of security is what Justin pointed out below about NVSv4 ACLs: ONTAP allows you to "nfsv4acls-nfsv3mounts".

Remains one more thing then (in addition to the first thing I wrote): the fact that NFSv4.x uses one port 2049. Everything. So it's easier to "open the Firewall". But seriously. How hard can it be with NFSv3 and those few side band protocols? It's just a few more ports, in total like 5 (TCP and UDP for the side band protocols: rpcbind, MOUNT, NLM, status. Who uses rquota these days...??) It's not rocket science. We have tons of FW openings for NFSv3 here in our complex network environments internally inside Ericsson's "Customer Zero" (our Engineering and Telco Test environments). It's rarely, if ever, a problem per se. The problem is that the storage traffic is very intense and tends to kill the Firewalls. There are so many sessions going on at the same time that if you put a FW in the middle of the (*one* Firewall mind you...) its CPUs will be at 100% sooner or later

Last but not least: yes there are fundamental performance issues (still) with v4.x. We have done investigations and it's not looking good vs NFSv3. Yes, it's possible to get v4.1 (and pNFS) to perform well if you can get your workload to parallelise very much -- scale out. Be prepared that the work to fiddle with everything and tune all the details to get it it to perform well for you in your particular environment is VERY cumbersome.

If not, if you have "vertical" workload threads that need to do *a lot* (incl metadata) on one (1) mount point, then high risk for a painful experience with ONTAP's v4.1 today. Bottlenecks which have to be removed first and it will take time

/M

-------- Original Message -------- Subject: RE: Re: Date: Fri, 2 Oct 2020 13:18:57 +0000 From: Parisi, Justin Justin.Parisi@netapp.com To: Sebastian P. Goetze spgoetze@gmail.com, Douglas Siggins siggins@gmail.com CC: Toasters Toasters@teaparty.net

We get lots of questions about NFSv4, especially for the security and locking aspects.

The main issues people see with it are performance (especially with high metadata workloads) and complexity in setup.

It's a stateful protocol, so it also sees disruption during failovers.

Mostly, people use it for the following reasons:

* Their business makes it a requirement for the security * They have an application that requires it for the locking * They want ACLs * They only want to open up a single port over a firewall

If you just want it for ACLs, ONTAP allows you to set v4 ACLs and still use NFSv3:

https://whyistheinternetbroken.wordpress.com/2017/08/11/nfsv4acls-nfsv3mount...

_________________________________________________ *From:* Toasters toasters-bounces@teaparty.net *On Behalf Of *Sebastian P. Goetze *Sent:* Friday, October 2, 2020 3:26 AM *To:* Douglas Siggins siggins@gmail.com *Cc:* Toasters Toasters@teaparty.net *Subject:* Re:

I teach NetApp and do have students once in a while using NFS v4, but usually only when necessary, e.g. routing through firewalls (which in my experience seems to be the #1 reason using v4). I did have people using pNFS in production, but that's rare.

Some are evaluating, though. Maybe when multipathing is supported (regular like VMware, not pNFS), there will be an uptick in adoption...

My 2c

Greetings from Northfriesland

Sebastian

Root gets access to Kerberos depending on what you mapped the client SPN to in ONTAP. If you mapped it to root, it gets root. If you mapped it to another user, it becomes that user and gets that user's permissions.

-----Original Message----- From: Michael Bergman michael.bergman@ericsson.com Sent: Monday, October 5, 2020 9:43 AM To: Parisi, Justin Justin.Parisi@netapp.com Cc: Toasters toasters@teaparty.net Subject: Re: Any Openstack/Netapp FAS deployments w cinder NFS, v3 or v4

NetApp Security WARNING: This is an external email. Do not click links or open attachments unless you recognize the sender and know the content is safe.

You're right. The krb5 is not covering the side band protocols pre- v4.x. I forgot about that bit, thanks for the reminder.

In some "top secret" environments, I could see that sniffing in the Network and seeing things like Computer Object name, uid/gid, mount path and FH could by someone be seen as a security risk. *Eve* if the actual datagrams with user data could never be seen. For most use cases, I contend that this is just an ISO 27001 tick box item, though.

A thought: OTOH if anyone, anyone at all, is root on an NFS client then can they not somehow from inside it see that MOUNT information anyway? (Even if the info is encrypted in transit with krb5 then.) I'm not sure how that bit works, never looked into it

Justin Parisi wrote:

you can configure v4.x to force domain string ID auth only [...] ,but also offers some security benefits when you tack on Kerberos UPN.

I did try a setup with this at some point in the past and then got really tired of the whole thing and tore it down. It's just... cumbersome. I can't think of any non esoteric Linux (Unix) based environment with lots of NFS traffic which would be reasonably manageable with such a config

/M

-------- Original Message -------- Subject: Re: Any Openstack/Netapp FAS deployments w cinder NFS, v3 or v4 Date: Mon, 5 Oct 2020 13:29:19 +0000 From: Parisi, Justin Justin.Parisi@netapp.com To: NGC-michael.bergman-ericsson.com michael.bergman@ericsson.com, Toasters toasters@teaparty.net

Re: v3 vs. v4.x security...

With NFSv3, the sideband protocols are not Kerberized when with krb5*. So all that gets kerberized is the NFS protocol; mount, portmap, nlm, etc are all plaintext. That's one aspect of security that v4.x provides with everything encapsulated. Those sideband protocols can contain some information about the mount that you don't want to be readable.

For example, MOUNT shows:

- machine account name - uid/gid mounting - auxiliary groups - mount path - filehandle info

NFSv4.x stores all that info with PUTROOTFH calls in the NFS packets, so that all gets GSS wrapped.

Plus, you can configure v4.x to force domain string ID auth only, whereas v3 is numeric ID only. That offers some flexibility (such as being able to have different UIDs on storage and client, provided they can both resolve name@domain.com to a user), but also offers some security benefits when you tack on Kerberos UPN.

V4.x does offer some security benefits that v3 does not, but you're right that you can accomplish most of the v4 security stuff in v3.