So, "access denied" doesn't mean you have a Kerberos issue necessarily. If it were Kerberos, I'd expect an error in the trace on the ticket exchange.

Does klist -e show you got a ticket from the storage?

What does the export look like for that volume? What about "fsecurity show /vol/volname"?

-----Original Message----- From: Ray Van Dolson [mailto:rvandolson@esri.com] Sent: Friday, July 29, 2016 10:10 AM To: Parisi, Justin Cc: toasters@teaparty.net Subject: Re: Kerberized NFSv4 in 7-Mode (AD)

Here you go (setspn for the filer):

C:\windows\system32>setspn -L REDSTRNAPC2P12 Registered ServicePrincipalNames for CN=REDSTRNAPC2P12,OU=XXX,DC=esri,DC=com: nfs/red-str-napc2-p12.esri.com host/red-str-napc2-p12.esri.com host/red-str-napc2-p12 HOST/redstrnapc2p12.esri.com HOST/redstrnapc2p12

Note, I ended up manually adding the nfs entry as I was seeing errors in rpc.gssd which disappeared once I did so.

The exact error from the Wireshark capture (client perspective) is:

V4 Reply (Call in 1) ACCESS, [Access Denied: RD LU MD XT], [Allowed: DL]

Details:

Network File System, Ops(3): PUTFH ACCESS GETATTR [Program Version: 4] [V4 Procedure: COMPOUND (1)] Status: NFS4_OK (0) Tag: <EMPTY> length: 0 contents: <EMPTY> Operations (count: 3) Opcode: PUTFH (22) Status: NFS4_OK (0) Opcode: ACCESS (3), [Access Denied: RD LU MD XT], [Allowed: DL] Status: NFS4_OK (0) Supported types (of requested): 0x1f .... ...1 = 0x01 READ: supported .... ..1. = 0x02 LOOKUP: supported .... .1.. = 0x04 MODIFY: supported .... 1... = 0x08 EXTEND: supported ...1 .... = 0x10 DELETE: supported Access rights (of requested): 0x10 .... ...0 = 0x01 READ: *Access Denied* .... ..0. = 0x02 LOOKUP: *Access Denied* .... .0.. = 0x04 MODIFY: *Access Denied* .... 0... = 0x08 EXTEND: *Access Denied* ...1 .... = 0x10 DELETE: allowed Opcode: GETATTR (9) Status: NFS4_OK (0) Attr mask[0]: 0x00000018 (CHANGE, SIZE) reqd_attr: CHANGE (3) changeid: 1578161353901524080 reqd_attr: SIZE (4) size: 4096 Attr mask[1]: 0x00300000 (TIME_METADATA, TIME_MODIFY) reco_attr: TIME_METADATA (52) seconds: 1469777295 nseconds: 294438000 reco_attr: TIME_MODIFY (53) seconds: 1469777295 nseconds: 294438000 [Main Opcode: ACCESS (3)]

Ray

On Fri, Jul 29, 2016 at 01:39:44PM +0000, Parisi, Justin wrote:

Can you show me the setspn -L output for that account?

Generally, yes, the filer adds those nfs/ entries and it doesn't matter if the account exists or not.

I'd also recommend getting a packet trace during the failure to see what the exact error is.

-----Original Message----- From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of Ray Van Dolson Sent: Friday, July 29, 2016 3:24 AM To: toasters@teaparty.net Subject: Kerberized NFSv4 in 7-Mode (AD)

Need some guidance in getting this working.

Have an 8.2.2P1 7-mode filer, joined to AD (working fine w/ CIFS) and have used 'nfs setup' to enable Kerberized NFS. However, I noticed that no nfs/* entries are present when I run setspn -L <FILER> from a Windows box. I was under the impression that nfs setup should create these? Or, since we're joined to AD already, perhaps the established machine account is sufficient to obtain one on the fly...

I have the following export on a volume with ntfs security style:

/vol/raytest -sec=krb5:krb5i:krb5p,rw,anon=0,nosuid

On my client (RHEL7+SSSD), I'm fairly confident my setup for Kerberos is good. I can SSH in with a Kerberos login, do kinit, etc and connect to other Kerberized services (like doing an ldapearch do AD without being prompted for a password using -Y GSSAPI as long as I have a valid tgt).

I've tried my krb5.conf with and without the following:

default_tgs_enctypes = RC4-HMAC DES-CBC-MD5 DES-CBC-CRC default_tkt_enctypes = RC4-HMAC DES-CBC-MD5 DES-CBC-CRC preferred_enctypes = RC4-HMAC DES-CBC-MD5 DES-CBC-CRC

I also ran 'net ads keytab add nfs' on my client and see that I have a valid nfs/ entry in my machine's keytab file.

If I run the following (either as root or as a user -- below is as a user with a valid TGT):

$ sudo mount.nfs4 -o vers=4,sec=krb5 red-str-napc2-p12.esri.com:/vol/raytest /mnt/nfs

I get a successful mount and also see a valid nfs ticket in klist. However, when I try to cd to /mnt/nfs, I get a "Permission Denied" error.

I have rpc.gssd running in the foreground in verbose mode but don't really see anything odd.

Wireshark tells me I've asked for read, lookup, modify, extend and delete permissions but that access was denied and only delete was allowed. I can see that the RPC portion of the packets are speaking Kerberos, but I can't tell much more than that.

DNS, NTP, etc. is all set up and working on both sides. I've used sectrace on the /vol/raytest path above to try and catch issues, but get no hits making me think it's NFS on the NetApp rejecting things, not the file system layer... fsecurity shows that everything is wide open and from a CIFS client using the same user as on the Linux box, I can create files on the same path with no issues.

Kinda stumped at this point. Any suggestions?

Ray

Oh yeah. Forgot to mention the name mapping piece.

In addition to Windows-> UNIX, you need a local user named "nfs" to map to the nfs/ SPN. Add a user to /etc/passwd or in LDAP.

As for TR-4073, it *does* have a 7-Mode LDAP section on page 130 (mapping attributes) and 270:

http://www.netapp.com/us/media/tr-4073.pdf

There is also a crusty old pair of TRs on Kerberos/LDAP and AD:

http://media.netapp.com/documents/tr-3458.pdf http://media.netapp.com/documents/tr-3457.pdf

-----Original Message----- From: toasters-bounces@teaparty.net [mailto:toasters-bounces@teaparty.net] On Behalf Of andrei.borzenkov@ts.fujitsu.com Sent: Friday, July 29, 2016 10:13 AM To: NGC-rvandolson-esri.com; toasters@teaparty.net Subject: RE: Kerberized NFSv4 in 7-Mode (AD)

You need to have valid Unix to Windows user mapping, otherwise filer does not know how to verify your credentials. You can quickly verify it by setting wafl.default_nt_user to valid NT user that has permissions to access this volume (you may want to clear WCC by "wcc -x").

Note that Kerberos integration has two parts - authentication and identity management. The former is simple and well documented and does not require any additional changes usually; it is trivial to obtain ticket for a valid NT user.

The latter is more challenging. You need to modify AD to support Unix user enumeration (i.e. you need to add Unix attributes to users in AD).

See e.g. TR-4073; while it is for cDOT, it describes steps necessary on AD side which remain valid for 7-Mode. It does not describe how to enable LDAP for user name lookup on 7-Mode, but presumably it should be known already :)

I just verified that

1) adding filer to AD (cifs setup) 2) enabling Kerberos for NFS (nfs setup) 3) configuring filer for LDAP lookup

enables access to both ntfs and unix qtrees.

Step 3 is *NOT* performed by enabling Kerberos for NFS! At the minimum you need to

a) options ldap.ADdomain your.windows.domain b) options ldap.enable on

and of course verify that ldap is listed in /etc/nsswtch.conf

I also had to add c) options ldap.nssmap.objectClass.posixAccount user

after enabling Unix attribute as per TR-4073 on Windows 2012 R2 server. Otherwise user lookup failed and all accesses were as nobody.

HTH

--- With best regards

Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS

FUJITSU Zemlyanoy Val Street, 9, 105 064 Moscow, Russian Federation Tel.: +7 495 730 62 20 ( reception) Mob.: +7 916 678 7208 Fax: +7 495 730 62 14 E-mail: Andrei.Borzenkov@ts.fujitsu.com Web: ru.fujitsu.com Company details: ts.fujitsu.com/imprint This communication contains information that is confidential, proprietary in nature and/or privileged.  It is for the exclusive use of the intended recipient(s). If you are not the intended recipient(s) or the person responsible for delivering it to the intended recipient(s), please note that any form of dissemination, distribution or copying of this communication is strictly prohibited and may be unlawful. If you have received this communication in error, please immediately notify the sender and delete the original communication. Thank you for your cooperation. Please be advised that neither Fujitsu, its affiliates, its employees or agents accept liability for any errors, omissions or damages caused by delays of receipt or by any virus infection in this message or its attachments, or which may otherwise arise as a result of this e-mail transmission.

-----Original Message----- From: toasters-bounces@teaparty.net [mailto:toasters- bounces@teaparty.net] On Behalf Of Ray Van Dolson Sent: Friday, July 29, 2016 10:24 AM To: toasters@teaparty.net Subject: Kerberized NFSv4 in 7-Mode (AD)

Need some guidance in getting this working.

Have an 8.2.2P1 7-mode filer, joined to AD (working fine w/ CIFS) and have used 'nfs setup' to enable Kerberized NFS. However, I noticed that no nfs/* entries are present when I run setspn -L <FILER> from a Windows box. I was under the impression that nfs setup should create these? Or, since we're joined to AD already, perhaps the established machine account is sufficient to obtain one on the fly...

I have the following export on a volume with ntfs security style:

/vol/raytest -sec=krb5:krb5i:krb5p,rw,anon=0,nosuid

On my client (RHEL7+SSSD), I'm fairly confident my setup for Kerberos is good. I can SSH in with a Kerberos login, do kinit, etc and connect to other Kerberized services (like doing an ldapearch do AD without being prompted for a password using -Y GSSAPI as long as I have a valid tgt).

I've tried my krb5.conf with and without the following:

default_tgs_enctypes = RC4-HMAC DES-CBC-MD5 DES-CBC-CRC default_tkt_enctypes = RC4-HMAC DES-CBC-MD5 DES-CBC-CRC preferred_enctypes = RC4-HMAC DES-CBC-MD5 DES-CBC-CRC

I also ran 'net ads keytab add nfs' on my client and see that I have a valid nfs/ entry in my machine's keytab file.

If I run the following (either as root or as a user -- below is as a user with a valid TGT):

$ sudo mount.nfs4 -o vers=4,sec=krb5 red-str-napc2- p12.esri.com:/vol/raytest /mnt/nfs

I get a successful mount and also see a valid nfs ticket in klist. However, when I try to cd to /mnt/nfs, I get a "Permission Denied" error.

I have rpc.gssd running in the foreground in verbose mode but don't really see anything odd.

Wireshark tells me I've asked for read, lookup, modify, extend and delete permissions but that access was denied and only delete was allowed. I can see that the RPC portion of the packets are speaking Kerberos, but I can't tell much more than that.

DNS, NTP, etc. is all set up and working on both sides. I've used sectrace on the /vol/raytest path above to try and catch issues, but get no hits making me think it's NFS on the NetApp rejecting things, not the file system layer... fsecurity shows that everything is wide open and from a CIFS client using the same user as on the Linux box, I can create files on the same path with no issues.

Kinda stumped at this point. Any suggestions?

Ray _______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

_______________________________________________ Toasters mailing list Toasters@teaparty.net http://www.teaparty.net/mailman/listinfo/toasters

My understanding is that NetApp requires valid UNIX credentials for every connection, even CIFS connection to ntfs qtree. This is the reason default UNIX user is set and /etc/passwd is created by cifs setup (pure NFS filers do not have it).

Kerberized NFS client sends principle name; server takes primary as user name and attempts to resolve it using whatever mechanisms are defined by nsswitch.conf. If this fails, user gets anonymous credentials; in my testing anon option is ignored in this case and UID is always set to 65534. I wonder if this is configurable.

So what likely happens in your case is that every user is mapped to "pcuser" which is then looked up in domain. As it does not exist, access is denied.

And do not forget that with default NT user all access to files will be on behalf of this user, not on behalf of user authenticated initially. Which is true for any variant of user mapping.

Отправлено с iPhone

29 июля 2016 г., в 22:03, Ray Van Dolson rvandolson@esri.com написал(а):

Thanks, Andrei! I validated that setting wafl.default_nt_user does in fact resolve my issue.

I think I'm missing a few foundtational concepts here that will connect the rest of the dots for me...

With NFSv3+NIS access to an NFS share built on an ntfs-style security export off NetApp, I'm used to dealing with the following for identity and access control:

• Client UID/GID is sent in NFSv3 packet to NetApp.
• NetApp takes UID and looks it up via NIS and gets a username (in

theory this could be looked up via LDAP and AD SFU extensions I suppose though I've never tried setting it up on a NetApp).

• NetApp takes the username and looks it up in AD (if there's no

corresponding username in NIS for the given UID, access is denied or some other default action is defined by us via configuration).

• If the AD user exists, that user is evaluated against the NTFS style

permissions and access is granted or not. If the AD user doesn't exist, some other behvaior is taken (which we can tune via the default_nt_user setting, etc.).

(I think we can also use usermap.cfg here)

With NFSv4 + Kerberos, how is the flow different? From looking at the NFSv4 ACCESS request in my packet capture, I see nothing resembling a UID. My hope was that the username was somehow encaspulated in the RPCSEC_GSS stuff and that the NetApp would actually have essentially a real AD username to work with which would completely eliminate the need for any mapping of a numeric Unix UID to an AD username.

Ray

...

On Fri, Jul 29, 2016 at 02:12:32PM +0000, andrei.borzenkov@ts.fujitsu.com wrote: You need to have valid Unix to Windows user mapping, otherwise filer does not know how to verify your credentials. You can quickly verify it by setting wafl.default_nt_user to valid NT user that has permissions to access this volume (you may want to clear WCC by "wcc -x").

Note that Kerberos integration has two parts - authentication and identity management. The former is simple and well documented and does not require any additional changes usually; it is trivial to obtain ticket for a valid NT user.

The latter is more challenging. You need to modify AD to support Unix user enumeration (i.e. you need to add Unix attributes to users in AD).

See e.g. TR-4073; while it is for cDOT, it describes steps necessary on AD side which remain valid for 7-Mode. It does not describe how to enable LDAP for user name lookup on 7-Mode, but presumably it should be known already :)

I just verified that

1. adding filer to AD (cifs setup)
2. enabling Kerberos for NFS (nfs setup)
3. configuring filer for LDAP lookup

enables access to both ntfs and unix qtrees.

Step 3 is *NOT* performed by enabling Kerberos for NFS! At the minimum you need to

a) options ldap.ADdomain your.windows.domain b) options ldap.enable on

and of course verify that ldap is listed in /etc/nsswtch.conf

I also had to add c) options ldap.nssmap.objectClass.posixAccount user

after enabling Unix attribute as per TR-4073 on Windows 2012 R2 server. Otherwise user lookup failed and all accesses were as nobody.

HTH

---

With best regards

Andrei Borzenkov Senior system engineer FTS WEMEAI RUC RU SC TMS FOS

...

-----Original Message----- From: toasters-bounces@teaparty.net [mailto:toasters- bounces@teaparty.net] On Behalf Of Ray Van Dolson Sent: Friday, July 29, 2016 10:24 AM To: toasters@teaparty.net Subject: Kerberized NFSv4 in 7-Mode (AD)

Need some guidance in getting this working.

Have an 8.2.2P1 7-mode filer, joined to AD (working fine w/ CIFS) and have used 'nfs setup' to enable Kerberized NFS. However, I noticed that no nfs/* entries are present when I run setspn -L <FILER> from a Windows box. I was under the impression that nfs setup should create these? Or, since we're joined to AD already, perhaps the established machine account is sufficient to obtain one on the fly...

I have the following export on a volume with ntfs security style:

/vol/raytest -sec=krb5:krb5i:krb5p,rw,anon=0,nosuid

On my client (RHEL7+SSSD), I'm fairly confident my setup for Kerberos is good. I can SSH in with a Kerberos login, do kinit, etc and connect to other Kerberized services (like doing an ldapearch do AD without being prompted for a password using -Y GSSAPI as long as I have a valid tgt).

I've tried my krb5.conf with and without the following:

default_tgs_enctypes = RC4-HMAC DES-CBC-MD5 DES-CBC-CRC default_tkt_enctypes = RC4-HMAC DES-CBC-MD5 DES-CBC-CRC preferred_enctypes = RC4-HMAC DES-CBC-MD5 DES-CBC-CRC

I also ran 'net ads keytab add nfs' on my client and see that I have a valid nfs/ entry in my machine's keytab file.

If I run the following (either as root or as a user -- below is as a user with a valid TGT):

$ sudo mount.nfs4 -o vers=4,sec=krb5 red-str-napc2- p12.esri.com:/vol/raytest /mnt/nfs

I get a successful mount and also see a valid nfs ticket in klist. However, when I try to cd to /mnt/nfs, I get a "Permission Denied" error.

I have rpc.gssd running in the foreground in verbose mode but don't really see anything odd.

Wireshark tells me I've asked for read, lookup, modify, extend and delete permissions but that access was denied and only delete was allowed. I can see that the RPC portion of the packets are speaking Kerberos, but I can't tell much more than that.

DNS, NTP, etc. is all set up and working on both sides. I've used sectrace on the /vol/raytest path above to try and catch issues, but get no hits making me think it's NFS on the NetApp rejecting things, not the file system layer... fsecurity shows that everything is wide open and from a CIFS client using the same user as on the Linux box, I can create files on the same path with no issues.

Kinda stumped at this point. Any suggestions?

Ray