NFSv4 Kerberos mount from Linux

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

NFSv4 Kerberos mount from Linux

Felix Winterhalter
Hello everyone,

I've been trying to get a kerberized nfsv4 mount to work from a Debian
Stretch client to a FreeBSD 11.2 server.

My export file looks like:

V4: / -sec=krb5p clients

/testexport -maproot=root -sec=krb5p clients

I am now trying to mount this directory as root first without having to
deal with user keytabs or tickets.

This works fine with -sec=sys and nfsv4.1 and nfsv3 and -sec=krb5p. This
does not however work with nfsv4 and krb5p or any other krb5 flavor.

The only errors we have been able to get is an error by gssd:

gssd_pname_to_uid: failed major=0xd0000 minor=-1765328227

Searching for this error has lead us to an old entry in the mailing list:

https://lists.freebsd.org/pipermail/freebsd-fs/2016-May/023132.html

Which apparently has this problem unresolved with extremely similar
symptoms.

Mounting from the Linux client to a similar Linux server under the same
KDC with nfsv4 krb5p works without any problem.

Also access to the FreeBSD server with sshd and GSSAPI works fine. So
the keytab for the FreeBSD host seems to work fine.

This is extremely frustrating as I have been at this problem for days
now without any real way to even debug the issue.

Any help would be greatly appreciated.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: NFSv4 Kerberos mount from Linux

Rick Macklem
Felix Winterhalter wrote:

>Hello everyone,
>
>I've been trying to get a kerberized nfsv4 mount to work from a Debian
>Stretch client to a FreeBSD 11.2 server.
>
>My export file looks like:
>
>V4: / -sec=krb5p clients
>
>/testexport -maproot=root -sec=krb5p clients
>
Btw, if you only mounting "/testexport", you can specify the "V4:" as
V4: /testexport -sec=krb5p clients
and then the mount on the client uses "/" as the server mountpoint, like
# mount -t nfs -o nfsvers=4 <server>:/ /mnt
(This avoids the server having to search for "testexport" in the "/" directory
 during mounting and might avoid some problems when "/" isn't an exported
 file system. There are "hooks" in the FreeBSD server to make the search work,
 but I've never been 100% certain they will work for Kerberos and/or ZFS.)

Btw, in case the Linux client is falling back on using AUTH_SYS at some point
during the mount, you could try allowing both krb5 and auth_sys by setting
"-sec=sys,krb5,krb5i,krb5p" for both of the above lines. (I'd also suggest you
try krb5 or krb5i until you get it working, since any packet traces are
easier to decode, although once one krb5 variant works, they all should.)

>I am now trying to mount this directory as root first without having to
>deal with user keytabs or tickets.
>
>This works fine with -sec=sys and nfsv4.1 and nfsv3 and -sec=krb5p.
> This does not however work with nfsv4 and krb5p or any other krb5 flavor.
Sorry, I'm not sure what you are saying here. Is it
1 - no version of NFS works for krb5p or
2 - NFSv4.1 works for krb5p, but NFSv4.0 does not or
3 - only nfsv3 works for krb5p

If it #3, that is what I would expect. For NFSv4 (and NFSv4.1, I believe) to work
a host based initiator credential is needed for the client host. The only ways I
know of that you can get this is by
- creating such an entry in your KDC and then putting a keytab entry for it in the client
or
- on the FreeBSD client, doing the mount as a user after that user has done a kinit.
  (There are mount options for this on FreeBSD and it also requires setting the sysctl
    vfs.usermount to 1 so non-root can do mounts.) Since you are using a Linux
    client I have no idea how this might be done on Linux.

I have no idea how the Linux server might allow an NFSv4 mount to work without
Kerberos credentials for the "state maintaining operations" done by root (or the
user doing the mount for the FreeBSD client)?
- Maybe they allow these operations to be done via AUTH_SYS. To me, this would
  sound like a security hole, but I'm not a security guy...

If you have #3, I know the FreeBSD server won't allow what you are trying to do.
If you want to find out what the Linux server does to make it work, you could
capture packets via tcpdump or similar and look at them in wireshark.
(I'd suggest krb5 or krb5i for this, so that the packet data isn't encrypted, since
 it makes the wireshark decoding a lot more useful.)
- If you get such a packet trace, you could email it to me and I can take a look.
  (I am curious how a Linux server might make this work.)

Most of the above only applies if you are experiencing #3, where NFSv3 works
for krb5, but NFSv4 (and 4.1) does not.

>The only errors we have been able to get is an error by gssd:
>
>gssd_pname_to_uid: failed major=0xd0000 minor=-1765328227
I can't remember what this means, but I think it is saying that the principal
name didn't exist in the password database. (Maybe Linux has some "special"
reserved principal name it uses for "state maintaining operations"?)

>Searching for this error has lead us to an old entry in the mailing list:
>
>https://lists.freebsd.org/pipermail/freebsd-fs/2016-May/023132.html
>
>Which apparently has this problem unresolved with extremely similar
>symptoms.
>
>Mounting from the Linux client to a similar Linux server under the same
>KDC with nfsv4 krb5p works without any problem.
>
>Also access to the FreeBSD server with sshd and GSSAPI works fine. So
>the keytab for the FreeBSD host seems to work fine.
>
>This is extremely frustrating as I have been at this problem for days
>now without any real way to even debug the issue.
Here's what I would do:
1 - Try an NFSv3 mount with krb5. (It wasn't obvious if you already have that
     working or not.) If that works, then...
2 - Try a mount from a FreeBSD client as a user, by doing
     # sysctl vfs.usermount=1
     - login as non-root user and kinit
     - as this user, try a mount like
     % mount -t nfs -o nfsv4,sec=krb5 <server>:/ /mnt
3 - If this works, then you probably need a hostbased client credential in the
      keytab on the client.

rick

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: NFSv4 Kerberos mount from Linux

Rick Macklem
I wrote:
[stuff snipped]
Btw, if you only mounting "/testexport", you can specify the "V4:" as
"if you are only.." typo
[more stuff snipped]
>Btw, in case the Linux client is falling back on using AUTH_SYS at some point
>during the mount, you could try allowing both krb5 and auth_sys by setting
>"-sec=sys,krb5,krb5i,krb5p" for both of the above lines. (I'd also suggest you
Oops, the syntax is "-sec=sys:krb5:krb5i:krb5p" (':'s and not ','s)

And if you want to capture packets during a Linux mount attempt, you can
run this on the FreeBSD server:
# tcpdump -s 0 -w out.pcap host <client-host>
However you will want to look at out.pcap in wireshark, since it can decode NFS.

Good luck with it and please let us know if you learn more, rick

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: NFSv4 Kerberos mount from Linux

Felix Winterhalter
In reply to this post by Rick Macklem
On 10/4/18 5:21 PM, Rick Macklem wrote:

> Felix Winterhalter wrote:
>> Hello everyone,
>>
>> I've been trying to get a kerberized nfsv4 mount to work from a Debian
>> Stretch client to a FreeBSD 11.2 server.
>>
>> My export file looks like:
>>
>> V4: / -sec=krb5p clients
>>
>> /testexport -maproot=root -sec=krb5p clients
>>
> Btw, if you only mounting "/testexport", you can specify the "V4:" as
> V4: /testexport -sec=krb5p clients
> and then the mount on the client uses "/" as the server mountpoint, like
> # mount -t nfs -o nfsvers=4 <server>:/ /mnt
> (This avoids the server having to search for "testexport" in the "/" directory
>  during mounting and might avoid some problems when "/" isn't an exported
>  file system. There are "hooks" in the FreeBSD server to make the search work,
>  but I've never been 100% certain they will work for Kerberos and/or ZFS.)
>
> Btw, in case the Linux client is falling back on using AUTH_SYS at some point
> during the mount, you could try allowing both krb5 and auth_sys by setting
> "-sec=sys,krb5,krb5i,krb5p" for both of the above lines. (I'd also suggest you
> try krb5 or krb5i until you get it working, since any packet traces are
> easier to decode, although once one krb5 variant works, they all should.)
True, however I had multiple exports below / set up as tests.

>
>> I am now trying to mount this directory as root first without having to
>> deal with user keytabs or tickets.
>>
>> This works fine with -sec=sys and nfsv4.1 and nfsv3 and -sec=krb5p.
>> This does not however work with nfsv4 and krb5p or any other krb5 flavor.
> Sorry, I'm not sure what you are saying here. Is it
> 1 - no version of NFS works for krb5p or
> 2 - NFSv4.1 works for krb5p, but NFSv4.0 does not or
> 3 - only nfsv3 works for krb5p
[snipped lots of text]

#3 is indeed what was happening. I could mount with krb5p for nfsv3
(which I was not aware was even doable) however nfsv4 would stubbornly
refuse to do any mounting.

I have now after a lot of try and error figured out what I need to do in
order to make it work.

To start with I have kerberos credentials with both host/ and nfs/ on
both client and server. Mounting nfsv4 shares with krb5p from a linux
server has also worked in this context.

I leave you to judge whether what I found out is intended behaviour or
if something weird is going on.

My exports file originally looked something like this:

/nfsTests/ /nfsTests/testexport /nfsTests/otherexport -maproot=root
-sec=krb5p clients

V4: /nfsTests -sec=krb5p clients

Which allowed me to do nfsv3 krb5p mounts but not nfsv4 krb5p mounts.

Changing the exports file to this:

/nfsTests/ /nfsTests/testexport /nfsTests/otherexport -maproot=root
-sec=krb5p clients

V4: /nfsTests -sec=krb5p,krb5i clients

Allows nfsv4 krb5p mounts to work for some reason I do not understand.
Not setting the -sec option on the V4 line apparently defaults to
-sec=sys and doesn't allow any krb5 mounts. I'm not sure that this is a
good default as I wasn't even aware that the -sec option needed to be
set on this line.

I've got packet traces of the nfsv3 krb5 and krb5i mounts and I'll make
traces of the two nfsv4 mount attempts and send them to you if you're
interested. I'm still not sure what exactly is happening here.
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: NFSv4 Kerberos mount from Linux

Rick Macklem
Felix Winterhalter wrote:
On 10/4/18 5:21 PM, Rick Macklem wrote:
[stuff snipped]
>>> I am now trying to mount this directory as root first without having to
>>> deal with user keytabs or tickets.
>>>
>>> This works fine with -sec=sys and nfsv4.1 and nfsv3 and -sec=krb5p.
>>> This does not however work with nfsv4 and krb5p or any other krb5 flavor.
>> Sorry, I'm not sure what you are saying here. Is it
>> 1 - no version of NFS works for krb5p or
>> 2 - NFSv4.1 works for krb5p, but NFSv4.0 does not or
>> 3 - only nfsv3 works for krb5p
[snipped lots of text]
>
>#3 is indeed what was happening. I could mount with krb5p for nfsv3
>(which I was not aware was even doable) however nfsv4 would stubbornly
>refuse to do any mounting.
Yes, RPCSEC_GSS was done by Sun for NFSv3 and it was a good fit, since NFSv3
does not have any server state to maintain. As such, all RPCs are atomic operations
done by users (which for Kerberized mounts must have a TGT in a credential cache).

NFSv4 wasn't really a good fit for the model, because the NFSv4 server maintains
lock state (NFSv4 Opens are a form of lock used by Windows at file open time).
There are "state maintenance" operations that must be done by the user doing
the mount (usually root), where they typically don't have a TGT in a credential
cache.
--> The ugly solution for this is typically a host-based client credential in a keytab
      on the client. (Usually a principal like "root/client-host.domain@REALM" or
      "host/client-host.domain@REALM" or "nfs/client-host.domain@REALM"
       in the default keytab on the client.)

>I have now after a lot of try and error figured out what I need to do in
>order to make it work.
>
>To start with I have kerberos credentials with both host/ and nfs/ on
>both client and server. Mounting nfsv4 shares with krb5p from a linux
>server has also worked in this context.
Yes, I'm assuming that satisfied the host-based client credential as I described
above.

>I leave you to judge whether what I found out is intended behaviour or
>if something weird is going on.
Yes, sounds like intended behaviour, since the client must have a Kerberos
credential to use for the "state maintenance" operations that are not done on
behalf of a user.

>My exports file originally looked something like this:
>
>/nfsTests/ /nfsTests/testexport /nfsTests/otherexport -maproot=root
>-sec=krb5p clients
>
>V4: /nfsTests -sec=krb5p clients
>
>Which allowed me to do nfsv3 krb5p mounts but not nfsv4 krb5p mounts.
>
>Changing the exports file to this:
>
>/nfsTests/ /nfsTests/testexport /nfsTests/otherexport -maproot=root
>-sec=krb5p clients
>
>V4: /nfsTests -sec=krb5p,krb5i clients
This suggests that there is a bug in the client, where it uses krb5i instead of krb5p
at some point in the mounting process. (I have also seen cases where the client
erroneously falls back on using sys at some point in the mounting process.)
(You did mention before you were using the Linux client. If you are using a FreeBSD
 client, I would be interested in looking at this.)

>Allows nfsv4 krb5p mounts to work for some reason I do not understand.
>Not setting the -sec option on the V4 line apparently defaults to
>-sec=sys and doesn't allow any krb5 mounts. I'm not sure that this is a
>good default as I wasn't even aware that the -sec option needed to be
>set on this line.
In FreeBSD, defaults are meant to maintain backwards compatibility. This means that
AUTH_SYS should work by default. Also, AUTH_SYS is what 99.9999% of FreeBSD
NFS mounts still use, from what I've seen.)

>I've got packet traces of the nfsv3 krb5 and krb5i mounts and I'll make
>traces of the two nfsv4 mount attempts and send them to you if you're
>interested. I'm still not sure what exactly is happening here.
The successful one for NFSv4 might be interesting. If you look at it in
wireshark, I suspect you'd find somewhere during the mount that it
did RPCs which were not krb5p and that would show why the addition
of krb5i made it work.

I did suggest you start with -sec=sys:krb5:krb5i:krb5p and, once that works,
remove the security flavours one at a time until the mount doesn't work.
(Then you capture packets for the minimal case that does work and look at
 what security flavours the client is using for all RPCs done during the mount.)

You now know why almost no one uses Kerberized NFSv4 mounts.
Unfortunately, the NFSv4 working group has never gotten around to
a better solution. Discussion of a host based encryption technique using
something like SSL has happened, but no one has gone beyond that.

rick
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: NFSv4 Kerberos mount from Linux

Peter Eriksson
Just a few comments (random brain-dump) in case someone else is having problems with NFS & Kerberos.


We’ve been using NFSv4 with Kerberos from Linux clients here for many years (with Solaris-based NFS servers and MIT Kerberos) and lately using FreeBSD as the NFS server OS (in an Microsoft AD Kerberos environment).

There are a few differences in behaviour (from a Solaris perspective), mainly regarding the “pseudo” NFSv4 filesystem but not something that can’t be handled. In the process of moving to FreeBSD based servers I’ve also been testing lots of different clients in order to make sure everything works as expected. Here are some notes:

General stuff:

1. Have a _kerberos.my.zone.com DNS TXT record containing the Kerberos realm (nice to have)
2. Have a _nfsv4domain.my.zone.com <http://nfsv4domain.my.zone.com/> DNS TXT record containing the NFSv4 “domain” (not all clients use it, but it’s nice to have)


* FreeBSD server (with ZFS filesystems), things below /export is NFS-shared as (for example) server:/staff/user1

1. /etc/exports (we _only_ allow sec=krb<various flavours> for home directories):
V4: /export -sec=krb5:krb5i:krb5p

Or (on a server also serving some (read-only) sec=sys filesystems below /exports)
V4: /export -sec=krb5:krb5i:krb5p:sys


2. /etc/zfs/exports (generated from “set sharenfs=xxx” on the ZFS filesystems)
Home-server:
/export/staff   -sec=krb5:krb5i:krb5p
/export/staff/user1   -sec=krb5:krb5i:krb5p
/export/staff/user2   -sec=krb5:krb5i:krb5p
/export/students         -sec=krb5:krb5i:krb5p
/export/students/user3       -sec=krb5:krb5i:krb5p

Software-server:
/export/db/icons         -sec=sys -ro
/export/old/ifm/solaris-x86     -sec=krb5:krb5i:krb5p:sys -ro
/export/old/ifm/windows-86     -sec=krb5:krb5i:krb5p:sys -ro
/export/software         -sec=krb5:krb5i:krb5p -ro


3. Make sure you have host/FQDN@REALM and nfs/FQDN@REALM Kerberos principals in /etc/krb5.keytab and that FQDN is listed in /etc/hosts like:

  1.2.3.4 foo.bar.com <http://foo.bar.com/> foo

4. rc.conf (we have a lot of users in our AD so we have to use a large number for usermax, replace “liu.se <http://liu.se/>” with your NFSv4 “domain” for nfsuserd_flags)

gssd_enable="YES"
nfs_server_enable="YES"
nfsv4_server_enable="YES"
nfscbd_enable="YES"
mountd_enable="YES"
nfsuserd_enable="YES"
nfsuserd_flags="-manage-gids -domain liu.se -usertimeout 10 -usermax 100000 16"

5. Make sure you use NTPD so the clock is correct.


* All clients (Solaris 10, OmniOS, MacOS 10.12-10.14, FreeBSD 11.0-11.2, CentOS 7, Debian 9, Ubuntu 17-18 tested):

1. Make sure FQDN is in /etc/hosts

2. Make sure you use NTPD so the clock is correct.

3. Have a “host/FQDN@REALM” Kerberos host principal in /etc/krb5.keytab (nfs or root is not needed for NFS-mounting to work)

4. We use a fairly default /etc/krb5.conf, sort of like:

> [libdefaults]
> default_realm = REALM
>         dns_lookup_realm = true
>
>         ticket_lifetime = 24h
>         renew_lifetime = 7d
>         forwardable = true
>
>         default_ccache_name = KEYRING:persistent:%{uid}

KEYRING probably only works on Linux and there are some problems with KEYRING in Debian & Ubuntu since not everything in it supports it due to them using Heimdal instead of MIT (like for smbclient) but it mostly works. Works fine in CentOS 7 though - in general CentOS 7 feels more “enterprise”-ready than Debian & Ubuntu. The old classic FILE-ccaches should work fine though.

For mounting we use the automounter and a "executable map” (perl script) that looks up records in DNS (Hesiod-style) since the built-in Hesiod support in most automounters is a bit.. lacking. Works quite well. You can find the scripts we use here:

        http://www.grebo.net/~peter/nfs <http://www.grebo.net/~peter/nfs>

(The dns-update scripts use data from an SQL database so probably isn’t directly usable to anybody else. We use the same SQL database to populate a locally developed BerkeleyDB-based NSS-database on each FreeBSD server in order to speed things up since AD/LDAP-looks with ~90k users and silly amounts of AD groups takes forever, even with cacheing).

Some Linux-specific stuff:

Packages needed:

  CentOS:
  - nfs-utils
  - libnfsidmap
  - nfs4-acl-tools
  - autofs

  Debian:
  - keyutils
  - nfs-kernel-server # rpc.idmapd needs this due to a bug in Debian

  Ubuntu:
  - keyutils

  Other nice-to have packages:
  - hesiod
  - autofs-hesiod

Some settings to check for:

  /etc/default/nfs-common:
    NEED_IDMAPD=yes
    NEED_GSSD=yes

  /etc/idmapd.conf (replace “liu.se <http://liu.se/>” with your NFSv4 “domain”):
    Domain=liu.se

  /etc/request-key.d/id_resolver.conf (should be there already if using a modern Linux and you’ve added the packages above):
    create id_resolver * * /usr/sbin/nfsidmap %k %d


MacOS:

Basically require the latest - 10.14 (Mojave) - for things to work smoothly. In theory 10.12 & 10.13 should work but there is some bug in them that causes the OS to panic when you try to use NFS & Kerberos. 10.11 and earlier doesn’t support good enough encryption for us…  But with 10.14 you just need to get a Kerberos ticket and then you can mount things just fine.

/etc/nfs.conf should contain (replace “liu.se <http://liu.se/>” with your NFSv4 “domain”):
nfs.client.default_nfs4domain=liu.se <http://liu.se/>



(There are a lot of problems you can run into with Microsofts AD implementation of Kerberos too that we’ve had to be fighting with, but that’s a whole other topic)

- Peter


> On 10 Oct 2018, at 23:47, Rick Macklem <[hidden email]> wrote:
>
> Felix Winterhalter wrote:
> On 10/4/18 5:21 PM, Rick Macklem wrote:
> [stuff snipped]
>>>> I am now trying to mount this directory as root first without having to
>>>> deal with user keytabs or tickets.
>>>>
>>>> This works fine with -sec=sys and nfsv4.1 and nfsv3 and -sec=krb5p.
>>>> This does not however work with nfsv4 and krb5p or any other krb5 flavor.
>>> Sorry, I'm not sure what you are saying here. Is it
>>> 1 - no version of NFS works for krb5p or
>>> 2 - NFSv4.1 works for krb5p, but NFSv4.0 does not or
>>> 3 - only nfsv3 works for krb5p
> [snipped lots of text]
>>
>> #3 is indeed what was happening. I could mount with krb5p for nfsv3
>> (which I was not aware was even doable) however nfsv4 would stubbornly
>> refuse to do any mounting.
> Yes, RPCSEC_GSS was done by Sun for NFSv3 and it was a good fit, since NFSv3
> does not have any server state to maintain. As such, all RPCs are atomic operations
> done by users (which for Kerberized mounts must have a TGT in a credential cache).
>
> NFSv4 wasn't really a good fit for the model, because the NFSv4 server maintains
> lock state (NFSv4 Opens are a form of lock used by Windows at file open time).
> There are "state maintenance" operations that must be done by the user doing
> the mount (usually root), where they typically don't have a TGT in a credential
> cache.
> --> The ugly solution for this is typically a host-based client credential in a keytab
>      on the client. (Usually a principal like "root/client-host.domain@REALM" or
>      "host/client-host.domain@REALM" or "nfs/client-host.domain@REALM"
>       in the default keytab on the client.)
>
>> I have now after a lot of try and error figured out what I need to do in
>> order to make it work.
>>
>> To start with I have kerberos credentials with both host/ and nfs/ on
>> both client and server. Mounting nfsv4 shares with krb5p from a linux
>> server has also worked in this context.
> Yes, I'm assuming that satisfied the host-based client credential as I described
> above.
>
>> I leave you to judge whether what I found out is intended behaviour or
>> if something weird is going on.
> Yes, sounds like intended behaviour, since the client must have a Kerberos
> credential to use for the "state maintenance" operations that are not done on
> behalf of a user.
>
>> My exports file originally looked something like this:
>>
>> /nfsTests/ /nfsTests/testexport /nfsTests/otherexport -maproot=root
>> -sec=krb5p clients
>>
>> V4: /nfsTests -sec=krb5p clients
>>
>> Which allowed me to do nfsv3 krb5p mounts but not nfsv4 krb5p mounts.
>>
>> Changing the exports file to this:
>>
>> /nfsTests/ /nfsTests/testexport /nfsTests/otherexport -maproot=root
>> -sec=krb5p clients
>>
>> V4: /nfsTests -sec=krb5p,krb5i clients
> This suggests that there is a bug in the client, where it uses krb5i instead of krb5p
> at some point in the mounting process. (I have also seen cases where the client
> erroneously falls back on using sys at some point in the mounting process.)
> (You did mention before you were using the Linux client. If you are using a FreeBSD
> client, I would be interested in looking at this.)
>
>> Allows nfsv4 krb5p mounts to work for some reason I do not understand.
>> Not setting the -sec option on the V4 line apparently defaults to
>> -sec=sys and doesn't allow any krb5 mounts. I'm not sure that this is a
>> good default as I wasn't even aware that the -sec option needed to be
>> set on this line.
> In FreeBSD, defaults are meant to maintain backwards compatibility. This means that
> AUTH_SYS should work by default. Also, AUTH_SYS is what 99.9999% of FreeBSD
> NFS mounts still use, from what I've seen.)
>
>> I've got packet traces of the nfsv3 krb5 and krb5i mounts and I'll make
>> traces of the two nfsv4 mount attempts and send them to you if you're
>> interested. I'm still not sure what exactly is happening here.
> The successful one for NFSv4 might be interesting. If you look at it in
> wireshark, I suspect you'd find somewhere during the mount that it
> did RPCs which were not krb5p and that would show why the addition
> of krb5i made it work.
>
> I did suggest you start with -sec=sys:krb5:krb5i:krb5p and, once that works,
> remove the security flavours one at a time until the mount doesn't work.
> (Then you capture packets for the minimal case that does work and look at
> what security flavours the client is using for all RPCs done during the mount.)
>
> You now know why almost no one uses Kerberized NFSv4 mounts.
> Unfortunately, the NFSv4 working group has never gotten around to
> a better solution. Discussion of a host based encryption technique using
> something like SSL has happened, but no one has gone beyond that.
>
> rick
> _______________________________________________
> [hidden email] <mailto:[hidden email]> mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs <https://lists.freebsd.org/mailman/listinfo/freebsd-fs>
> To unsubscribe, send any mail to "[hidden email] <mailto:[hidden email]>"

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: NFSv4 Kerberos mount from Linux

freebsd-fs mailing list
On Thu, 2018-10-11 at 10:36 +0200, Peter Eriksson wrote:
> Just a few comments (random brain-dump) in case someone else is
> having problems with NFS & Kerberos.

"Just a few comments", hot dog, that was a great write-up! It would be
great, I think, to have this on the FreeBSD wiki or in the Handbook
even, because I agree not many people use Kerberized NFS since it´s
such a hassle and not that much info about it.

/K

>
>
> We’ve been using NFSv4 with Kerberos from Linux clients here for many
> years (with Solaris-based NFS servers and MIT Kerberos) and lately
> using FreeBSD as the NFS server OS (in an Microsoft AD Kerberos
> environment).
>
> There are a few differences in behaviour (from a Solaris
> perspective), mainly regarding the “pseudo” NFSv4 filesystem but not
> something that can’t be handled. In the process of moving to FreeBSD
> based servers I’ve also been testing lots of different clients in
> order to make sure everything works as expected. Here are some notes:
>
> General stuff:
>
> 1. Have a _kerberos.my.zone.com DNS TXT record containing the
> Kerberos realm (nice to have)
> 2. Have a _nfsv4domain.my.zone.com <http://nfsv4domain.my.zone.com/>
> DNS TXT record containing the NFSv4 “domain” (not all clients use it,
> but it’s nice to have)
>
>
> * FreeBSD server (with ZFS filesystems), things below /export is NFS-
> shared as (for example) server:/staff/user1
>
> 1. /etc/exports (we _only_ allow sec=krb<various flavours> for home
> directories):
> V4: /export -sec=krb5:krb5i:krb5p
>
> Or (on a server also serving some (read-only) sec=sys filesystems
> below /exports)
> V4: /export -sec=krb5:krb5i:krb5p:sys
>
>
> 2. /etc/zfs/exports (generated from “set sharenfs=xxx” on the ZFS
> filesystems)
> Home-server:
> /export/staff   -sec=krb5:krb5i:krb5p
> /export/staff/user1   -sec=krb5:krb5i:krb5p
> /export/staff/user2   -sec=krb5:krb5i:krb5p
> /export/students         -sec=krb5:krb5i:krb5p
> /export/students/user3       -sec=krb5:krb5i:krb5p
>
> Software-server:
> /export/db/icons         -sec=sys -ro
> /export/old/ifm/solaris-x86     -sec=krb5:krb5i:krb5p:sys -ro
> /export/old/ifm/windows-86     -sec=krb5:krb5i:krb5p:sys -ro
> /export/software         -sec=krb5:krb5i:krb5p -ro
>
>
> 3. Make sure you have host/FQDN@REALM and nfs/FQDN@REALM Kerberos
> principals in /etc/krb5.keytab and that FQDN is listed in /etc/hosts
> like:
>
>   1.2.3.4 foo.bar.com <http://foo.bar.com/> foo
>
> 4. rc.conf (we have a lot of users in our AD so we have to use a
> large number for usermax, replace “liu.se <http://liu.se/>” with your
> NFSv4 “domain” for nfsuserd_flags)
>
> gssd_enable="YES"
> nfs_server_enable="YES"
> nfsv4_server_enable="YES"
> nfscbd_enable="YES"
> mountd_enable="YES"
> nfsuserd_enable="YES"
> nfsuserd_flags="-manage-gids -domain liu.se -usertimeout 10 -usermax
> 100000 16"
>
> 5. Make sure you use NTPD so the clock is correct.
>
>
> * All clients (Solaris 10, OmniOS, MacOS 10.12-10.14, FreeBSD 11.0-
> 11.2, CentOS 7, Debian 9, Ubuntu 17-18 tested):
>
> 1. Make sure FQDN is in /etc/hosts
>
> 2. Make sure you use NTPD so the clock is correct.
>
> 3. Have a “host/FQDN@REALM” Kerberos host principal in
> /etc/krb5.keytab (nfs or root is not needed for NFS-mounting to work)
>
> 4. We use a fairly default /etc/krb5.conf, sort of like:
>
> > [libdefaults]
> > default_realm = REALM
> >         dns_lookup_realm = true
> >
> >         ticket_lifetime = 24h
> >         renew_lifetime = 7d
> >         forwardable = true
> >
> >         default_ccache_name = KEYRING:persistent:%{uid}
>
> KEYRING probably only works on Linux and there are some problems with
> KEYRING in Debian & Ubuntu since not everything in it supports it due
> to them using Heimdal instead of MIT (like for smbclient) but it
> mostly works. Works fine in CentOS 7 though - in general CentOS 7
> feels more “enterprise”-ready than Debian & Ubuntu. The old classic
> FILE-ccaches should work fine though.
>
> For mounting we use the automounter and a "executable map” (perl
> script) that looks up records in DNS (Hesiod-style) since the built-
> in Hesiod support in most automounters is a bit.. lacking. Works
> quite well. You can find the scripts we use here:
>
> http://www.grebo.net/~peter/nfs <
> http://www.grebo.net/~peter/nfs>
>
> (The dns-update scripts use data from an SQL database so probably
> isn’t directly usable to anybody else. We use the same SQL database
> to populate a locally developed BerkeleyDB-based NSS-database on each
> FreeBSD server in order to speed things up since AD/LDAP-looks with
> ~90k users and silly amounts of AD groups takes forever, even with
> cacheing).
>
> Some Linux-specific stuff:
>
> Packages needed:
>
>   CentOS:
>   - nfs-utils
>   - libnfsidmap
>   - nfs4-acl-tools
>   - autofs
>
>   Debian:
>   - keyutils
>   - nfs-kernel-server # rpc.idmapd needs this due to a bug in Debian
>
>   Ubuntu:
>   - keyutils
>
>   Other nice-to have packages:
>   - hesiod
>   - autofs-hesiod
>
> Some settings to check for:
>
>   /etc/default/nfs-common:
>     NEED_IDMAPD=yes
>     NEED_GSSD=yes
>
>   /etc/idmapd.conf (replace “liu.se <http://liu.se/>” with your NFSv4
> “domain”):
>     Domain=liu.se
>
>   /etc/request-key.d/id_resolver.conf (should be there already if
> using a modern Linux and you’ve added the packages above):
>     create id_resolver * * /usr/sbin/nfsidmap %k
> %d
>
>
> MacOS:
>
> Basically require the latest - 10.14 (Mojave) - for things to work
> smoothly. In theory 10.12 & 10.13 should work but there is some bug
> in them that causes the OS to panic when you try to use NFS &
> Kerberos. 10.11 and earlier doesn’t support good enough encryption
> for us…  But with 10.14 you just need to get a Kerberos ticket and
> then you can mount things just fine.
>
> /etc/nfs.conf should contain (replace “liu.se <http://liu.se/>” with
> your NFSv4 “domain”):
> nfs.client.default_nfs4domain=liu.se <http://liu.se/>
>
>
>
> (There are a lot of problems you can run into with Microsofts AD
> implementation of Kerberos too that we’ve had to be fighting with,
> but that’s a whole other topic)
>
> - Peter
>
>
> > On 10 Oct 2018, at 23:47, Rick Macklem <[hidden email]>
> > wrote:
> >
> > Felix Winterhalter wrote:
> > On 10/4/18 5:21 PM, Rick Macklem wrote:
> > [stuff snipped]
> > > > > I am now trying to mount this directory as root first without
> > > > > having to
> > > > > deal with user keytabs or tickets.
> > > > >
> > > > > This works fine with -sec=sys and nfsv4.1 and nfsv3 and
> > > > > -sec=krb5p.
> > > > > This does not however work with nfsv4 and krb5p or any other
> > > > > krb5 flavor.
> > > >
> > > > Sorry, I'm not sure what you are saying here. Is it
> > > > 1 - no version of NFS works for krb5p or
> > > > 2 - NFSv4.1 works for krb5p, but NFSv4.0 does not or
> > > > 3 - only nfsv3 works for krb5p
> >
> > [snipped lots of text]
> > >
> > > #3 is indeed what was happening. I could mount with krb5p for
> > > nfsv3
> > > (which I was not aware was even doable) however nfsv4 would
> > > stubbornly
> > > refuse to do any mounting.
> >
> > Yes, RPCSEC_GSS was done by Sun for NFSv3 and it was a good fit,
> > since NFSv3
> > does not have any server state to maintain. As such, all RPCs are
> > atomic operations
> > done by users (which for Kerberized mounts must have a TGT in a
> > credential cache).
> >
> > NFSv4 wasn't really a good fit for the model, because the NFSv4
> > server maintains
> > lock state (NFSv4 Opens are a form of lock used by Windows at file
> > open time).
> > There are "state maintenance" operations that must be done by the
> > user doing
> > the mount (usually root), where they typically don't have a TGT in
> > a credential
> > cache.
> > --> The ugly solution for this is typically a host-based client
> > credential in a keytab
> >      on the client. (Usually a principal like "
> > root/client-host.domain@REALM" or
> >      "host/client-host.domain@REALM" or "
> > nfs/client-host.domain@REALM"
> >       in the default keytab on the client.)
> >
> > > I have now after a lot of try and error figured out what I need
> > > to do in
> > > order to make it work.
> > >
> > > To start with I have kerberos credentials with both host/ and
> > > nfs/ on
> > > both client and server. Mounting nfsv4 shares with krb5p from a
> > > linux
> > > server has also worked in this context.
> >
> > Yes, I'm assuming that satisfied the host-based client credential
> > as I described
> > above.
> >
> > > I leave you to judge whether what I found out is intended
> > > behaviour or
> > > if something weird is going on.
> >
> > Yes, sounds like intended behaviour, since the client must have a
> > Kerberos
> > credential to use for the "state maintenance" operations that are
> > not done on
> > behalf of a user.
> >
> > > My exports file originally looked something like this:
> > >
> > > /nfsTests/ /nfsTests/testexport /nfsTests/otherexport
> > > -maproot=root
> > > -sec=krb5p clients
> > >
> > > V4: /nfsTests -sec=krb5p clients
> > >
> > > Which allowed me to do nfsv3 krb5p mounts but not nfsv4 krb5p
> > > mounts.
> > >
> > > Changing the exports file to this:
> > >
> > > /nfsTests/ /nfsTests/testexport /nfsTests/otherexport
> > > -maproot=root
> > > -sec=krb5p clients
> > >
> > > V4: /nfsTests -sec=krb5p,krb5i clients
> >
> > This suggests that there is a bug in the client, where it uses
> > krb5i instead of krb5p
> > at some point in the mounting process. (I have also seen cases
> > where the client
> > erroneously falls back on using sys at some point in the mounting
> > process.)
> > (You did mention before you were using the Linux client. If you are
> > using a FreeBSD
> > client, I would be interested in looking at this.)
> >
> > > Allows nfsv4 krb5p mounts to work for some reason I do not
> > > understand.
> > > Not setting the -sec option on the V4 line apparently defaults to
> > > -sec=sys and doesn't allow any krb5 mounts. I'm not sure that
> > > this is a
> > > good default as I wasn't even aware that the -sec option needed
> > > to be
> > > set on this line.
> >
> > In FreeBSD, defaults are meant to maintain backwards compatibility.
> > This means that
> > AUTH_SYS should work by default. Also, AUTH_SYS is what 99.9999% of
> > FreeBSD
> > NFS mounts still use, from what I've seen.)
> >
> > > I've got packet traces of the nfsv3 krb5 and krb5i mounts and
> > > I'll make
> > > traces of the two nfsv4 mount attempts and send them to you if
> > > you're
> > > interested. I'm still not sure what exactly is happening here.
> >
> > The successful one for NFSv4 might be interesting. If you look at
> > it in
> > wireshark, I suspect you'd find somewhere during the mount that it
> > did RPCs which were not krb5p and that would show why the addition
> > of krb5i made it work.
> >
> > I did suggest you start with -sec=sys:krb5:krb5i:krb5p and, once
> > that works,
> > remove the security flavours one at a time until the mount doesn't
> > work.
> > (Then you capture packets for the minimal case that does work and
> > look at
> > what security flavours the client is using for all RPCs done during
> > the mount.)
> >
> > You now know why almost no one uses Kerberized NFSv4 mounts.
> > Unfortunately, the NFSv4 working group has never gotten around to
> > a better solution. Discussion of a host based encryption technique
> > using
> > something like SSL has happened, but no one has gone beyond that.
> >
> > rick
> > _______________________________________________
> > [hidden email] <mailto:[hidden email]> mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs <
> > https://lists.freebsd.org/mailman/listinfo/freebsd-fs>
> > To unsubscribe, send any mail to "
> > [hidden email] <mailto:
> > [hidden email]>"
>
> _______________________________________________
> [hidden email] mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "[hidden email]"

signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: NFSv4 Kerberos mount from Linux

Rick Macklem
In reply to this post by Peter Eriksson
From: Peter Eriksson wrote:
>Just a few comments (random brain-dump) in case someone else is having problems >with NFS & Kerberos.
>
>
>We’ve been using NFSv4 with Kerberos from Linux clients here for many years (with >Solaris-based NFS servers and MIT Kerberos) and lately using FreeBSD as the NFS >server OS (in an Microsoft AD Kerberos environment).
I agree with the other post that this is a very useful document and it would be
nice to keep it somewhere where it is easier to find than in the mailing list
archive.
I don't know where that could be, so hopefully someone else in FreeBSD-land
can suggest a good place?

The one area you don't discuss (and maybe isn't really a problem?) is what
ticket encryption type(s) you use.
Kerberized NFS still uses DES (someday this may change, but I think that requires
implementation of RPCSEC_GSS V3), so it needs an 8byte session key.
(I have never seen a documented way to convert a session key of greater than
 8bytes into an 8byte session key for RPCSEC_GSS to use. As such, I have no idea
 what happens if you choose a ticket encryption type that results in a greater
 than 8byte key.)

Here's a couple of minor comments:
[lots of good stuff snipped]
>4. rc.conf (we have a lot of users in our AD so we have to use a large number for >usermax, replace “liu.se<http://liu.se>” with your NFSv4 “domain” for >nfsuserd_flags)
>
>gssd_enable="YES"
>nfs_server_enable="YES"
>nfsv4_server_enable="YES"
>nfscbd_enable="YES"
Since the nfscbd is only used by the client (and only when delegations or pNFS is
 enabled in the server), I believe this is harmless, but unnecessary for the server.
>mountd_enable="YES"
>nfsuserd_enable="YES"
>nfsuserd_flags="-manage-gids -domain liu.se<http://liu.se> -usertimeout 10 -usermax 100000 16"
>
Btw, if you have many clients doing NFSv4.0 mounts, tuning of your DRC is advised.
(The defaults are for extremely small NFS servers.) NFSv4.1 mounts don't use the
DRC, so if that is what your clients are doing, this isn't necessary.
It's a little off topic, but for NFSv4.0 (and/or NFSv3) mounts and a reasonable
sized NFS server, reasonable settings in /etc/sysctl.conf might be:
vfs.nfsd.tcpcachetimeo=600
vfs.nfsd.tcphighwater=100000
vfs.nfsd.v4statelimit=10000000
vfs.nfsd.clienthashsize=10000
vfs.nfsd.statehashsize=10000
- If most/all of your mounts are NFSv4.1, then instead of the above, you might want:
vfs.nfsd.sessionhashsize=10000

>5. Make sure you use NTPD so the clock is correct.
>
>
>* All clients (Solaris 10, OmniOS, MacOS 10.12-10.14, FreeBSD 11.0-11.2, CentOS 7, >Debian 9, Ubuntu 17-18 tested):
>
>1. Make sure FQDN is in /etc/hosts
>
>2. Make sure you use NTPD so the clock is correct.
>
>3. Have a “host/FQDN@REALM” Kerberos host principal in /etc/krb5.keytab (nfs or >root is not needed for NFS-mounting to work)
I'm guessing that you use the "gssname=host" mount option for your FreeBSD
clients? (The name after "gssname=" is whatever name you have before
"/FQDN@REALM" for this principal name.)
This is what I referred to as the host-based initiator credential.

Thanks for posting this, rick

4. We use a fairly default /etc/krb5.conf, sort of like:

[libdefaults]
default_realm = REALM
        dns_lookup_realm = true

        ticket_lifetime = 24h
        renew_lifetime = 7d
        forwardable = true

        default_ccache_name = KEYRING:persistent:%{uid}

KEYRING probably only works on Linux and there are some problems with KEYRING in Debian & Ubuntu since not everything in it supports it due to them using Heimdal instead of MIT (like for smbclient) but it mostly works. Works fine in CentOS 7 though - in general CentOS 7 feels more “enterprise”-ready than Debian & Ubuntu. The old classic FILE-ccaches should work fine though.

For mounting we use the automounter and a "executable map” (perl script) that looks up records in DNS (Hesiod-style) since the built-in Hesiod support in most automounters is a bit.. lacking. Works quite well. You can find the scripts we use here:

http://www.grebo.net/~peter/nfs

(The dns-update scripts use data from an SQL database so probably isn’t directly usable to anybody else. We use the same SQL database to populate a locally developed BerkeleyDB-based NSS-database on each FreeBSD server in order to speed things up since AD/LDAP-looks with ~90k users and silly amounts of AD groups takes forever, even with cacheing).

Some Linux-specific stuff:

Packages needed:

  CentOS:
  - nfs-utils
  - libnfsidmap
  - nfs4-acl-tools
  - autofs

  Debian:
  - keyutils
  - nfs-kernel-server # rpc.idmapd needs this due to a bug in Debian

  Ubuntu:
  - keyutils

  Other nice-to have packages:
  - hesiod
  - autofs-hesiod

Some settings to check for:

  /etc/default/nfs-common:
    NEED_IDMAPD=yes
    NEED_GSSD=yes

  /etc/idmapd.conf (replace “liu.se<http://liu.se>” with your NFSv4 “domain”):
    Domain=liu.se<http://liu.se>

  /etc/request-key.d/id_resolver.conf (should be there already if using a modern Linux and you’ve added the packages above):
    create id_resolver * * /usr/sbin/nfsidmap %k %d


MacOS:

Basically require the latest - 10.14 (Mojave) - for things to work smoothly. In theory 10.12 & 10.13 should work but there is some bug in them that causes the OS to panic when you try to use NFS & Kerberos. 10.11 and earlier doesn’t support good enough encryption for us…  But with 10.14 you just need to get a Kerberos ticket and then you can mount things just fine.

/etc/nfs.conf should contain (replace “liu.se<http://liu.se>” with your NFSv4 “domain”):
nfs.client.default_nfs4domain=liu.se<http://liu.se>



(There are a lot of problems you can run into with Microsofts AD implementation of Kerberos too that we’ve had to be fighting with, but that’s a whole other topic)

- Peter


On 10 Oct 2018, at 23:47, Rick Macklem <[hidden email]<mailto:[hidden email]>> wrote:

Felix Winterhalter wrote:
On 10/4/18 5:21 PM, Rick Macklem wrote:
[stuff snipped]
I am now trying to mount this directory as root first without having to
deal with user keytabs or tickets.

This works fine with -sec=sys and nfsv4.1 and nfsv3 and -sec=krb5p.
This does not however work with nfsv4 and krb5p or any other krb5 flavor.
Sorry, I'm not sure what you are saying here. Is it
1 - no version of NFS works for krb5p or
2 - NFSv4.1 works for krb5p, but NFSv4.0 does not or
3 - only nfsv3 works for krb5p
[snipped lots of text]

#3 is indeed what was happening. I could mount with krb5p for nfsv3
(which I was not aware was even doable) however nfsv4 would stubbornly
refuse to do any mounting.
Yes, RPCSEC_GSS was done by Sun for NFSv3 and it was a good fit, since NFSv3
does not have any server state to maintain. As such, all RPCs are atomic operations
done by users (which for Kerberized mounts must have a TGT in a credential cache).

NFSv4 wasn't really a good fit for the model, because the NFSv4 server maintains
lock state (NFSv4 Opens are a form of lock used by Windows at file open time).
There are "state maintenance" operations that must be done by the user doing
the mount (usually root), where they typically don't have a TGT in a credential
cache.
--> The ugly solution for this is typically a host-based client credential in a keytab
     on the client. (Usually a principal like "root/client-host.domain@REALM" or
     "host/client-host.domain@REALM" or "nfs/client-host.domain@REALM"
      in the default keytab on the client.)

I have now after a lot of try and error figured out what I need to do in
order to make it work.

To start with I have kerberos credentials with both host/ and nfs/ on
both client and server. Mounting nfsv4 shares with krb5p from a linux
server has also worked in this context.
Yes, I'm assuming that satisfied the host-based client credential as I described
above.

I leave you to judge whether what I found out is intended behaviour or
if something weird is going on.
Yes, sounds like intended behaviour, since the client must have a Kerberos
credential to use for the "state maintenance" operations that are not done on
behalf of a user.

My exports file originally looked something like this:

/nfsTests/ /nfsTests/testexport /nfsTests/otherexport -maproot=root
-sec=krb5p clients

V4: /nfsTests -sec=krb5p clients

Which allowed me to do nfsv3 krb5p mounts but not nfsv4 krb5p mounts.

Changing the exports file to this:

/nfsTests/ /nfsTests/testexport /nfsTests/otherexport -maproot=root
-sec=krb5p clients

V4: /nfsTests -sec=krb5p,krb5i clients
This suggests that there is a bug in the client, where it uses krb5i instead of krb5p
at some point in the mounting process. (I have also seen cases where the client
erroneously falls back on using sys at some point in the mounting process.)
(You did mention before you were using the Linux client. If you are using a FreeBSD
client, I would be interested in looking at this.)

Allows nfsv4 krb5p mounts to work for some reason I do not understand.
Not setting the -sec option on the V4 line apparently defaults to
-sec=sys and doesn't allow any krb5 mounts. I'm not sure that this is a
good default as I wasn't even aware that the -sec option needed to be
set on this line.
In FreeBSD, defaults are meant to maintain backwards compatibility. This means that
AUTH_SYS should work by default. Also, AUTH_SYS is what 99.9999% of FreeBSD
NFS mounts still use, from what I've seen.)

I've got packet traces of the nfsv3 krb5 and krb5i mounts and I'll make
traces of the two nfsv4 mount attempts and send them to you if you're
interested. I'm still not sure what exactly is happening here.
The successful one for NFSv4 might be interesting. If you look at it in
wireshark, I suspect you'd find somewhere during the mount that it
did RPCs which were not krb5p and that would show why the addition
of krb5i made it work.

I did suggest you start with -sec=sys:krb5:krb5i:krb5p and, once that works,
remove the security flavours one at a time until the mount doesn't work.
(Then you capture packets for the minimal case that does work and look at
what security flavours the client is using for all RPCs done during the mount.)

You now know why almost no one uses Kerberized NFSv4 mounts.
Unfortunately, the NFSv4 working group has never gotten around to
a better solution. Discussion of a host based encryption technique using
something like SSL has happened, but no one has gone beyond that.

rick
_______________________________________________
[hidden email]<mailto:[hidden email]> mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "[hidden email]<mailto:[hidden email]>"

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: NFSv4 Kerberos mount from Linux

Benjamin Kaduk-2
On Fri, Oct 12, 2018 at 12:37:55AM +0000, Rick Macklem wrote:

> From: Peter Eriksson wrote:
> >Just a few comments (random brain-dump) in case someone else is having problems >with NFS & Kerberos.
> >
> >
> >We’ve been using NFSv4 with Kerberos from Linux clients here for many years (with >Solaris-based NFS servers and MIT Kerberos) and lately using FreeBSD as the NFS >server OS (in an Microsoft AD Kerberos environment).
> I agree with the other post that this is a very useful document and it would be
> nice to keep it somewhere where it is easier to find than in the mailing list
> archive.
> I don't know where that could be, so hopefully someone else in FreeBSD-land
> can suggest a good place?
>
> The one area you don't discuss (and maybe isn't really a problem?) is what
> ticket encryption type(s) you use.
> Kerberized NFS still uses DES (someday this may change, but I think that requires
> implementation of RPCSEC_GSS V3), so it needs an 8byte session key.

This isn't true anymore; you can use stronger session keys just fine.
(See also RFC 6649 -- don't use single-DES!)

> (I have never seen a documented way to convert a session key of greater than
>  8bytes into an 8byte session key for RPCSEC_GSS to use. As such, I have no idea
>  what happens if you choose a ticket encryption type that results in a greater
>  than 8byte key.)

We did have this problem in the AFS world, and are currently using this
hack: https://tools.ietf.org/html/draft-kaduk-afs3-rxkad-k5-kdf-00

> Here's a couple of minor comments:
> [lots of good stuff snipped]
> >4. rc.conf (we have a lot of users in our AD so we have to use a large number for >usermax, replace “liu.se<http://liu.se>” with your NFSv4 “domain” for >nfsuserd_flags)
> >
> >gssd_enable="YES"
> >nfs_server_enable="YES"
> >nfsv4_server_enable="YES"
> >nfscbd_enable="YES"
> Since the nfscbd is only used by the client (and only when delegations or pNFS is
>  enabled in the server), I believe this is harmless, but unnecessary for the server.
> >mountd_enable="YES"
> >nfsuserd_enable="YES"
> >nfsuserd_flags="-manage-gids -domain liu.se<http://liu.se> -usertimeout 10 -usermax 100000 16"
> >
> Btw, if you have many clients doing NFSv4.0 mounts, tuning of your DRC is advised.
> (The defaults are for extremely small NFS servers.) NFSv4.1 mounts don't use the
> DRC, so if that is what your clients are doing, this isn't necessary.
> It's a little off topic, but for NFSv4.0 (and/or NFSv3) mounts and a reasonable
> sized NFS server, reasonable settings in /etc/sysctl.conf might be:
> vfs.nfsd.tcpcachetimeo=600
> vfs.nfsd.tcphighwater=100000
> vfs.nfsd.v4statelimit=10000000
> vfs.nfsd.clienthashsize=10000
> vfs.nfsd.statehashsize=10000
> - If most/all of your mounts are NFSv4.1, then instead of the above, you might want:
> vfs.nfsd.sessionhashsize=10000
>
> >5. Make sure you use NTPD so the clock is correct.
> >
> >
> >* All clients (Solaris 10, OmniOS, MacOS 10.12-10.14, FreeBSD 11.0-11.2, CentOS 7, >Debian 9, Ubuntu 17-18 tested):
> >
> >1. Make sure FQDN is in /etc/hosts
> >
> >2. Make sure you use NTPD so the clock is correct.
> >
> >3. Have a “host/FQDN@REALM” Kerberos host principal in /etc/krb5.keytab (nfs or >root is not needed for NFS-mounting to work)
> I'm guessing that you use the "gssname=host" mount option for your FreeBSD
> clients? (The name after "gssname=" is whatever name you have before
> "/FQDN@REALM" for this principal name.)
> This is what I referred to as the host-based initiator credential.

I'm told that the Linux client will automatically attempt to use any
of root/, host/, or nfs/ credentials as an NFS client.  This is incredibly
frustrating to me (not least because of the amount of confusion it
perpetuates about how NFS with Kerberos is hard!), and is decidedly not
Kerberos best practices.  I would probably have specified a new host-based
service name, like nfscl/, though I guess there is some argument that root/
is reasonable.

> Thanks for posting this, rick

And thank you for the extra tips!

-Ben
_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: NFSv4 Kerberos mount from Linux

Rick Macklem
Benjamin Kaduk wrote:
>I wrote:
>>
>> The one area you don't discuss (and maybe isn't really a problem?) is what
>> ticket encryption type(s) you use.
>> Kerberized NFS still uses DES (someday this may change, but I think that requires
>> implementation of RPCSEC_GSS V3), so it needs an 8byte session key.
>
>This isn't true anymore; you can use stronger session keys just fine.
>(See also RFC 6649 -- don't use single-DES!)
I haven't read RFC6649, but from looking at the kgssapi code in FreeBSD's
head/current, it appears that newer encryption types are used for wrap/unwrap
(krb5p).
From what I can see, the following appear to be supported:
DES, DES3, AES128, AES256, Arcfour, Arcfour_56
(I'll have to look at RFC6649 someday, because I've never seen an RFC specifying
 anything but DES for RPCSEC_GSS.)

I won't even try to guess whether all of the above work for all implementations,
but it appears that it uses whatever the session key is (krb5_key_state?).

Peter, do you happen to know what encryption type(s) you have been using?

>> (I have never seen a documented way to convert a session key of greater than
>>  8bytes into an 8byte session key for RPCSEC_GSS to use. As such, I have no idea
>>  what happens if you choose a ticket encryption type that results in a greater
>>  than 8byte key.)
Ignore this. I just wasn't correct.

rick
[good stuff snipped]

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "[hidden email]"
Reply | Threaded
Open this post in threaded view
|

Re: NFSv4 Kerberos mount from Linux

Rick Macklem
I wrote:
>Benjamin Kaduk wrote:
>>I wrote:
>>>
>>> The one area you don't discuss (and maybe isn't really a problem?) is what
>>> ticket encryption type(s) you use.
>>> Kerberized NFS still uses DES (someday this may change, but I think that requires
>>> implementation of RPCSEC_GSS V3), so it needs an 8byte session key.
In case my previous post wasn't clear, this appears to have already changed and
did not require implementation of RPCSEC_V2 or RPCSEC_GSS_v3.

>>
>>This isn't true anymore; you can use stronger session keys just fine.
>>(See also RFC 6649 -- don't use single-DES!)
>I haven't read RFC6649, but from looking at the kgssapi code in FreeBSD's
>head/current, it appears that newer encryption types are used for wrap/unwrap
>(krb5p).
>From what I can see, the following appear to be supported:
>DES, DES3, AES128, AES256, Arcfour, Arcfour_56
>(I'll have to look at RFC6649 someday, because I've never seen an RFC specifying
> anything but DES for RPCSEC_GSS.)
>I won't even try to guess whether all of the above work for all implementations,
>but it appears that it uses whatever the session key is (krb5_key_state?).
I just received a reply to a query on the [hidden email] mailing list and the set
of encryption types supported by Linux is the same as above except they do
no support Arcfour_56.
However, they are planning on deleting support for all encryption types
except for the AES ones.
As such, it sounds like you may need to configure Kerberos to only use those
to ensure interoperability in the future.

Hope this is useful and hasn't added to the confusion, rick

_______________________________________________
[hidden email] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-fs
To unsubscribe, send any mail to "[hidden email]"