blob: bc388f1006e01b08ffdb3961f58a5455ad64ac09 (
plain) (
blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
|
% Kerberized NFS: access denied by server while mounting
# Introduction
Protecting a NFS share with Kerberos is not very easy to do but definitely
doable with a good setup manual. A very helpful website is
https://wiki.ubuntuusers.de although some of the pages
have since been archived.
# Setup
The hostnames are different in my actual setup and will certainly be
different for you.
There are two machines involved. The first one is the server.
It's running a krb5 KDC and admin server as well as a NFS server.
The NFS export is configured to allow any source address
but requires krb5i or krb5p security.
The client computer is running a krb5 client and a NFS client with
the necessary rpc daemons.
#### Kerberos principals
* admin/admin, has full access to kadmin
* himbeerserverde, a regular user
* host/srv.himbeerserver.de, server host key
* host/clt.himbeerserver.de, client host key
* nfs/srv.himbeerserver.de, server NFS key
* nfs/clt.himbeerserver.de, client NFS key
The users are synced across other clients and the server using LDAP.
The clients use SSSD to cache credentials. This way they can operate
without a permanent connection to the LDAP server. They also keep working
in case of a server failure.
The server uses local auth for the actual accounts. The other accounts
are not intended to be logged into. A LDAP failure will only result
in a broken NFS.
I'm aware this isn't the best solution. I'm probably going to come up
with a better one in about half a decade.
# The Error
This is the command I use to mount the NFS share:
```sh
sudo mount -t nfs4 -o sec=krb5i,async,soft srv.himbeerserver.de:/media/ssd /mnt/himbeerserverde/nfs
```
This suddenly resulted in the above error. I couldn't really figure out
what was going on. This has happened several times and could sometimes be
fixed by rebooting both machines. Unfortunately rebooting didn't help
most of the time.
# Debugging
The logs are not very helpful for debugging this error.
Adding `-vvvv` to the mount command outputs more but still only shows
that permission was denied, not why it's happening.
Looking at the traffic with wireshark I didn't see any Kerberos packets.
The syslog eventually lead me to the systemd service `auth-rpcgss-module`.
It failed to start. The reason was a kernel update that had been installed
but not yet activated. Rebooting fixed this by restoring synchronization
of the kernel version and the modules' required kernel version.
I'm not sure if that module is required but given its name it seems to be.
Reading the krb5 logs (using `journalctl -xeu krb5-kdc.service`) I could
see that the KDC refused to issue service tickets to the server.
There were attempts from the client to get a service ticket earlier that
day that were also denied. In both cases the reason was failing authentication.
The fact that the server was experiencing the issue made me think that it
was a host authentication issue that had nothing to do with the user.
This later turned out to be correct.
# The Solution
After spending days googling for a solution and trying different things
I decided to completely reconfigure host-related principals.
Here's exactly what I did:
Server:
```sh
srv# rm /etc/krb5.keytab
srv# kadmin -p admin/admin
kadmin: purgekeys host/srv.himbeerserver.de
kadmin: purgekeys nfs/srv.himbeerserver.de
kadmin: delprinc host/srv.himbeerserver.de
kadmin: delprinc nfs/srv.himbeerserver.de
kadmin: addprinc -randkey host/srv.himbeerserver.de
kadmin: addprinc -randkey nfs/srv.himbeerserver.de
kadmin: ktadd host/srv.himbeerserver.de
kadmin: ktadd nfs/srv.himbeerserver.de
kadmin: quit
srv# systemctl restart nfs-kernel-server rpc-gssd rpc-svcgssd
```
It's important to restart rpc-gssd to make it reload the keytab.
I'm not sure if restarting rpc-svcgssd is necessary.
Purging the user keys is *probably* not needed either but you can
do it if the above steps didn't work.
Client (repeat for all affected clients with the corresponding keys):
```sh
clt# rm /etc/krb5.keytab
clt# kadmin -p admin/admin
kadmin: purgekeys host/clt.himbeerserver.de
kadmin: purgekeys nfs/clt.himbeerserver.de
kadmin: delprinc host/clt.himbeerserver.de
kadmin: delprinc nfs/clt.himbeerserver.de
kadmin: addprinc -randkey host/clt.himbeerserver.de
kadmin: addprinc -randkey nfs/clt.himbeerserver.de
kadmin: ktadd host/clt.himbeerserver.de
kadmin: ktadd nfs/clt.himbeerserver.de
kadmin: quit
clt# systemctl restart rpc-gssd
```
Once again purging the user keys is *probably* not needed but you
can do it if the above steps didn't work.
Now mount the NFS share again. If it still doesn't work, reboot
the server and the client. If that doesn't fix it unfortunately
I can't help you.
[Return to Guide List](/cgi-bin/guides.lua)
[Return to Index Page](/cgi-bin/index.lua)
|
