NFS and Lustre: a curious corner case
Jun. 17th, 2016 01:00 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
sawylFor various reasons, we export our Lustre file systems via NFS. For other reasons, not all users have accounts on the Lustre appliance. Naively, given the way NFS and POSIX file permissions work, you'd expect this be OK. But no: unless a user and their groups are defined in a way that can be resolved by the Lustre MDS, the result is an intermittent mishmash of confusing errors and failures.
The problem can be understood as follows:
The user attempts to list the contents of one of the NFS mounted Lustre file systems. The NFS client attempts to satisfy the request using the metadata cache. If the cache has been populated with user data — if, for example, someone else with access to the file system has listed its contents on the same client within the expiry period — the requests are satisfied locally and the user will get a sensible response. If the cache is not fully populated or some items have expired, the user will get partially incomplete results; for example user, group and file size information replaced by multiple question marks.
If the cache is empty or the user attempts to access the contents of a file, the query cannot be satisfied by the NFS client cache and will be passed to the NFS server for the underlying Lustre file system to service the request. When the request is passed to the Lustre metadata server, it carries out an identity upcall request to obtain the user's UID and group privileges. Because the user is not defined, the upcall fails and Lustre responds with an error — either permission denied or an IO error depending on the requested action — which propagates back through NFS to the client.
Consequently users without Lustre accounts cannot access or list files on NFS shares backed by Lustre even when the raw POSIX file permissions would seem to allow them.
All of this looks pretty baffling at first, especially if you haven't remembered to account for the weirdness caused by NFS caching. But once you've though it all thorough — I have to admit, I had to go away and think about it overnight before I understood what was going on — it all seems pretty obvious...
The problem can be understood as follows:
The user attempts to list the contents of one of the NFS mounted Lustre file systems. The NFS client attempts to satisfy the request using the metadata cache. If the cache has been populated with user data — if, for example, someone else with access to the file system has listed its contents on the same client within the expiry period — the requests are satisfied locally and the user will get a sensible response. If the cache is not fully populated or some items have expired, the user will get partially incomplete results; for example user, group and file size information replaced by multiple question marks.
If the cache is empty or the user attempts to access the contents of a file, the query cannot be satisfied by the NFS client cache and will be passed to the NFS server for the underlying Lustre file system to service the request. When the request is passed to the Lustre metadata server, it carries out an identity upcall request to obtain the user's UID and group privileges. Because the user is not defined, the upcall fails and Lustre responds with an error — either permission denied or an IO error depending on the requested action — which propagates back through NFS to the client.
Consequently users without Lustre accounts cannot access or list files on NFS shares backed by Lustre even when the raw POSIX file permissions would seem to allow them.
All of this looks pretty baffling at first, especially if you haven't remembered to account for the weirdness caused by NFS caching. But once you've though it all thorough — I have to admit, I had to go away and think about it overnight before I understood what was going on — it all seems pretty obvious...
