Module sds-nfs

The sds-nfs module turns one ReadWriteOnce volume into ReadWriteMany storage. For an NFSServer resource it provisions a volume of the chosen StorageClass, formats it with XFS, serves it over NFS from a single pod and publishes an NFSStorageClass of the csi-nfs module with the name of the resource. Every claim of that class gets its own directory on the volume, limited to the size of the claim by an XFS project quota; the module grows the volume as the claims ask for more.

The module ships in the EE and FE editions and requires the csi-nfs module enabled, version 0.4.14 or newer, which issues the client certificate of mTLS for the nodes and publishes its CA in the NFSStorageClass (see Encryption). That module owns the NFSStorageClass resource and mounts the volumes of its classes. The NFSStorageClass of every NFSServer is created and kept by sds-nfs: do not edit it by hand, changes are overwritten, and it is deleted together with the NFSServer. An NFSServer cannot be created under the name of an existing StorageClass or NFSStorageClass: the request is refused, because the class of the server would take that name.

Users with the viewer role of the storage subsystem, or the User access level of user-authz, can read NFSServer resources; the manager role of the storage subsystem, or ClusterEditor and above, can create, change and delete them.

Example

apiVersion: storage.deckhouse.io/v1alpha1
kind: NFSServer
metadata:
  name: shared
spec:
  backend:
    size: 50Gi

The backend volume takes the default StorageClass of the cluster unless spec.backend.storageClassName names another one, and the server runs on the system nodes unless spec.nodeSelector names others (see Placement). spec.backend.storageClassName and spec.storageClass.volumeBindingMode cannot change after the server is created: a volume keeps its class, and csi-nfs cannot change the binding mode of a published NFSStorageClass. spec.backend.size can only grow.

Claims then use the class shared:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data
spec:
  accessModes: [ReadWriteMany]
  storageClassName: shared
  resources:
    requests:
      storage: 5Gi

kubectl get nfsserver shared shows the backend size, the size the claims ask for together, the node the server runs on and the Ready condition; the other conditions say what is not ready and why.

How it behaves

• The backend StorageClass decides the fault tolerance. Any class that provisions ReadWriteOnce volumes with volumeMode: Block will do. With a replicated class the server moves to another node when its node is lost; with a local disk the volumes of the class wait for the node to come back.
• Placement. Without spec.nodeSelector the server runs on the system nodes (node-role.deckhouse.io/system) and tolerates their dedicated.deckhouse.io=system taint, as the system components of Deckhouse do; in a cluster without system nodes it runs on any node but the control plane. spec.nodeSelector replaces this choice. When system nodes appear or the last of them goes, the server moves, with the pause of a move.
• Fencing is required for the move. The server pod leaves a lost node only after the node is fenced, so fencing has to be enabled on the NodeGroups the server may run on (spec.fencing.mode: Watchdog); the FencingReady condition lists the groups without it and the nodes on which the fencing agent has not armed the watchdog.
• Losing the node of the server. When the node of the server is NotReady, the module waits until the node is proven dead:
  • its Node object is deleted by the fencing of the platform, or it has stayed NotReady without renewing its lease for the watchdog timeout of its NodeGroup plus 30 seconds while its watchdog is armed: the fencing agent of the platform runs on it in the Watchdog mode and the node is not in maintenance (update.node.deckhouse.io/approved, update.node.deckhouse.io/disruption-approved or node-manager.deckhouse.io/fencing-disable, under which the agent disarms the watchdog).
  • Then the module deletes the server pod and releases the backend volume from the node, and the server starts on another node; the clients of the dead node, found by the address they mounted from, are forgotten, so the others do not wait for them to reclaim their locks. A node that is only unreachable keeps the server.
  • status.interruption says what the server waits for (WaitingForNodeFencing, ReleasingVolume, Scheduling, Starting, Recovering — the server runs and the clients reclaim their state in the NFSv4 grace period, which is part of the pause), and status.lastInterruption how long the last interruption lasted. An interruption starts when the platform marks the node NotReady, about 40 seconds after the node is lost, so the pause the clients see is that much longer.
  • A move may also wait in Starting for up to six minutes: the attach-detach controller of Kubernetes keeps the backend volume attached to the lost node until it detaches it by force (maxWaitForUnmountDuration), unless the node has the node.kubernetes.io/out-of-service taint, and the fencing of the platform deletes the Node without setting it.
  • In the module’s tests status.lastInterruption of a move off a lost node was 110–160 seconds when the volume was let go at once and about 430 seconds when it was not.
• One server serves every volume of the class. A restart or a move of the server pauses the I/O of all of them until it is back: about 15–30 seconds for a restart in place, longer for a move to another node. Applications do not get I/O errors (the mounts are hard); their writes wait.
• Updates of the module. An update that changes the image of the server restarts the servers of every class at once, not one after another: each class pauses its volumes for the restart, about 15–30 seconds. Plan updates of the module as such a pause for all the NFS volumes of the cluster.
• Maintenance of a node. Before servicing the node of a server, drain it (kubectl drain, or a disruptive update of Deckhouse, which drains the node by default). The server pod is evicted, the StatefulSet starts it on another node of its placement, and the volumes of the class pause as for any move; status.interruption shows it. The drained node stays cordoned until it is uncordoned, so the server does not come back to it, and after the maintenance the server stays where it moved.
• Quotas. df in a pod shows the size of the claim. A write past it fails with ENOSPC (XFS reports a full project as a full file system). Writes past the limit are refused on the server: an application has to check the result of write, fsync and close.
• Synchronous writes are slow. A write the client sends as stable — O_DIRECT, O_SYNC, or an fsync after every small write, as databases do — makes the server sync the file system and the backend volume for that one write. In the module’s load test, 4 KiB random writes with direct=1 reached about 250–1600 IOPS per server, with a p99 of 200–800 ms, while the same writes buffered by the client reached about 2800 IOPS. The module suits shared files and moderate writing, not databases. The server is several times faster when its node holds a full copy of the backend volume.
• Growth. The module asks the backend volume for the size of the claims together with a reserve and grows the file system online. When the claims ask for more than the backend has, the CapacityReady condition is False and says why; a claim already shows its new size by then.
• Access. Every PersistentVolume of the class has an export of its own, and a node may mount it only while it runs a pod that uses the volume; the nodes of the csi-nfs controller may mount every volume and the root of the export, because they create and delete the directories of the volumes. Every other node may only cross the root, read-only and without its data: the volumes it may not mount are hidden from it. A pod cannot mount anything from the pod network, and csi-nfs refuses inline volumes of the servers of the cluster.
  • A node gets a volume within seconds after a pod of it is scheduled there, and loses it once the last such pod is gone; kubelet retries the mount meanwhile. A directory gets its export the moment csi-nfs creates it, before its volume exists, and loses it once the claim of the volume is gone, so that csi-nfs can delete it.
  • The volumes are mounted on the nodes with nosuid and nodev, so a set-user-ID binary or a device file on a volume takes no effect on the node. The container runtime bind-mounts a volume into a pod without these flags; inside the pods securityContext.allowPrivilegeEscalation: false, which the Restricted Pod Security Standard requires, keeps a set-user-ID binary from raising privileges.
  • Root on a node, or a privileged pod there, can read the volumes of the pods that run on it. That is the limit of any storage a node mounts.
• Encryption and the client certificate. The server takes RPC-with-TLS connections (RFC 9289) alone, and only from a client that presents the client certificate of the nodes (mTLS): the traffic between the nodes and the server is always encrypted, a node checks that it talks to the server, and the server checks that it talks to a node. A pod in the host network of a node shares the address of the node, but without the key of the node, which only root or a hostPath mount can read, the server refuses it. There is nothing to configure.
  • The module keeps its CA in the Secret d8-sds-nfs/sds-nfs-ca and issues every server a certificate for the address of its Service. The CA reaches the nodes in spec.tlsParameters.ca of the NFSStorageClass of the server; tlsParameters of the csi-nfs module are not needed for it.
  • The client certificate is the one csi-nfs issues for the nodes; csi-nfs publishes its CA in status.clientCA of the NFSStorageClass of the server, and the server trusts that CA. A node presents the same client certificate to every NFS server it mounts with mTLS, so a client certificate set in tlsParameters.mtls of csi-nfs takes the place of the issued one for the servers of the module as well, and needs tlsParameters.mtls.clientCA; without it the TLSReady condition is False and says that the server waits for the client CA.
  • The module ships in the EE and FE editions and needs a csi-nfs whose NFSStorageClass has spec.tlsParameters.ca, status.clientCA and spec.mountOptions.nosuid; with an older csi-nfs the TLSReady condition is False and says to update it.
  • Every node that mounts the volumes needs a kernel with RPC-with-TLS: 6.5 or later, with CONFIG_TLS and CONFIG_NET_HANDSHAKE. The ClientNodesReady condition lists the nodes with an older kernel: pods there cannot mount the volumes. On such nodes (Astra and Debian with kernel 6.1, for example) the pods of csi-nfs do not start either once the cluster has a CA.
  • The certificates are valid for ten years and are not rotated on a schedule: ganesha reads a new certificate only when it restarts, and a restart pauses every volume of the class. A new certificate is issued when the address of the Service changes, and the server restarts with it.
  • A server created by a version of the module without mTLS does not serve its volumes after the update: its NFSStorageClass was published without mTLS, its connection cannot change, and the NFSStorageClassReady condition says that the server has to be created again.
• Deletion. An NFSServer is not deleted while PersistentVolumes of its class exist: until the last of them is gone it keeps serving them, and moves off a lost node, with Ready=False and the reason DeletionBlocked. Deleting it deletes the backend volume and the data on it.
• Monitoring. The dashboard SDS-NFS / NFS servers in Grafana shows the state of the servers, their interruptions and backend volumes, the operations, latency and throughput of NFS from the metrics of ganesha, and how full every volume is (sds_nfs_volume_used_bytes against sds_nfs_volume_limit_bytes, by namespace and claim). The alerts SdsNfsServerInterrupted, SdsNfsServerNotReady, SdsNfsServerCapacityNotReady and SdsNfsServerFencingNotReady fire on the conditions of the servers.