The csi-yadro-tatlin-unified module: FAQ

How to check module health?

To do this, you need to check the status of the pods in the d8-csi-yadro-tatlin-unified namespace. All pods should be in the Running or Completed state and should be running on all nodes.

kubectl -n d8-csi-yadro-tatlin-unified get pod -owide -w

How much of a volume is reserved for the superuser?

Nothing, by default: a volume formatted with ext4 gets -m0, so the whole volume is available to the workload.

Where the classic 5% ext4 reserve is wanted — for example to keep a filesystem writable for a privileged process after a workload has filled it — annotate the YadroTatlinUnifiedStorageClass with the percentage to reserve:

d8 k annotate yadrotatlinunifiedstorageclass <name> storage.deckhouse.io/ext4-reserved-percent=5

The value is a whole number of percent between 0 and 50; an invalid one leaves the YadroTatlinUnifiedStorageClass with Ready=False and the reason in its status, instead of breaking volume creation later.

The reserve applies only to volumes created after the annotation was set — filesystems that already exist keep the reserve they were created with. Only the ext filesystems keep blocks for the superuser, so a class with fsType: xfs ignores it.

Changing the annotation makes the controller recreate the StorageClass, because parameters of an existing StorageClass are immutable in Kubernetes. Existing volumes and PVCs are not affected.

Where do open-iscsi, multipath-tools and nvme-cli on a node come from?

From the module itself. The NodeGroupConfiguration never asks the node’s package manager for them, so a node in a closed environment with no route to the distribution’s repositories is set up the same way as any other, and no bashible run waits on those repositories.

On every node it serves, bashible pulls the module’s package images from the module’s registry, unpacks them and runs their install scripts. There are two such images, and they are independent on purpose: iscsi-tools goes on every node the module serves, nvme-tools only where the module is configured for NVMe-TCP, and a host may have either stack of its own without the other.

iscsi-tools carries two pairs — iscsiadm with iscsid, and multipath with multipathd — and its install script decides for each pair separately:

• a pair the host already has of its own stays the host’s, and the module only enables and configures the distribution’s unit for its daemon;
• a pair the host lacks lands under /var/lib/deckhouse/sds/csi-yadro-tatlin-unified, every binary is started through its own dynamic loader with its own library path, and a unit of the module’s comes up for its daemon.

A client and its daemon always come from the same source, because iscsiadm of one version does not speak to iscsid of another. The two pairs do not talk to each other, so a node may well run the host’s iscsid next to the module’s multipathd.

nvme-tools carries nvme alone — no daemon, and so no version to match — and lands under /var/lib/deckhouse/sds/csi-yadro-tatlin-unified-nvme only on a host that has no nvme of its own. The host needs it for its own configuration rather than for the driver’s calls: the NodeGroupConfiguration runs nvme gen-hostnqn to give the node a stable NVMe Qualified Name, which the driver then registers in the array’s access group. Without it the driver reports initiators is empty and no NVMe-TCP volume can be published to that node.

Two things the iscsi-tools install script arranges for the pairs it lands are worth knowing about, because both are invisible until something does not mount. The driver reaches the host’s iscsiadm through the container’s /bin/iscsiadm, which chroots into /host and looks the client up there against a PATH of its own — the script leaves a wrapper at /usr/local/sbin/iscsiadm for it. And multipathd from the package reads its configuration under the prefix it was built with, so the script links multipath.conf and multipath from that prefix onto the node’s own, which is how the module’s Tatlin device block reaches the daemon.

To tell which pair a node took from where, look at what is running on it:

# the distribution's daemons
systemctl is-active iscsid multipathd
# the module's daemons and binaries
systemctl is-active d8-csi-yadro-tatlin-unified-iscsid.service d8-csi-yadro-tatlin-unified-multipathd.service
ls /var/lib/deckhouse/sds/csi-yadro-tatlin-unified/bin
ls /var/lib/deckhouse/sds/csi-yadro-tatlin-unified-nvme/bin
command -v iscsiadm nvme

The module’s packages change little on the host itself. Their libraries are never merged into the host’s /lib64 or /usr/lib; the only files they put outside their own directories are the wrappers /usr/local/sbin/iscsiadm and /usr/local/sbin/nvme, the multipath links above, and the multipath path checkers in /usr/lib/multipath, where multipathd looks for them — each only for a part the package lands, and none where the host already has a file of its own. And they do not touch /etc/iscsi/initiatorname.iscsi or /etc/nvme/hostnqn when the node already has them: those names are the node’s identity on the array, registered there, and a node that comes back under a different one is a node the array has never heard of.