The module lifecycle stage: Preview

The module has requirements for installation

How to check module health?

To do this, you need to check the status of the pods in the d8-csi-scsi-generic namespace. All pods should be in the Running or Completed state and should be running on all nodes.

kubectl -n d8-csi-scsi-generic get pod -owide -w

How to verify Fibre Channel prerequisites on a node?

Check that Fibre Channel adapters (FC HBA) are available on the node and that LUN paths are visible after Storage Area Network (SAN) zoning:

To verify that, run the following commands:

ls /sys/class/fc_host/
ls -l /dev/disk/by-path/ | grep -E 'fc-|/fc-'

Possible results:

  • If the /sys/class/fc_host/ directory is empty, the node has no FC HBA or the corresponding driver is not loaded.
  • If there are no fc-* entries in the /dev/disk/by-path/ directory, check zoning and LUN masking settings on the storage system.
  • If the FC paths are displayed, the node is ready to detect LUN provided via Fibre Channel.

Why are no SCSIDevice objects created for an Fibre Channel SCSITarget?

If no SCSIDevice objects are present after a SCSITarget is created, check the following:

  • WWPN values set in the spec.fibreChannel.WWNs field match the storage system ports.
  • Cluster nodes have access to LUNs and can detect FC paths (there are fc-* entries in the /dev/disk/by-path/ directory).
  • Only one connection type is specified under spec in the SCSITarget: iSCSI or fibreChannel.

Also inspect controller logs in the d8-csi-scsi-generic namespace and verify the target WWPN with your storage administrator.

Where do open-iscsi and multipath-tools on a node come from?

From the module itself. The NodeGroupConfiguration never asks the node’s package manager for them, so a node in a closed environment with no route to the distribution’s repositories is set up the same way as any other, and no bashible run waits on those repositories.

On every node it serves, bashible pulls the module’s package image iscsi-tools from the module’s registry, unpacks it and runs its install script. The stack is two pairs — iscsiadm with iscsid, and multipath with multipathd — and the script decides for each pair separately:

  • a pair the host already has of its own stays the host’s, and the module only enables and configures the distribution’s unit for its daemon;
  • a pair the host lacks lands under /var/lib/deckhouse/sds/csi-scsi-generic, every binary is started through its own dynamic loader with its own library path, and a unit of the module’s comes up for its daemon.

A client and its daemon always come from the same source, because iscsiadm of one version does not speak to iscsid of another. The two pairs do not talk to each other, so a node may well run the host’s iscsid next to the module’s multipathd.

To tell which pair a node took from where, look at what is running on it:

# the distribution's daemons
systemctl is-active iscsid multipathd
# the module's daemons and binaries
systemctl is-active d8-csi-scsi-generic-iscsid.service d8-csi-scsi-generic-multipathd.service
ls /var/lib/deckhouse/sds/csi-scsi-generic/bin

The module’s stack changes little on the host itself. Its libraries are never merged into the host’s /lib64 or /usr/lib; the only files it puts outside its own directory are the multipath path checkers in /usr/lib/multipath, where multipathd looks for them, and only those the host does not already have. And it does not touch /etc/iscsi/initiatorname.iscsi when the node already has one: the IQN is the node’s identity on the array, registered there in a host object, and a node that comes back under a different name is a node the array has never heard of.

sg3-utils is not installed at all: no code path in the driver invokes its tools.

How much of a volume is reserved for the superuser?

Nothing, by default: the module formats ext4 with -m 0, so the whole volume is available to the workload.

Where the classic 5% ext4 reserve is wanted — for example to keep a filesystem writable for a privileged process after a workload has filled it — annotate the SCSIStorageClass with the percentage to reserve:

d8 k annotate scsistorageclass <name> storage.deckhouse.io/ext4-reserved-percent=5

The value is a whole number of percent between 0 and 50; an invalid one puts the SCSIStorageClass into the Failed phase with the reason in status.reason, instead of breaking volume creation later.

The reserve applies only to volumes created after the annotation was set — filesystems that already exist keep the reserve they were created with. Only the ext filesystems keep blocks for the superuser, so with fsType: xfs the annotation is ignored and the node logs a warning.

Changing the annotation makes the controller recreate the StorageClass, because parameters of an existing StorageClass are immutable in Kubernetes. Existing volumes and PVCs are not affected.

Why does a PersistentVolumeClaim stay in Resizing after its size was increased?

csi-scsi-generic does not manage LUNs on the storage array, so it cannot grow a volume itself. When a PersistentVolumeClaim asks for more than its LUN holds, the driver refuses the resize with the message administrator action required and records the request as a PendingResizeRequest in the namespace of the PVC. The PVC stays in the Resizing state, and the application keeps the old size:

d8 k get pendingresizerequests -A

The PVC, RequestedSize and WWID columns name the claim, the size it asks for and the LUN to grow. To finish the resize:

  1. On the storage array, grow the LUN with that WWID to at least the requested size.
  2. Wait for the module to see the new size: it rescans the devices of every SCSITarget every 5 minutes by default, and the SCSIDevice of the LUN shows the size in status.size.
  3. Kubernetes retries the resize on its own. Once it succeeds, the PendingResizeRequest is deleted.

A PendingResizeRequest is also deleted together with its PVC. One left from a PVC deleted before the module set the PVC as its owner has to be deleted by hand.

A request that waits longer than 10 minutes raises the D8CSIScsiGenericPendingResizeRequest alert, which names the PVC, the WWID and both sizes.

How to give a PVC or a VirtualDisk a specific LUN?

A SCSIStorageClass gives a new volume any Available SCSIDevice it selects that is at least the requested size. To give a volume one particular LUN, make a class that selects only that LUN:

  1. Find the SCSIDevice of the LUN, for example by its WWID:

    d8 k get scsidevices -o custom-columns=NAME:.metadata.name,WWID:.status.wwid,SIZE:.status.size,PHASE:.status.phase

    The device must be in the Available phase, that is, not given to another volume.

  2. Put your own label on it. The controller keeps labels it did not set itself:

    d8 k label scsidevice <device name> storage.example.com/lun=db-data
  3. Create a SCSIStorageClass that selects only that label:

    apiVersion: storage.deckhouse.io/v1alpha1
    kind: SCSIStorageClass
    metadata:
      name: lun-db-data
    spec:
      scsiDeviceSelector:
        matchLabels:
          storage.example.com/lun: db-data
      reclaimPolicy: Retain
  4. Request a volume of that class no larger than the LUN. A PVC:

    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: db-data
    spec:
      storageClassName: lun-db-data
      volumeMode: Block
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: <LUN size or less>

    or a VirtualDisk of a virtual machine:

    apiVersion: virtualization.deckhouse.io/v1alpha2
    kind: VirtualDisk
    metadata:
      name: db-data
    spec:
      persistentVolumeClaim:
        storageClassName: lun-db-data
        size: <LUN size or less>

The module does not write to a LUN it gives to a Block volume; a Filesystem volume gets a filesystem only if the LUN has none. What a VirtualDisk writes there is up to the virtualization module and the disk’s dataSource. When a volume of a class with reclaimPolicy: Delete is deleted, it wipes the LUN (see PV cleanup), so for a LUN that holds data you need, keep reclaimPolicy: Retain as in the example. With Retain the module neither wipes nor frees the LUN when its volume is deleted: the LUN keeps its data and is not given to another volume.

A virtual machine on such a disk migrates to the nodes that see the LUN: present it to every node the VM may run on, through its SCSITarget.

How does the module hide its LUNs from sds-node-configurator?

When the sds-node-configurator module is installed, the controller keeps the csi-scsi-generic-block-device-filter BlockDeviceFilter resource. sds-node-configurator creates no BlockDevice for a device that the filter excludes and deletes an existing one, so an LVMVolumeGroup cannot run pvcreate or vgextend on a LUN of the module. The filter excludes a device by either of two rules:

  • By WWN: the status.blockdevice.storage.deckhouse.io/wwn label of the BlockDevice is one of the WWNs of the SCSIDevice resources. This covers the LUNs the controller has discovered; partitions and multipath maps on top of them carry the same WWN.
  • By target: the status.blockdevice.storage.deckhouse.io/scsitarget label is the IQN of an iSCSI SCSITarget or one of the WWPNs of a Fibre Channel one. This covers every LUN of a declared target from the moment it appears on a node: one mapped to the target since the last discovery, one whose SCSIDevice was deleted by hand, one masked to the initiator of a single node. The label is set by sds-node-configurator versions that support it; with an older version only the WWN rule works.

Before a node logs in to a target or scans a LUN for a volume, the driver checks that the filter excludes the WWN of that LUN. This happens on every attach: when a volume is staged on a node, and when the LUN is attached to be wiped after its volume is deleted. If the filter does not exist yet, cannot be read or does not exclude the WWN, the attach fails, NodeStage returns Unavailable and is retried; the SCSIDevice reports it in the Error condition with the AttachFailed reason. When sds-node-configurator is not installed, nothing is checked.

A node brings onto itself only the LUN it attaches. iSCSI sessions are created with node.session.scan=manual, and after the login only the LUN of the volume is scanned. For Fibre Channel, only the LUN of the volume is scanned, through the remote ports whose WWPN is listed in the SCSITarget.

To check what the filter hides:

kubectl get blockdevicefilters.storage.deckhouse.io csi-scsi-generic-block-device-filter -o yaml
kubectl get blockdevices.storage.deckhouse.io -L status.blockdevice.storage.deckhouse.io/scsitarget

The value of the scsitarget label is the IQN in lower case with colons replaced by underscores (iqn.2000-01.com.example:target becomes iqn.2000-01.com.example_target), or the WWPN as 16 lower-case hex digits. See the sds-node-configurator FAQ for the full format.

Limitations:

  • The filter decides which BlockDevice resources exist. LVM commands run on a node by hand, and other software that scans disks, still see the LUNs.
  • A full rescan run on a node by someone else (rescan-scsi-bus.sh, iscsiadm -m session --rescan, echo "- - -" to a SCSI host’s scan file) brings every LUN of the target onto the node. The rule by target still hides them from sds-node-configurator.
  • Discovery scans all Fibre Channel hosts of the controller’s node, so LUNs of other storage systems zoned to that node appear on it too. They do not belong to the module and are not hidden.
  • When the paths of a LUN lead to several targets (several Fibre Channel ports of a storage system, or an iSCSI storage system with a target name per controller), the label names the smallest of them. Declare all the target ports zoned to the nodes, otherwise such a LUN stays visible to sds-node-configurator until the controller discovers it.