The csi-hpe module: FAQ

What are the requirements for the csi-hpe module?

The csi-hpe module requires:

• Presence of a deployed and configured HPE SAN.
• Unique iqn in /etc/iscsi/initiatorname.iscsi on each of Kubernetes Nodes
• The snapshot-controller module must be enabled in the cluster (required for volume snapshot functionality).

How to check module health?

To do this, you need to check the status of the pods in the d8-csi-hpe namespace. All pods should be in the Running or Completed state and should be running on all nodes.

kubectl -n d8-csi-hpe get pod -owide -w

How much of a volume is reserved for the superuser?

Nothing, by default: the module tells the driver to format ext4 with -m0, so the whole volume is available to the workload.

Where the classic 5% ext4 reserve is wanted — for example to keep a filesystem writable for a privileged process after a workload has filled it — annotate the HPEStorageClass with the percentage to reserve:

d8 k annotate hpestorageclass <name> storage.deckhouse.io/ext4-reserved-percent=5

The value is a whole number of percent between 0 and 50; an invalid one leaves the HPEStorageClass with Ready=False and the reason in its status, instead of breaking volume creation later.

The reserve applies only to volumes created after the annotation was set — filesystems that already exist keep the reserve they were created with. A class with fsType: xfs is unaffected: XFS keeps no blocks for the superuser.

Changing the annotation makes the controller recreate the StorageClass, because parameters of an existing StorageClass are immutable in Kubernetes. Existing volumes and PVCs are not affected.

Where do open-iscsi and multipath-tools on a node come from?

From the module itself. The NodeGroupConfiguration never asks the node’s package manager for them, so a node in a closed environment with no route to the distribution’s repositories is set up the same way as any other, and no bashible run waits on those repositories.

On every node it serves, bashible pulls the module’s package image iscsi-tools from the module’s registry, unpacks it and runs its install script. The stack is two pairs — iscsiadm with iscsid, and multipath with multipathd — and the script decides for each pair separately:

• a pair the host already has of its own stays the host’s, and the module only enables and configures the distribution’s unit for its daemon;
• a pair the host lacks lands under /var/lib/deckhouse/sds/csi-hpe, every binary is started through its own dynamic loader with its own library path, and a unit of the module’s comes up for its daemon.

A client and its daemon always come from the same source, because iscsiadm of one version does not speak to iscsid of another. The two pairs do not talk to each other, so a node may well run the host’s iscsid next to the module’s multipathd.

Two things the install script arranges for the module’s pairs are worth knowing about, because both are invisible until something does not mount.

The driver reaches the host’s iscsiadm through the container’s /bin/iscsiadm, which is the in-tree host-iscsiadm binary: it chroots into /host and looks the client up there, against a PATH of its own. When iscsiadm is the module’s, the script leaves a wrapper at /usr/local/sbin/iscsiadm, which that PATH searches ahead of the distribution’s directories, and the wrapper defers to a distribution iscsiadm the moment one appears.

And multipathd from the package reads its configuration under the prefix it was built with, not under /etc. When multipathd is the module’s, the script links multipath.conf and multipath from that prefix onto the node’s own. That matters more here than anywhere: the driver locates a staged LUN only through its device-mapper map, so the find_multipaths no this module writes into /etc/multipath/conf.d — on every node, whichever multipathd runs there — is the difference between a working attach and device not found with serial <wwn>.

To tell which pair a node took from where, look at what is running on it:

# the distribution's daemons
systemctl is-active iscsid multipathd
# the module's daemons and binaries
systemctl is-active d8-csi-hpe-iscsid.service d8-csi-hpe-multipathd.service
ls /var/lib/deckhouse/sds/csi-hpe/bin
multipathd show config | grep find_multipaths

The module’s stack changes little on the host itself. Its libraries are never merged into the host’s /lib64 or /usr/lib; the only files it puts outside its own directory are the iscsiadm wrapper, the configuration links and the multipath path checkers in /usr/lib/multipath, where multipathd looks for them — each only for a pair that is the module’s, only where the host has no file of its own, and taken back once that pair becomes the host’s. And it does not touch /etc/iscsi/initiatorname.iscsi when the node already has one: the IQN is the node’s identity on the array, registered there in a host object, and a node that comes back under a different name is a node the array has never heard of.

sg3-utils is not installed at all, and nothing is lost by that: the driver’s only call into it was rescan-scsi-bus.sh, which this module’s patches replaced with direct sysfs writes.

What does the module leave on the HPE array?

The Container Storage Provider for 3PAR, Primera and Alletra 9000 keeps no state in the cluster; everything it needs to find a volume again lives on the array itself:

• a volume is named after the PVC (pvc-<uuid>, cut to the 31 characters the array allows) and carries the comment Volume provisioned by the HPE CSI Driver; its id in the PersistentVolume (spec.csi.volumeHandle) is the volume WWN;
• the full PVC or VolumeSnapshot name is kept in the volume’s key-value metadata (showvv -metadata or GET /api/v1/volumes/<name>/objectKeyValues) under d8_csi_name; a clone being copied also carries d8_copy_task, d8_base_snapshot and, when restored into a larger size, d8_target_mib; a volume requested in a size that is not a whole number of MiB carries d8_requested_bytes;
• a node is registered as a host named iqn-<node> (iSCSI) or wwn-<node> (Fibre Channel) with persona Generic-ALUA, and the node UUID is written into the host comment as csi.hpe.com/uuid=<uuid>.

Do not rename these volumes or hosts and do not remove the d8_* keys: the module finds its objects by them. A host that stays after a node is gone can be removed on the array once it has no VLUNs.

Snapshots and clones need the Virtual Copy licence on the array; without it the array refuses createSnapshot and the VolumeSnapshot reports the array’s own message.

How to verify the array certificate?

By default the CSP does not verify the TLS certificate of the array’s WSAPI endpoint and says so with a warning in its log at every login. Put the CA bundle that signs the array certificates into the module settings:

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: csi-hpe
spec:
  settings:
    arrayCACertificate: |
      -----BEGIN CERTIFICATE-----
      ...
      -----END CERTIFICATE-----

Both Deployments, primera3par-csp and nimble-csp, restart and from then on refuse arrays whose certificate is not signed by that bundle.

Note that the driver reaches the CSP over plain HTTP inside the cluster, as the CSP specification prescribes for this deployment mode, and the csi-controller Pods run in the host network, so the CSP Service cannot be fenced by a NetworkPolicy. Anyone with the array credentials and network access to primera3par-csp-svc or nimble-csp-svc in d8-csi-hpe can manage volumes; keep the HPEStorageConnection credentials and the Secret in d8-csi-hpe protected accordingly.

What does the module leave on a Nimble or Alletra 5000/6000 array?

The Container Storage Provider for HPE Nimble Storage and Alletra 5000/6000 is the same service as the 3PAR one, started with --backend=nimble, and keeps no state in the cluster either:

• a volume carries the full PVC name (the array allows it), the description Volume provisioned by the HPE CSI Driver and agent_type: none; its id in the PersistentVolume is the array volume id; a volume requested in a size that is not a whole number of MiB carries the metadata key d8_requested_bytes;
• a node is registered as an initiator group named after the node (with the suffix -fc for Fibre Channel), one per access protocol, with the node UUID in the metadata key d8_node_uuid; access control records of type both export volumes to it. The CSP API has no call to remove a host, so the initiator group stays on the array after the node leaves the cluster;
• a snapshot carries the VolumeSnapshot name; a restored PVC is an array clone of that snapshot, so the snapshot cannot be deleted while the clone exists;
• when a PVC is deleted the volume is taken offline and destroyed; a volume that still has snapshots is refused. The vendor’s provider kept the volume offline on the array unless the storage class said destroyOnDelete: "true"; with this provider the volume is kept offline on the array when the HPEStorageClass says parameters: {destroy_on_delete: "false"}, and reclaimPolicy: Retain keeps the PersistentVolume as well.

The provider accepts the StorageClass parameters of the vendor’s provider under snake_case names: pool, folder, performance_policy, thick, dedupe_enabled, encrypted, limit_iops, limit_mbps, description, destroy_on_delete. spec.pool of the HPEStorageClass is forwarded as pool; the rest go through spec.parameters. cpg and provisioning_type, which the module always sends, are ignored by this provider (provisioning_type: full maps onto a thick volume). CHAP is not supported.