Available in: Ultimate/EE
Included in extensions: Storage Integration
The module lifecycle stage: Preview
The module has requirements for installation
The module provides a CSI driver for managing volumes using storage systems connected via SCSI.
Supported features:
- Logical Unit Number (LUN) discovery via iSCSI or Fibre Channel
- Creation of PersistentVolume (PV) from pre-provisioned LUNs
- Deletion of a PV and wiping data on the LUN
- Connecting LUN to nodes via iSCSI or Fibre Channel
- Creating multipath devices and mounting them to pods
- Detaching LUN from nodes
Not supported:
- LUN creation on the storage system
- Resizing LUN
- Creating snapshots
Supported access modes for the module: RWO, RWX — only in DVP.
System Requirements and Recommendations
Requirements
-
A deployed and configured storage system providing access to LUN via iSCSI or Fibre Channel.
-
For iSCSI connections:
- A unique iSCSI Qualified Name (IQN) must be configured on each Kubernetes node in the
/etc/iscsi/initiatorname.iscsifile. - The
multipath-toolspackage must be installed on nodes.
- A unique iSCSI Qualified Name (IQN) must be configured on each Kubernetes node in the
-
For Fibre Channel connections:
- Fibre Channel Host Bus Adapters (FC HBA) must be installed and available on cluster nodes (
/sys/class/fc_host/host*). - In the Storage Area Network (SAN), LUN zoning and masking must be configured, providing node initiators with access to the storage target ports.
- Necessary LUNs must be pre-provisioned on the storage system. The module does not create LUNs.
- The
multipath-toolspackage must be installed on nodes.
- Fibre Channel Host Bus Adapters (FC HBA) must be installed and available on cluster nodes (
Quick Start
All commands should be executed on a machine with access to the Kubernetes API and administrator rights.
Enabling the Module
- Enable the
csi-scsi-genericmodule. This will ensure that the following happens on all cluster nodes:- The CSI driver is registered.
- Auxiliary pods for the
csi-scsi-genericcomponents are launched.
kubectl apply -f - <<EOF
apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
name: csi-scsi-generic
spec:
enabled: true
version: 1
EOF- Wait for the module to transition to the
Readystate.
kubectl get module csi-scsi-generic -wCreating an SCSITarget (iSCSI)
The SCSITarget resource describes connection to a single SCSI target. Under spec, specify one of the connection types: iSCSI or fibreChannel.
Every LUN of a target declared in an SCSITarget belongs to the module, including LUNs mapped to the target after the resource was created. The module hands any of them out as a volume and wipes it when the volume is deleted. Do not use LUNs of such a target for anything else — another CSI driver, LVM or file systems on the nodes: their data will be lost. Expose such LUNs through a target that no SCSITarget declares.
The module also hides these LUNs from the sds-node-configurator module, so that an LVMVolumeGroup cannot take them. See How does the module hide its LUNs from sds-node-configurator? for how this works and where it stops.
The following is a configuration example for connecting via iSCSI. In this example, two SCSITarget resources are created. You can create multiple resources for either the same or different storage systems. This lets you use multipath to improve failover and performance.
kubectl apply -f -<<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: SCSITarget
metadata:
name: hpe-3par-1
spec:
deviceTemplate:
metadata:
labels:
my-key: some-label-value
iSCSI:
auth:
login: ""
password: ""
iqn: iqn.2000-05.com.3pardata:xxxx1
portals:
- 192.168.1.1
---
apiVersion: storage.deckhouse.io/v1alpha1
kind: SCSITarget
metadata:
name: hpe-3par-2
spec:
deviceTemplate:
metadata:
labels:
my-key: some-label-value
iSCSI:
auth:
login: ""
password: ""
iqn: iqn.2000-05.com.3pardata:xxxx2
portals:
- 192.168.1.2
EOFCreating an SCSITarget (Fibre Channel)
For a Fibre Channel connection, define a World Wide Port Name (WWPN) of the target storage ports in the spec.fibreChannel.WWNs field, which expose the necessary LUNs. The module discovers devices by matching the defined WWPNs against paths under /dev/disk/by-path/.
Define the WWPNs in one of the following formats:
- 16 hexadecimal characters (
2001c89f1acd6117) - With colons (
20:01:c8:9f:1a:cd:61:17) - With the
0xprefix (0x2001c89f1acd6117)
Before creating the SCSITarget resource, configure zoning and LUN masking on the SAN so that cluster nodes can access the necessary LUNs.
During discovery, the controller scans all FC hosts of its node. When a volume is attached to a node, only the LUN of that volume is scanned, and only through the remote ports whose WWPN is listed in the SCSITarget and whose state is Online. The module does not configure the commutators or log in to targets.
Configuration example with two target storage system ports for multipath:
kubectl apply -f -<<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: SCSITarget
metadata:
name: hpe-3par-fc-1
spec:
deviceTemplate:
metadata:
labels:
my-key: some-label-value
fibreChannel:
WWNs:
- 2001c89f1acd6117
---
apiVersion: storage.deckhouse.io/v1alpha1
kind: SCSITarget
metadata:
name: hpe-3par-fc-2
spec:
deviceTemplate:
metadata:
labels:
my-key: some-label-value
fibreChannel:
WWNs:
- 2001c89f1acd6118
EOFBefore applying the resource verify that FC paths are available on cluster nodes:
# FC hosts present.
ls /sys/class/fc_host/
# Target WWPNs and LUNs available after zoning.
ls -l /dev/disk/by-path/ | grep -E 'fc-|/fc-'After the SCSITarget is created, the controller discovers available LUNs and creates SCSIDevice objects. Use the same SCSIStorageClass resource and scsiDeviceSelector workflow as for iSCSI.
To verify that the object has been created, run the following command:
kubectl get scsitargets.storage.deckhouse.io <scsitarget name>An object is considered to be created successfully if it says Created in the corresponding Phase column in the output.
Creating a StorageClass
To create a StorageClass, use the SCSIStorageClass. An example of commands to create such a resource:
kubectl apply -f -<<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: SCSIStorageClass
metadata:
name: scsi-all
spec:
scsiDeviceSelector:
matchLabels:
my-key: some-label-value
reclaimPolicy: Delete
EOFPay attention to the scsiDeviceSelector. This field is used to select the SCSITarget for PV creation based on labels. In the example above, all SCSITargets with the label my-key: some-label-value are selected. This label will be applied to all devices detected within the specified SCSITarget.
To pick particular LUNs of a SCSITarget, add your own label to their SCSIDevice resources and put it in scsiDeviceSelector. The controller manages only the labels it takes from deviceTemplate, the storage.deckhouse.io/scsi-target-<SCSITarget name> label and the automatic labels described below, and keeps all other labels of the device. If a label from deviceTemplate has the same key as a label you set, the deviceTemplate value wins.
Automatic SCSIDevice labels
The controller also puts labels derived from the status of each SCSIDevice on it, under the reserved prefix status.scsidevice.storage.deckhouse.io/:
| Label | Value |
|---|---|
status.scsidevice.storage.deckhouse.io/wwid |
status.wwid as it is, for example naa.60014050000000000000000000000001 |
status.scsidevice.storage.deckhouse.io/size |
status.size in binary units, for example 100Gi |
status.scsidevice.storage.deckhouse.io/vendor |
status.vendor, the vendor the LUN reports in its INQUIRY data, for example HPE |
status.scsidevice.storage.deckhouse.io/model |
status.model, the model the LUN reports in its INQUIRY data, for example MSA-2050-SAN |
status.scsidevice.storage.deckhouse.io/rota |
status.rotational: true for a rotating medium, false otherwise |
The values of vendor and model are converted the same way as the labels of BlockDevice in the sds-node-configurator module, with the gosimple/slug library: characters that are not allowed in a label value, such as spaces, . and /, become -, the case is kept, and the value is cut to 63 characters. Read the value to put in a selector off the device itself rather than writing it from the status.
The vendor, the model and rotational are read only for Fibre Channel LUNs, from the sysfs of the node the controller runs on (device/vendor, device/model and queue/rotational of the LUN). rotational is what the kernel reports: a LUN that does not report its medium type is counted as rotating. For iSCSI LUNs these fields and labels are not filled in yet.
An automatic label wins over a deviceTemplate label and over your own label with the same key, and keys with this prefix in deviceTemplate are ignored. A label whose value is unknown, or is not a valid label value, is not set. The size label matches one size exactly and changes when the LUN is resized; to select several sizes, use matchExpressions with the In operator. To see the labels of the devices, run d8 k get scsidevices --show-labels.
One LUN reached through several SCSITargets
A LUN reached through several SCSITargets gets the deviceTemplate labels of all of them. If two of them set one key to different values, the value of the SCSITarget whose name comes first in lexical order wins. The device then reports the LabelsConsistent condition with status False and reason LabelKeyConflict, whose message names the key, both SCSITargets and their values, and a Warning Event is recorded on the device. The phase of the device does not change. When the conflict is gone, the condition turns True and a Normal Event is recorded.
- To verify that the object has been created (Phase should be
Created), run:
kubectl get scsistorageclasses.storage.deckhouse.io <scsistorageclass name>PV cleanup
The module does not manage the storage system and re-uses pre-provisioned LUNs, so the LUN of a deleted PV is wiped before its SCSIDevice is returned to the pool and given to another volume. This applies to classes with reclaimPolicy: Delete; for reclaimPolicy: Retain, see the FAQ.
The wipe runs in the background on the CSI controller, so it is not bounded by the timeout of a single request:
- When the volume is deleted, the CSI controller records the request in
status.cleanupof the SCSIDevice and sets theCleanupInProgresscondition toTruewith reasonCleanupPending. Until the LUN is wiped, every deletion attempt fails withAbortedand a message saying how far the wipe is, for examplepass 1 of 1, 42% (4.2 GiB of 10.0 GiB); the message is shown in the events of the PV, which staysReleased. - The CSI controller connects the LUN to the node it runs on, sets
CleanupInProgressto reasonCleanupStarted, with the method and the command in the message, and wipes the LUN with the method below.status.cleanupshows the method, the pass, the bytes wiped, the start time and the error of the last failed attempt. - Once the LUN is wiped, the CSI controller disconnects it, records the time in
status.cleanup.completedAtand setsCleanupInProgresstoFalsewith reasonCleanupCompleted. - The next deletion attempt of the PV removes
status.cleanupand the reservation and succeeds, and the device goes back to theAvailablephase. If the PV is already gone, for example deleted by hand, the CSI controller releases the device itself within 5 minutes.
The device stays reserved for the deleted PV until the PV itself is gone, so it is not given to another volume while the wipe runs or while the old PV still exists.
If an attempt fails, CleanupInProgress turns False with reason CleanupFailed, the error is recorded in status.cleanup.error and in a CleanupFailed event, and the CSI controller retries after 30 seconds, doubling the delay up to 10 minutes. An attempt is also stopped if it makes no progress for 10 minutes. The progress is recorded each time a range is done, at most every 30 seconds, and a wipe interrupted by a failure, a restart of the CSI controller or a change of its leader resumes from the last recorded point. At most two LUNs are wiped at the same time; the others wait for their turn.
The method is set by the volumeCleanup parameter of the SCSIStorageClass. It is read from the StorageClass of the PV when the volume is deleted, so a change applies to existing volumes as well.
volumeCleanup |
What is done |
|---|---|
| Not set | wipefs -a, then blkdiscard if the LUN supports discard, otherwise blkdiscard -z |
Discard |
The same as when not set |
RandomFillSinglePass |
wipefs -a, then the whole LUN is overwritten with random data once |
RandomFillThreePass |
wipefs -a, then the whole LUN is overwritten with random data three times |
The LUN supports discard if /sys/block/<device>/queue/discard_max_bytes of its multipath device is greater than zero. blkdiscard -z writes zeroes over the whole LUN. blkdiscard is run over consecutive ranges, 1 GiB with -z and up to 16 GiB without it, so that the progress can be recorded and a slow but steady wipe is not taken for a stalled one.
The random data is written by the CSI controller itself rather than by a tool of the node: four writers write 4 MiB blocks with O_DIRECT, each from its own ChaCha8 generator seeded from the kernel’s random source, every pass with new data. The progress is recorded every 256 MiB, at most every 30 seconds.
Keep in mind:
- A plain
blkdiscard(SCSI UNMAP) marks the blocks as free but does not guarantee that they read back as zeroes. Whether the old data can still be read from the LUN depends on the storage system. blkdiscard -zon a large LUN whose storage system does not offload zeroing, andRandomFillSinglePassandRandomFillThreePassalways, write the whole LUN over the network of the CSI controller node and take as long as writing that much data, once per pass: at 500 MB/s a pass over 1 TiB takes about 35 minutes. The PV staysReleasedall that time.- A LUN returned to the pool with the
storage.deckhouse.io/scsi-device-releaseannotation is always wiped as whenvolumeCleanupis not set.