The module lifecycle stage: Experimental

The module has requirements for installation

Run the commands on a machine with administrator access to the Kubernetes API.

Quick start

The following steps enable the module, connect the cluster to uStor and provision the first volume.

Enabling the module

The module is in the Experimental stage. Before enabling it, allow experimental modules in the deckhouse ModuleConfig:

d8 k patch moduleconfig deckhouse --type=merge --patch '{"spec":{"settings":{"allowExperimentalModules":true}}}'

Then enable the module.

After the module is enabled, it installs the uStor client on the nodes and starts the CSI node plugin. The CSI controller starts once a MindUstorStorageConnection exists.

Preparing the uStor user

The CSI controller manages images as a uStor user. Create a dedicated user with the user role for it instead of using a user with the admin role.

The uStor password policy applies to this user as to any other:

  • The user name must not contain a hyphen. The password can contain only the special characters @$!%*?&.
  • A password set by an administrator, at creation or by a reset, is temporary: uStor rejects the login until the user changes it. Log in as the user once and change the password with ucli user-passwd before putting it into the Secret.
  • A password expires in 60 days by default. After that, uStor rejects the login until the password is changed.
  • A user that has not logged in for 45 days is locked until an administrator unlocks it. The module logs in with the credentials every hour, so the user stays active even if no volumes are created.
  • After three failed logins in a row, the user is locked for 15 minutes.

The password lifetime and the inactivity period are set in /etc/ustor/ustor-middleware.conf on the uStor management nodes (the “Password policy” section of the uStor administrator guide). They apply to every uStor user, including people.

Creating Secrets

The connection refers to two Secrets in the d8-csi-mind-ustor namespace: one with the credentials of the uStor user, the other with the uStor client certificates.

Copy ustor_ca.crt, ustor_client.crt, ustor_client.key, ustor_etcd-client.crt and ustor_etcd-client.key from /etc/ustor/certs of a uStor node to the current directory, then create the Secrets:

d8 k -n d8-csi-mind-ustor create secret generic ustor-credentials \
  --from-literal=username='<USERNAME>' \
  --from-literal=password='<PASSWORD>'

d8 k -n d8-csi-mind-ustor create secret generic ustor-client-tls \
  --from-file=ca.crt=ustor_ca.crt \
  --from-file=client.crt=ustor_client.crt \
  --from-file=client.key=ustor_client.key \
  --from-file=etcd-client.crt=ustor_etcd-client.crt \
  --from-file=etcd-client.key=ustor_etcd-client.key

In the command:

  • <USERNAME>: Name of the uStor user prepared for the CSI controller.
  • <PASSWORD>: Password of this user.

Describing the connection

Describe the connection to the uStor cluster with a MindUstorStorageConnection. Set the address of the uStor management API, the client endpoints of the uStor etcd and the network of the uStor OSDs:

d8 k apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: MindUstorStorageConnection
metadata:
  name: main
spec:
  controlPlane:
    address: "https://ustor-api.example.lab:23004"
    credentialsSecretName: ustor-credentials
  dataPlane:
    etcdAddresses:
      - "https://10.101.0.1:23000"
      - "https://10.101.0.2:23000"
      - "https://10.101.0.3:23000"
    osdNetwork: "10.101.0.0/24"
    tlsSecretName: ustor-client-tls
EOF

The controlPlane.address parameter cannot be changed after the connection is created: the volumes stay on the uStor cluster they were created on.

Check the state of the connection:

d8 k get mindustorstorageconnections.storage.deckhouse.io -o wide

The Created phase means that the connection is in use, both Secrets contain every key it needs and the uStor management API accepts the login of the user. If the phase is Failed, the REASON column says what is wrong.

Creating a StorageClass

Describe the StorageClass with a MindUstorStorageClass that refers to the connection and to a uStor pool:

d8 k apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: MindUstorStorageClass
metadata:
  name: ustor-nbd
spec:
  connectionName: main
  pool: data
  fsType: ext4
  reclaimPolicy: Delete
  volumeBindingMode: Immediate
EOF

Check that the resource is in the Created phase and that the StorageClass exists:

d8 k get mindustorstorageclasses.storage.deckhouse.io -o wide
d8 k get storageclass ustor-nbd

The module creates a StorageClass with the same name for the ustor-nbd.csi.mindsw.io driver.

Creating a volume

Create a PersistentVolumeClaim with the new StorageClass:

d8 k apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ustor-test
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ustor-nbd
  resources:
    requests:
      storage: 1Gi
EOF

Check that the PersistentVolumeClaim is in the Bound state:

d8 k get pvc ustor-test

Limiting the nodes that use uStor

By default, the uStor client configuration is placed on all nodes of the CloudEphemeral, CloudPermanent, CloudStatic and Static types, and the CSI node plugin runs on those of them that have the uStor client installed. If only some nodes reach the uStor data network, select them with dataPlane.nodeSelector of the connection:

apiVersion: storage.deckhouse.io/v1alpha1
kind: MindUstorStorageConnection
metadata:
  name: main
spec:
  # ...
  dataPlane:
    # ...
    nodeSelector:
      matchLabels:
        node.deckhouse.io/group: storage-clients

The StorageClasses and PersistentVolumes of the module are limited to the nodes where the CSI node plugin runs, so pods that use uStor volumes are scheduled only on these nodes, with both the Immediate and WaitForFirstConsumer volume binding modes.

Do not exclude nodes on which uStor volumes are mounted: the CSI node plugin is removed from them, and the volumes cannot be unmounted.

Changing the password of the uStor user

Change the password before it expires, or after an administrator resets it.

While uStor rejects the login, the connection is in the Failed phase, the Authenticated condition of the connection says why (for example, PasswordExpired or UserLocked), and the D8CsiMindUstorStorageConnectionNotInUse alert fires. Mounted volumes keep working, because the data path does not go through the management API. Creating, deleting and expanding volumes stops until the login succeeds.

To change the password:

  1. Change the password in uStor as the user itself with ucli user-passwd.

  2. Put the new password into the Secret:

    d8 k -n d8-csi-mind-ustor create secret generic ustor-credentials \
  --from-literal=username='<USERNAME>' \
  --from-literal=password='<NEW_PASSWORD>' \
  --dry-run=client -o yaml | d8 k apply -f -

    In the command:

    • <USERNAME>: Name of the uStor user of the connection.
    • <NEW_PASSWORD>: New password of this user.

The module and the CSI controller use the new password without a restart: the module logs in again as soon as the Secret changes.

uStor allows a user to change the password only once in 7 days. To change it sooner, an administrator resets the password with ucli user-modify <USERNAME> --reset-password, and the user then changes the temporary password as described above.

Updating the client certificates

To replace the uStor client certificates, update the keys of the TLS Secret of the connection. The module writes the new certificates to the nodes without restarting the CSI node plugin.

Creating snapshots and restoring volumes

Volume snapshots require a module that provides the snapshot.storage.k8s.io API, for example snapshot-controller. With such a module, csi-mind-ustor creates the csi-mind-ustor VolumeSnapshotClass and names it in the storage.deckhouse.io/volumesnapshotclass annotation of every StorageClass it creates, so a VolumeSnapshot does not have to name the class.

A snapshot is stored in the pool of its volume. A volume restored from a snapshot is a full independent copy: deleting the snapshot or the original volume does not affect it.

To create a snapshot of the data PersistentVolumeClaim and restore it into a new PersistentVolumeClaim, apply the manifests:

d8 k apply -f - <<EOF
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: data-snapshot
spec:
  volumeSnapshotClassName: csi-mind-ustor
  source:
    persistentVolumeClaimName: data
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-restored
spec:
  storageClassName: ustor-nbd
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  dataSource:
    apiGroup: snapshot.storage.k8s.io
    kind: VolumeSnapshot
    name: data-snapshot
EOF

The restored volume can be larger than the snapshot: its filesystem is grown to the requested size when the volume is mounted. A restore copies all data of the snapshot, so restoring a large volume takes as long as the copy.

Using raw block volumes

A PersistentVolumeClaim with volumeMode: Block provides the uStor volume to the pod as a raw device, without a filesystem. Such a volume can be used by one pod at a time, so the PersistentVolumeClaim must request the ReadWriteOncePod access mode. The ReadWriteOnce mode is rejected for a block volume.

Example of a PersistentVolumeClaim for a block volume:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: raw
spec:
  storageClassName: ustor-nbd
  volumeMode: Block
  accessModes:
    - ReadWriteOncePod
  resources:
    requests:
      storage: 10Gi

In the pod, refer to the volume in volumeDevices instead of volumeMounts. Block volumes can be expanded online, and they support snapshots and restore as filesystem volumes do.

Block volumes for virtual machines

The live migration of a virtual machine requires its disk to be attached to the source and the target node at once. For this, the driver accepts the ReadWriteMany access mode for block volumes.

The module accepts a ReadWriteMany PersistentVolumeClaim of the driver only from the service accounts of the virtualization module and rejects it for other users. ReadWriteMany with volumeMode: Filesystem is rejected for all users: ext4 and xfs cannot be mounted on several nodes at once.

The module adds the following annotations to every StorageClass it creates:

  • virtualdisk.virtualization.deckhouse.io/volume-mode: Block
  • virtualdisk.virtualization.deckhouse.io/access-mode: ReadWriteMany

With these annotations, the VirtualDisks of the virtualization module on this StorageClass are ReadWriteMany block volumes, and the virtual machines that use them can be live-migrated. If an annotation is already set on the StorageClass, the module keeps its value.

Changing and deleting a StorageClass

The parameters of a MindUstorStorageClass cannot be changed. To provision volumes with other parameters, for example in another pool, create another MindUstorStorageClass.

Deleting a MindUstorStorageClass deletes its StorageClass. The volumes that are already provisioned stay in place.