The functionality of the module might change, but the main features will remain. Compatibility with future versions is guaranteed, but might require additional migration steps.
The module is only guaranteed to work if requirements are met. As for any other configurations, the module may work, but its smooth operation is not guaranteed.
This module manages replicated block storage based on DRBD
. Currently, LINSTOR
is used as a control-plane. The module allows you to create a Storage Pool
in LINSTOR
as well as a StorageClass
in Kubernetes
by creating Kubernetes custom resources.
To create a Storage Pool
, you will need the LVMVolumeGroup
configured on the cluster nodes. The LVM
configuration is done by the sds-node-configurator module.
Caution! Before enabling the
sds-replicated-volume
module, you must enable thesds-node-configurator
module.Caution! The user is not allowed to configure the
LINSTOR
backend directly.Caution! Data synchronization during volume replication is carried out in synchronous mode only, asynchronous mode is not supported.
After you enable the sds-replicated-volume
module in the Deckhouse configuration, your cluster will be automatically set to use the LINSTOR
backend. You will only have to create storage pools and StorageClasses.
Caution! The user is not allowed to create a
StorageClass
for the replicated.csi.storage.deckhouse.io CSI driver.
Two modes are supported: LVM and LVMThin. Each mode has its advantages and disadvantages. Read FAQ to learn more and compare them.
Quickstart guide
Note that all commands must be run on a machine that has administrator access to the Kubernetes API.
Enabling modules
- Enable the sds-node-configurator module
kubectl apply -f - <<EOF
apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
name: sds-node-configurator
spec:
enabled: true
version: 1
EOF
- Wait for it to become
Ready
. At this stage, you do NOT need to check the pods in thed8-sds-node-configurator
namespace.
kubectl get module sds-node-configurator -w
- Enable the
sds-replicated-volume
module. Refer to the configuration to learn more about module settings. In the example below, the module is launched with the default settings. This will result in the following actions across all cluster nodes:- installation of the
DRBD
kernel module; - registration of the CSI driver;
- launch of service pods for the
sds-replicated-volume
components.
- installation of the
kubectl apply -f - <<EOF
apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
name: sds-replicated-volume
spec:
enabled: true
version: 1
EOF
- Wait for the module to become
Ready
.
kubectl get module sds-replicated-volume -w
- Make sure that all pods in
d8-sds-replicated-volume
andd8-sds-node-configurator
namespaces areRunning
orCompleted
and are running on all nodes whereDRBD
resources are intended to be used.
kubectl -n d8-sds-replicated-volume get pod -owide -w
kubectl -n d8-sds-node-configurator get pod -o wide -w
Configuring storage on nodes
You need to create LVM
volume groups on the nodes using LVMVolumeGroup
custom resources. As part of this quickstart guide, we will create a regular Thick
storage. See usage examples to learn more about custom resources.
To configure the storage:
- List all the BlockDevice resources available in your cluster:
kubectl get bd
NAME NODE CONSUMABLE SIZE PATH
dev-0a29d20f9640f3098934bca7325f3080d9b6ef74 worker-0 true 30Gi /dev/vdd
dev-457ab28d75c6e9c0dfd50febaac785c838f9bf97 worker-0 false 20Gi /dev/vde
dev-49ff548dfacba65d951d2886c6ffc25d345bb548 worker-1 true 35Gi /dev/vde
dev-75d455a9c59858cf2b571d196ffd9883f1349d2e worker-2 true 35Gi /dev/vdd
dev-ecf886f85638ee6af563e5f848d2878abae1dcfd worker-0 true 5Gi /dev/vdb
- Create an LVMVolumeGroup resource for
worker-0
:
kubectl apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: LvmVolumeGroup
metadata:
name: "vg-1-on-worker-0" # The name can be any fully qualified resource name in Kubernetes. This LvmVolumeGroup resource name will be used to create ReplicatedStoragePool in the future
spec:
type: Local
blockDeviceNames: # specify the names of the BlockDevice resources that are located on the target node and whose CONSUMABLE is set to true. Note that the node name is not specified anywhere since it is derived from BlockDevice resources.
- dev-0a29d20f9640f3098934bca7325f3080d9b6ef74
- dev-ecf886f85638ee6af563e5f848d2878abae1dcfd
actualVGNameOnTheNode: "vg-1" # the name of the LVM VG to be created from the above block devices on the node
EOF
- Wait for the created
LVMVolumeGroup
resource to becomeOperational
:
kubectl get lvg vg-1-on-worker-0 -w
-
The resource becoming
Operational
means that an LVM VG namedvg-1
made up of the/dev/vdd
and/dev/vdb
block devices has been created on theworker-0
node. -
Next, create an LVMVolumeGroup resource for
worker-1
:
kubectl apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: LvmVolumeGroup
metadata:
name: "vg-1-on-worker-1"
spec:
type: Local
blockDeviceNames:
- dev-49ff548dfacba65d951d2886c6ffc25d345bb548
actualVGNameOnTheNode: "vg-1"
EOF
- Wait for the created
LVMVolumeGroup
resource to becomeOperational
:
kubectl get lvg vg-1-on-worker-1 -w
-
The resource becoming
Operational
means that an LVM VG namedvg-1
made up of the/dev/vde
block device has been created on theworker-1
node. -
Create an LVMVolumeGroup resource for
worker-2
:
kubectl apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: LvmVolumeGroup
metadata:
name: "vg-1-on-worker-2"
spec:
type: Local
blockDeviceNames:
- dev-75d455a9c59858cf2b571d196ffd9883f1349d2e
actualVGNameOnTheNode: "vg-1"
EOF
- Wait for the created
LVMVolumeGroup
resource to becomeOperational
:
kubectl get lvg vg-1-on-worker-2 -w
-
The resource becoming
Operational
means that an LVM VG namedvg-1
made up of the/dev/vdd
block device has been created on theworker-2
node. -
Now that we have all the LVM VGs created on the nodes, create a ReplicatedStoragePool out of those VGs:
kubectl apply -f -<<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: ReplicatedStoragePool
metadata:
name: data
spec:
type: LVM
lvmVolumeGroups: # Here, specify the names of the LvmVolumeGroup resources you created earlier
- name: vg-1-on-worker-0
- name: vg-1-on-worker-1
- name: vg-1-on-worker-2
EOF
- Wait for the created
ReplicatedStoragePool
resource to becomeCompleted
:
kubectl get rsp data -w
- Confirm that the
data
Storage Pool has been created on nodesworker-0
,worker-1
andworker-2
in LINSTOR:
alias linstor='kubectl -n d8-sds-replicated-volume exec -ti deploy/linstor-controller -- linstor'
linstor sp l
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ StoragePool ┊ Node ┊ Driver ┊ PoolName ┊ FreeCapacity ┊ TotalCapacity ┊ CanSnapshots ┊ State ┊ SharedName ┊
╞═════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ DfltDisklessStorPool ┊ worker-0 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊ worker-0;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ worker-1 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊ worker-1;DfltDisklessStorPool ┊
┊ DfltDisklessStorPool ┊ worker-2 ┊ DISKLESS ┊ ┊ ┊ ┊ False ┊ Ok ┊ worker-2;DfltDisklessStorPool ┊
┊ data ┊ worker-0 ┊ LVM ┊ vg-1 ┊ 35.00 GiB ┊ 35.00 GiB ┊ False ┊ Ok ┊ worker-0;data ┊
┊ data ┊ worker-1 ┊ LVM ┊ vg-1 ┊ 35.00 GiB ┊ 35.00 GiB ┊ False ┊ Ok ┊ worker-1;data ┊
┊ data ┊ worker-2 ┊ LVM ┊ vg-1 ┊ 35.00 GiB ┊ 35.00 GiB ┊ False ┊ Ok ┊ worker-2;data ┊
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
- Create a ReplicatedStorageClass resource for a zone-free cluster (see use cases for details on how zonal ReplicatedStorageClasses work):
kubectl apply -f -<<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: ReplicatedStorageClass
metadata:
name: replicated-storage-class
spec:
storagePool: data # Here, specify the name of the ReplicatedStoragePool you created earlier
reclaimPolicy: Delete
topology: Ignored # - note that setting "ignored" means there should be no zones (nodes labeled topology.kubernetes.io/zone) in the cluster
EOF
- Wait for the created
ReplicatedStorageClass
resource to becomeCreated
:
kubectl get rsc replicated-storage-class -w
- Confirm that the corresponding
StorageClass
has been created:
kubectl get sc replicated-storage-class
- If
StorageClass
with the namereplicated-storage-class
is shown, then the configuration of thesds-replicated-volume
module is complete. Now users can create PVs by specifyingStorageClass
with the namereplicated-storage-class
. Given the above settings, a volume will be created with 3 replicas on different nodes.
System requirements and recommendations
Requirements
- Stock kernels shipped with the supported distributions.
- High-speed 10Gbps network.
- Do not use another SDS (Software defined storage) to provide disks to our SDS.