The functionality of the module might change, but the main features will remain. Compatibility with future versions is guaranteed, but might require additional migration steps.
This module manages local block storage based on LVM
. The module allows you to create a StorageClass
in Kubernetes
by creating Kubernetes custom resources LocalStorageClass
(see example below).
To create a Storage Class
, you will need the LVMVolumeGroup
configured on the cluster nodes. The LVM
configuration is done by the sds-node-configurator module.
Caution! Before enabling the
sds-local-volume
module, you must enable thesds-node-configurator
module.
After you enable the sds-local-volume
module in the Deckhouse Kubernetes Platform configuration, you have to create StorageClasses.
Caution! The user is not allowed to create a
StorageClass
for the local.csi.storage.deckhouse.io CSI driver.
Two modes are supported: LVM and LVMThin. Each mode has its advantages and disadvantages. Read FAQ to learn more and compare them.
Quickstart guide
Note that all commands must be run on a machine that has administrator access to the Kubernetes API.
Enabling modules
- Enable the sds-node-configurator module
kubectl apply -f - <<EOF
apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
name: sds-node-configurator
spec:
enabled: true
version: 1
EOF
- Wait for it to become
Ready
. At this stage, you do NOT need to check the pods in thed8-sds-node-configurator
namespace.
kubectl get modules sds-node-configurator -w
- Enable the
sds-local-volume
module. Refer to the configuration to learn more about module settings. In the example below, the module is launched with the default settings. This will result This will cause the service pads of thesds-local-volume
components to be launched on all nodes of the cluster. - This will cause the service pods of the
sds-local-volume
components to be launched on all nodes of the cluster.
kubectl apply -f - <<EOF
apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
name: sds-local-volume
spec:
enabled: true
version: 1
EOF
- Wait for the module to become
Ready
.
kubectl get mc sds-local-volume -w
- Make sure that all pods in
d8-sds-local-volume
andd8-sds-node-configurator
namespaces areRunning
orCompleted
and are running on all nodes whereLVM
resources are intended to be used.
kubectl -n d8-sds-local-volume get pod -owide -w
kubectl -n d8-sds-node-configurator get pod -o wide -w
Preparing nodes for storage provisioning
To create storage on nodes, it’s necessary for the sds-local-volume-csi-node
pods to be running on the selected nodes.
By default, the pods will be scheduled on all nodes in the cluster. You can verify their presence using the command:
kubectl -n d8-sds-local-volume get pod -owide
The location of the pod data is determined by special labels (nodeSelector) specified in the
spec.settings.dataNodes.nodeSelector
field in the module settings. For more detailed information on the configuration, please follow the link.
Configuring storage on nodes
You need to create LVM
volume groups on the nodes using LVMVolumeGroup
custom resources. As part of this quickstart guide, we will create a regular Thick
storage.
Please ensure that the
sds-local-volume-csi-node
pod is running on the node before creating theLVMVolumeGroup
. You can do this using the command:
kubectl -n d8-sds-local-volume get pod -owide
To configure the storage:
- List all the BlockDevice resources available in your cluster:
kubectl get bd
NAME NODE CONSUMABLE SIZE PATH
dev-ef4fb06b63d2c05fb6ee83008b55e486aa1161aa worker-0 false 976762584Ki /dev/nvme1n1
dev-0cfc0d07f353598e329d34f3821bed992c1ffbcd worker-0 false 894006140416 /dev/nvme0n1p6
dev-7e4df1ddf2a1b05a79f9481cdf56d29891a9f9d0 worker-1 false 976762584Ki /dev/nvme1n1
dev-b103062f879a2349a9c5f054e0366594568de68d worker-1 false 894006140416 /dev/nvme0n1p6
dev-53d904f18b912187ac82de29af06a34d9ae23199 worker-2 false 976762584Ki /dev/nvme1n1
dev-6c5abbd549100834c6b1668c8f89fb97872ee2b1 worker-2 false 894006140416 /dev/nvme0n1p6
- Create an LVMVolumeGroup resource for
worker-0
:
kubectl apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: LVMVolumeGroup
metadata:
name: "vg-1-on-worker-0" # The name can be any fully qualified resource name in Kubernetes. This LVMVolumeGroup resource name will be used to create LocalStorageClass in the future
spec:
type: Local
local:
nodeName: "worker-0"
blockDeviceSelector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: In
values:
- dev-ef4fb06b63d2c05fb6ee83008b55e486aa1161aa
- dev-0cfc0d07f353598e329d34f3821bed992c1ffbcd
actualVGNameOnTheNode: "vg-1" # the name of the LVM VG to be created from the above block devices on the node
EOF
- Wait for the created
LVMVolumeGroup
resource to becomeOperational
:
kubectl get lvg vg-1-on-worker-0 -w
-
The resource becoming
Operational
means that an LVM VG namedvg-1
made up of the/dev/nvme1n1
and/dev/nvme0n1p6
block devices has been created on theworker-0
node. -
Next, create an LVMVolumeGroup resource for
worker-1
:
kubectl apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: LVMVolumeGroup
metadata:
name: "vg-1-on-worker-1"
spec:
type: Local
local:
nodeName: "worker-1"
blockDeviceSelector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: In
values:
- dev-7e4df1ddf2a1b05a79f9481cdf56d29891a9f9d0
- dev-b103062f879a2349a9c5f054e0366594568de68d
actualVGNameOnTheNode: "vg-1"
EOF
- Wait for the created
LVMVolumeGroup
resource to becomeOperational
:
kubectl get lvg vg-1-on-worker-1 -w
-
The resource becoming
Operational
means that an LVM VG namedvg-1
made up of the/dev/nvme1n1
and/dev/nvme0n1p6
block device has been created on theworker-1
node. -
Create an LVMVolumeGroup resource for
worker-2
:
kubectl apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: LVMVolumeGroup
metadata:
name: "vg-1-on-worker-2"
spec:
type: Local
local:
nodeName: "worker-2"
blockDeviceSelector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: In
values:
- dev-53d904f18b912187ac82de29af06a34d9ae23199
- dev-6c5abbd549100834c6b1668c8f89fb97872ee2b1
actualVGNameOnTheNode: "vg-1"
EOF
- Wait for the created
LVMVolumeGroup
resource to becomeOperational
:
kubectl get lvg vg-1-on-worker-2 -w
-
The resource becoming
Operational
means that an LVM VG namedvg-1
made up of the/dev/nvme1n1
and/dev/nvme0n1p6
block device has been created on theworker-2
node. -
Create a LocalStorageClass resource:
kubectl apply -f -<<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: LocalStorageClass
metadata:
name: local-storage-class
spec:
lvm:
lvmVolumeGroups:
- name: vg-1-on-worker-0
- name: vg-1-on-worker-1
- name: vg-1-on-worker-2
type: Thick
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
EOF
- Wait for the created
LocalStorageClass
resource to becomeCreated
:
kubectl get lsc local-storage-class -w
- Confirm that the corresponding
StorageClass
has been created:
kubectl get sc local-storage-class
- If
StorageClass
with the namelocal-storage-class
is shown, then the configuration of thesds-local-volume
module is complete. Now users can create PVs by specifyingStorageClass
with the namelocal-storage-class
.
System requirements and recommendations
Requirements
- Stock kernels shipped with the supported distributions.
- Do not use another SDS (Software defined storage) to provide disks to our SDS.