The documentation is under development and may contain incomplete information.
Using local storage helps avoid network latencies, which improves performance compared to remote storage that requires network connectivity. This approach is ideal for test environments and EDGE clusters.
To create local block StorageClass objects, you can use the sds-local-volume
module.
Enabling the module
Configuring local block storage is based on the Logical Volume Manager (LVM).
LVM is managed by the sds-node-configurator
module, which must be enabled before activating the sds-local-volume
module.
To enable the module, apply the following ModuleConfig
resource:
d8 k apply -f - <<EOF
apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
name: sds-node-configurator
spec:
enabled: true
version: 1
EOF
Wait for the sds-node-configurator
module to reach the Ready
status.
To check the status, run the following command:
d8 k get modules sds-node-configurator -w
In the output, you should see information about the sds-node-configurator
module:
NAME WEIGHT STATE SOURCE STAGE STATUS
sds-node-configurator 900 Enabled deckhouse Ready
Then, to enable the sds-local-volume
module with default settings, run the following command:
d8 k apply -f - <<EOF
apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
name: sds-local-volume
spec:
enabled: true
version: 1
EOF
This will launch service pods of the sds-local-volume
components on all cluster nodes.
To check the module status, run the following command:
d8 k get modules sds-local-volume -w
In the output, you should see information about the sds-local-volume
module:
NAME WEIGHT STATE SOURCE STAGE STATUS
sds-local-volume 920 Enabled Embedded Ready
To verify that all pods in the d8-sds-local-volume
and d8-sds-node-configurator
namespaces are in the Running
or Completed
state and are deployed on all nodes where LVM resources are planned to be used, use the following commands:
d8 k -n d8-sds-local-volume get pod -w
d8 k -n d8-sds-node-configurator get pod -w
Node preconfiguration
Creating LVM volume groups
Ensure that on all nodes where LVM resources are intended to be used, service pods sds-local-volume-csi-node
are running. These pods provide interaction with nodes where LVM components are located.
This can be verified using the following command:
d8 k -n d8-sds-local-volume get pod -l app=sds-local-volume-csi-node -owide
The placement of these pods on nodes is determined based on specific labels (nodeSelector) defined in the spec.settings.dataNodes.nodeSelector
field in the module settings. For more details on configuration, refer to the documentation.
Before setting up the creation of StorageClass objects, available block devices on nodes need to be combined into LVM volume groups. These volume groups will subsequently be used to host PersistentVolume resources.
To list available block devices, you can use the BlockDevices
resource, which reflects their current state:
d8 k get bd
In the output, you should see a list of available block devices:
NAME NODE CONSUMABLE SIZE PATH
dev-ef4fb06b63d2c05fb6ee83008b55e486aa1161aa worker-0 false 976762584Ki /dev/nvme1n1
dev-0cfc0d07f353598e329d34f3821bed992c1ffbcd worker-0 false 894006140416 /dev/nvme0n1p6
dev-7e4df1ddf2a1b05a79f9481cdf56d29891a9f9d0 worker-1 false 976762584Ki /dev/nvme1n1
dev-b103062f879a2349a9c5f054e0366594568de68d worker-1 false 894006140416 /dev/nvme0n1p6
dev-53d904f18b912187ac82de29af06a34d9ae23199 worker-2 false 976762584Ki /dev/nvme1n1
dev-6c5abbd549100834c6b1668c8f89fb97872ee2b1 worker-2 false 894006140416 /dev/nvme0n1p6
In the example above, six block devices are available across three nodes.
To group block devices on one node, you need to create an LVM volume group using the LVMVolumeGroup
resource.
To create an LVMVolumeGroup
resource on node worker-0, apply the following resource, replacing the node and block device names with your own:
d8 k apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: LVMVolumeGroup
metadata:
name: "vg-on-worker-0"
spec:
type: Local
local:
# Replace with the name of your node for which the volume group is being created.
nodeName: "worker-0"
blockDeviceSelector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: In
values:
# Replace with the names of your node's block devices for which the volume group is being created.
- dev-ef4fb06b63d2c05fb6ee83008b55e486aa1161aa
- dev-0cfc0d07f353598e329d34f3821bed992c1ffbcd
# Name of the LVM volume group that will be created from the specified block devices on the selected node.
actualVGNameOnTheNode: "vg"
# Uncomment if thin provisioning is required. Details will be discussed later.
# thinPools:
# - name: thin-pool-0
# size: 70%
EOF
Details about the configuration options for the LVMVolumeGroup
resource can be found in the reference section.
Wait until the created LVMVolumeGroup
resource transitions to the Ready
phase.
To check the resource phase, run the following command:
d8 k get lvg vg-on-worker-0 -w
In the output, you should see information about the resource phase:
NAME THINPOOLS CONFIGURATION APPLIED PHASE NODE SIZE ALLOCATED SIZE VG AGE
vg-on-worker-0 1/1 True Ready worker-0 360484Mi 30064Mi vg 1h
If the resource transitions to the Ready
phase, this indicates that an LVM volume group named vg
has been created on node worker-0 using the block devices /dev/nvme1n1
and /dev/nvme0n1p6
.
Next, you need to repeat the creation of LVMVolumeGroup
resources for the remaining nodes (worker-1 and worker-2), modifying the resource name, node name, and block device names accordingly.
Ensure that LVM volume groups are created on all nodes where they are intended for use by running the following command:
d8 k get lvg -w
In the output, you should see a list of created volume groups:
NAME THINPOOLS CONFIGURATION APPLIED PHASE NODE SIZE ALLOCATED SIZE VG AGE
vg-on-worker-0 0/0 True Ready worker-0 360484Mi 30064Mi vg 1h
vg-on-worker-1 0/0 True Ready worker-1 360484Mi 30064Mi vg 1h
vg-on-worker-2 0/0 True Ready worker-2 360484Mi 30064Mi vg 1h
Creating a StorageClass
Thick type StorageClass
The creation of StorageClass objects is done through the LocalStorageClass
resource, which defines the configuration for the desired storage class. Manually creating a StorageClass without a LocalStorageClass
can result in errors.
When creating a LocalStorageClass
, it’s crucial to select the storage type, which can be either Thick
or Thin
.
Thick pools offer high performance comparable to the storage device itself but do not support snapshots, whereas Thin pools allow for snapshots and overprovisioning (resource over-allocation) at the cost of reduced performance.
Overprovisioning should be used with caution, ensuring free space in the pool is carefully monitored (the cluster monitoring system generates events when free space falls to 20%, 10%, 5%, and 1%). A lack of free space in the pool can lead to degradation of the module’s operation and poses a real risk of data loss.
Example of creating a LocalStorageClass
resource with a Thick
type:
d8 k apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: LocalStorageClass
metadata:
name: local-storage-class-thick
spec:
lvm:
lvmVolumeGroups:
- name: vg-on-worker-0
- name: vg-on-worker-1
- name: vg-on-worker-2
type: Thick
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
EOF
Check that the created LocalStorageClass
has transitioned to the Created
phase by running the following command:
d8 k get lsc local-storage-class -w
In the output, you should see information about the created LocalStorageClass
:
NAME PHASE AGE
local-storage-class-thick Created 1h
Check that the corresponding StorageClass has been generated by running the following command:
d8 k get sc local-storage-class
In the output, you should see information about the generated StorageClass:
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
local-storage-class-thick local.csi.storage.deckhouse.io Delete WaitForFirstConsumer true 1h
Thin type StorageClass
The previously created LVMVolumeGroup
resources are suitable for creating Thick storage. If you require the ability to create Thin
storage, update the configuration of the LVMVolumeGroup
resources by adding a definition for a Thin pool:
d8 k patch lvg vg-on-worker-0 --type='json' -p='[
{
"op": "add",
"path": "/spec/thinPools",
"value": [
{
"name": "thin-pool-0",
"size": "70%"
}
]
}
]'
In the updated version of the LVMVolumeGroup
, 70% of the available space will be allocated for creating Thin storage. The remaining 30% can be used for Thick storage.
Repeat the addition of Thin pools for the remaining nodes (worker-1 and worker-2).
Example of creating a LocalStorageClass
resource with a Thin type:
d8 k apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: LocalStorageClass
metadata:
name: local-storage-class-thin
spec:
lvm:
lvmVolumeGroups:
- name: vg-on-worker-0
thin:
- name: thin-pool-0
- name: vg-on-worker-1
thin:
- name: thin-pool-0
- name: vg-on-worker-2
thin:
- name: thin-pool-0
type: Thin
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
EOF
Check that the created LocalStorageClass
has transitioned to the Created
phase by running the following command:
d8 k get lsc local-storage-class -w
In the output, you should see information about the created LocalStorageClass
:
NAME PHASE AGE
local-storage-class-thin Created 1h
Check that the corresponding StorageClass has been generated by running the following command:
d8 k get sc local-storage-class
In the output, you should see information about the generated StorageClass:
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
local-storage-class-thin local.csi.storage.deckhouse.io Delete WaitForFirstConsumer true 1h