Usage

Enabling Required Modules

Enable sds-elastic together with its companion modules:

• sds-node-configurator — owns the BlockDevice and LVMVolumeGroup CRDs that ElasticCluster selects from.
• csi-ceph — owns the CephClusterConnection and CephStorageClass CRDs the controller writes into.
• snapshot-controller — required for VolumeSnapshot support (optional).

d8 k apply -f - <<EOF
apiVersion: v1
kind: List
items:
  - apiVersion: deckhouse.io/v1alpha1
    kind: ModuleConfig
    metadata:
      name: sds-node-configurator
    spec:
      enabled: true
      version: 1
  - apiVersion: deckhouse.io/v1alpha1
    kind: ModuleConfig
    metadata:
      name: snapshot-controller
    spec:
      enabled: true
      version: 1
  - apiVersion: deckhouse.io/v1alpha1
    kind: ModuleConfig
    metadata:
      name: csi-ceph
    spec:
      enabled: true
      version: 1
  - apiVersion: deckhouse.io/v1alpha1
    kind: ModuleConfig
    metadata:
      name: sds-elastic
    spec:
      enabled: true
      version: 1
EOF

Wait until every module reaches the Ready state:

d8 k get module sds-node-configurator snapshot-controller csi-ceph sds-elastic -w

Selecting Data Nodes

settings.dataNodes.nodeSelector declares which Kubernetes Nodes are eligible to host sds-elastic data. The controller places the label storage.deckhouse.io/sds-elastic-node="" on every matching Node and removes it from Nodes that no longer match.

The label has one consumer: the sds-node-configurator agent, whose DaemonSet requires it (or the equivalent label of another SDS module) as a nodeAffinity term. A Node without it runs no agent, so it publishes no BlockDevice and cannot back an OSD.

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: sds-elastic
spec:
  enabled: true
  version: 1
  settings:
    dataNodes:
      nodeSelector:
        node-role.deckhouse.io/storage: ""

If the field is omitted, the empty selector matches every Node — every Node in the cluster gets storage.deckhouse.io/sds-elastic-node="", control-plane Nodes included, and the sds-node-configurator agent is scheduled on all of them. Set an explicit selector to keep it to the Nodes that carry storage.

ElasticCluster.spec.storage.nodeSelector (see below) is a separate, per-cluster choice and is not derived from this label: it picks which of the Nodes an ElasticCluster may use. Nothing reconciles the two, so keep the cluster’s selector inside the set of data nodes. A Node selected there but excluded here has no agent and no BlockDevice, which the cluster reports on its StorageReady condition:

$ d8 k -n d8-sds-elastic get elasticcluster ceph-prod -o jsonpath='{.status.conditions[?(@.type=="StorageReady")]}'
{"type":"StorageReady","status":"False","reason":"NodesAreNotDataNodes",
 "message":"none of the 2 Node(s) matched by storage.nodeSelector carries
 storage.deckhouse.io/sds-elastic-node, so the sds-node-configurator agent does not
 run there and no BlockDevice is discovered (worker-1, worker-2); widen
 settings.dataNodes.nodeSelector in the module configuration to cover them"}

Preparing Storage Nodes

ElasticCluster consumes BlockDevice CRs (managed by sds-node-configurator) selected by labels and provisions one OSD per matched device.

1. Pick the nodes that will host Ceph daemons and label them. The example uses node-role.deckhouse.io/storage:

   d8 k label node <node-name> node-role.deckhouse.io/storage=

2. Make sure each storage node has at least one unused raw block device (no partitions, filesystem, or LVM signatures). sds-node-configurator discovers them and creates a corresponding BlockDevice CR. Verify:

   d8 k get blockdevices.storage.deckhouse.io -o wide

3. Add a label that the ElasticCluster will use to select OSD-eligible devices. The example uses app=elastic-osd:

   d8 k label blockdevice <bd-name> app=elastic-osd

Namespaces and the number of Ceph clusters

ElasticCluster and ElasticStorageClass are namespaced, and the namespace is what separates one Ceph cluster from another:

• One ElasticCluster per namespace. Rook runs a single CephCluster in a namespace, and the module deploys one Ceph cluster per namespace, so a second ElasticCluster beside the first is rejected at admission with a message naming the one already there. Several Ceph clusters therefore means several namespaces.

• Everything the cluster needs goes into that same namespace. For an ElasticCluster in team-a the mons, mgrs, OSDs and pools land in team-a too, alongside the ServiceAccounts, Roles, RoleBindings and the copy of the registry pull Secret the controller mirrors there from d8-sds-elastic. All of it is removed when the cluster is deleted; the namespace itself is yours and stays.

• The namespace has to run under the privileged Pod Security profile. Ceph mons and OSDs run privileged, on host networking, with host paths, so the namespace needs security.deckhouse.io/pod-policy: privileged — which exempts everything else running in that namespace as well. Give a Ceph cluster a namespace of its own unless you are content with that trade.

  • The namespace of a Project (on a current Deckhouse, every namespace is one: multitenancy-manager adopts any namespace that belongs to no Project) is labelled by its Project, and the module does not change it. Until the Project grants the profile, the cluster stops at NamespaceReady=False saying so. Set podSecurityProfile: Privileged in the Project’s spec.parameters, or, for a template without that parameter such as simple, add the label under spec.parameters.namespace.labels:

    apiVersion: deckhouse.io/v1alpha2
    kind: Project
    metadata:
      name: team-a
    spec:
      projectTemplateName: simple
      parameters:
        namespace:
          labels:
            security.deckhouse.io/pod-policy: privileged

    The profile stays the Project’s when the cluster is deleted; take it back in the Project when it is no longer needed.

  • A namespace outside any Project is labelled by the controller itself. It records that it did (annotation sds-elastic.deckhouse.io/added-labels) and removes the label again when the cluster is deleted.

  • A namespace you have deliberately labelled security.deckhouse.io/pod-policy: baseline or restricted is never overridden: the cluster stops at NamespaceReady=False naming the label, and you move it to another namespace.

• An ElasticStorageClass lives in the namespace of the cluster it names. spec.clusterRef is resolved there and nowhere else. Its name, on the other hand, has to be unique across the whole cluster: it becomes the name of a StorageClass, and those are not namespaced. A name already taken in another namespace is rejected at admission.

• Clusters created before the switch to namespaced resources stay where they are. ElasticCluster used to be cluster-scoped; on upgrade such a resource is moved into the d8-sds-elastic namespace. Rook keeps running in that same namespace, with the same object names and the same host paths, so nothing about the running cluster changes.

Deploying an ElasticCluster

The example below bootstraps a Ceph cluster on every node carrying the node-role.deckhouse.io/storage label, consuming every BlockDevice labelled app=elastic-osd. It is created in d8-sds-elastic; put it in a namespace of your own to run a second Ceph cluster beside it.

d8 k apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: ElasticCluster
metadata:
  name: ceph-prod
  namespace: d8-sds-elastic
spec:
  storage:
    nodeSelector:
      matchExpressions:
        - { key: node-role.deckhouse.io/storage, operator: Exists }
    blockDeviceSelector:
      matchLabels:
        app: elastic-osd
  network:
    public:
      - 10.12.0.0/16
    cluster:
      - 10.12.0.0/16
EOF

Both public and cluster must be host subnets, not the Pod network. The module runs Ceph with network.provider: host, so every daemon (mon/mgr/osd) binds directly to a host interface and must own an IP inside one of the CIDRs you specify. Putting the Pod subnet into either field is invalid — the daemons have no interface in that range and will fail to bind. The two fields are not “where clients live” vs “where Ceph Pods live”; both are networks the Ceph daemons themselves sit on:

public — carries client traffic (Kubernetes Pods reaching mon/osd through csi-ceph) plus all inter-daemon traffic except OSD replication.

cluster — OSD-to-OSD backend only: replication, backfill, recovery and heartbeat.

Each field is a list, so you can pass several CIDRs when storage nodes span more than one host subnet:

network:
  public:
    - 10.12.0.0/16
    - 10.20.0.0/16
  cluster:
    - 10.12.0.0/16
    - 10.20.0.0/16

Use a dedicated storage NIC subnet for cluster if you have one; otherwise set cluster to the same CIDR(s) as public (as in the example above). Clients must be able to route to the public subnet(s).

spec.network stays immutable after creation. A cluster created before the fields became lists holds a single CIDR string in each of them; the module rewrites those into one-element lists on upgrade, keeping the addresses in use, so nothing has to be done by hand. Until that rewrite has happened the cluster is not reconciled, because the API server rejects every write to an object whose stored value no longer matches the schema.

Wait until the ElasticCluster reports Ready:

d8 k -n d8-sds-elastic get elasticcluster ceph-prod -w

The Phase column is expected to switch from Pending to InProgress and finally to Ready. The full per-stage progression is exposed through conditions: NamespaceReady → StorageReady → CephClusterReady → CredentialsReady → CsiCephReady → aggregate Ready. NamespaceReady covers the namespace Rook is deployed into; for a cluster in d8-sds-elastic there is nothing to prepare and it passes at once.

Check that the controller created the objects backing the OSDs — LVM volume groups, logical volumes and local PVs — and that the Ceph Pods are running:

d8 k get lvmvolumegroup -l sds-elastic.deckhouse.io/cluster=ceph-prod
d8 k get lvmlogicalvolume -l sds-elastic.deckhouse.io/cluster=ceph-prod
d8 k get pv -l sds-elastic.deckhouse.io/cluster=ceph-prod
d8 k -n d8-sds-elastic get pod -owide

The controller also creates an internal ElasticClusterCredential, a copy of the rook-ceph-mon Secret (the cluster fsid and the admin key). It is cluster-scoped; for a cluster in d8-sds-elastic it carries the cluster’s name, for a cluster in any other namespace it is named <namespace>-<cluster-name>:

d8 k get elasticclustercredential ceph-prod -o yaml

Encrypting OSDs

spec.storage.encrypted: true wraps every OSD of the cluster in dm-crypt (LUKS2). Rook hands each OSD block device to cryptsetup before ceph-volume takes it, so the OSD data and the Ceph metadata beside it are unreadable once the disk leaves the node:

spec:
  storage:
    encrypted: true
    nodeSelector:
      matchExpressions:
        - { key: node-role.deckhouse.io/storage, operator: Exists }
    blockDeviceSelector:
      matchLabels:
        app: elastic-osd

What it does and does not cover:

• Covers a disk that leaves the cluster: pulled, replaced under warranty, or decommissioned. Both the object data and the Ceph metadata on it stay ciphertext.
• Does not cover a running cluster. Anything that can read through a live OSD — the OSD process, a node with root access, a client with Ceph credentials — sees plaintext. Encryption of the objects themselves is a different layer and belongs to the client or to an object-storage module.
• Does not cover traffic. Ceph’s on-the-wire encryption (network.connections.encryption) is a separate setting and is off in this module.

Operational constraints:

• The field is immutable. Rook sets up dm-crypt while creating an OSD, so it cannot be turned on later — the API server rejects a write that changes the flag on an existing ElasticCluster. Encrypting an existing cluster means draining and re-creating every OSD, which is not automated. Decide at cluster creation.
• Keys live in the same cluster as the data. Rook stores a key encryption key per OSD in a Secret of the cluster’s namespace (d8-sds-elastic in the examples). Losing those Secrets makes the OSDs unrecoverable, so they belong in whatever backup covers cluster state. To keep them outside the cluster instead, see the Stronghold section below.
• The cost on the data path is measurable. Every read and write of every OSD passes through dm-crypt: roughly a tenth of the throughput on small random I/O and up to a quarter on large sequential I/O, see “What encryption costs” below.
• OSDs take longer to start. Every OSD opens its own LUKS volume at startup, which makes a node reboot and an update more expensive than they are without encryption.
• The nodes need dm_crypt. The kernel module is loaded on demand by device-mapper on the distributions the module supports; on a stripped or custom kernel, verify it is available before creating the cluster, otherwise OSD preparation fails.

What encryption costs

The price of dm-crypt is proportional to bytes, not to operations: the larger the block, the more data passes through encryption per operation and the larger its share of the operation’s time. Hence the spread across load profiles — from nine percent to a quarter:

Average latencies grow in proportion to the throughput lost, and the tail of write latency grows more than the average: p99 in the mixed profile rises from 45 ms to 60 ms. Read p99 barely moves.

Per unit of data transferred, more CPU is needed: 15% more on small random writes and 26% more on large sequential ones. Total CPU consumption under load may not rise at all, though — a load with a fixed queue depth is bound by latency, so encryption does not so much add CPU as cut throughput within the same budget.

OSDs take about twice as long to come up: restarting every OSD took 63 seconds against 35 without encryption. Deploying a cluster from scratch changes little — 219 seconds against 208 — because monitors, mgr and Ceph version detection dominate it, and none of them care about encryption. What matters is the OSD startup time: it is paid on every node reboot and every update.

The numbers come from a four-node stand with 4 cores and 8 GiB per node (Ceph 19.2.5, one OSD per node, three-way replication, fio against a raw RBD block volume with --direct=1) and carry over to other hardware only as an order of magnitude. The stand’s nodes ran at about a third of their CPU capacity: where nodes have more CPU headroom the drop will be smaller, and on nodes where Ceph already saturates the CPU it will be larger. Measure on your own hardware before enabling encryption on a cluster that is close to its ceiling.

Keeping the keys outside the cluster (Deckhouse Stronghold)

By default Rook writes a key encryption key per OSD into a Secret of the cluster’s namespace. spec.storage.encryptionKeyStore moves them into Deckhouse Stronghold — or any store serving the same Vault API — over its KV v2 engine:

spec:
  storage:
    encrypted: true
    encryptionKeyStore:
      type: Stronghold
      stronghold:
        address: https://stronghold.example.com
        kvPath: sds-elastic
        role: sds-elastic-osd
        caCertSecretName: stronghold-ca   # Secret in the cluster's namespace, key `cert`

Prepare Stronghold first — the module creates nothing inside it:

# a KV v2 mount for the keys
d8 stronghold secrets enable -path=sds-elastic kv-v2

# a policy over that mount, plus the mount listing Rook probes before writing
d8 stronghold policy write sds-elastic-osd-kek - <<'POLICY'
path "sds-elastic/*" {
  capabilities = ["create", "read", "update", "delete", "list"]
}
path "sys/mounts" {
  capabilities = ["read", "list"]
}
path "sys/mounts/sds-elastic" {
  capabilities = ["read"]
}
POLICY

# a role bound to the service accounts Rook uses for KMS access
d8 stronghold write auth/kubernetes_local/role/sds-elastic-osd \
  bound_service_account_names=rook-ceph-osd,rook-ceph-system \
  bound_service_account_namespaces=d8-sds-elastic \
  policies=sds-elastic-osd-kek ttl=10m

The example is for a cluster in d8-sds-elastic. For a cluster in another namespace, append that namespace to bound_service_account_namespaces (for example, d8-sds-elastic,team-a) and create the CA Secret in that namespace.

kubernetes_local is the path where Deckhouse Stronghold mounts the Kubernetes auth method of its own cluster; authMountPath defaults to it and only needs setting for a different store or a remote cluster.

The sys/mounts rules are not optional. Rook’s Vault client reads the mount list to work out the KV engine version before it writes the first key, and a policy limited to the KV path answers 403 permission denied — which reaches you as failed to start ceph osds: failed to init vault kms and says nothing about policies. The cost of those rules is that the OSD token can enumerate the store’s mount paths (not their contents).

Two properties of this setup decide whether it is safe to use:

• The key is read on every OSD start, not once at creation. Stronghold has to be reachable and unsealed whenever an OSD starts, and the address has to resolve from the storage nodes themselves — Ceph daemons run with host networking.
• Stronghold must not be backed by the ElasticCluster it protects. Its storageClass setting accepts any StorageClass, including one served by this very Ceph; that combination deadlocks a cold start, because the OSDs wait for Stronghold and Stronghold waits for its volume on those OSDs. There is no automatic way out of that deadlock: no timeout resolves it, and it takes manual intervention — see “When the key store is unavailable” below. Keep Stronghold on its default local storage on the control-plane nodes, or on storage this cluster does not serve.

insecureSkipTLSVerify: true exists for test clusters with a self-signed certificate and no CA to hand over; it is mutually exclusive with caCertSecretName, and anything holding real data should ship the CA.

The store, like encrypted, is immutable: Rook writes the key to the store it was told about while creating the OSD and never migrates keys between stores.

Verify that the OSDs really came up encrypted — every OSD device shows a crypt layer on the storage node:

d8 k -n d8-sds-elastic get pod -l app=rook-ceph-osd
lsblk -o NAME,TYPE,MOUNTPOINT   # on a storage node: expect TYPE=crypt above the LVM volume

When the key store is unavailable

The key is read on every OSD start, so an unavailable key store breaks not the cluster’s work but the start of its daemons. The practical consequence: while nothing restarts, an unreachable Stronghold goes unnoticed; the trouble shows up on a node reboot, a module update, and any other event that brings OSD pods up again.

What happens, step by step:

• Already running OSDs are untouched. Their volumes are open and they keep serving requests for as long as the store stays unavailable. No data is lost or damaged.
• A restarted OSD does not come up. The pod sits in Init:CrashLoopBackOff, failing on the encryption-kms-get-kek init container. The reason is in that container’s log: it could not obtain a token, or could not fetch the key.
• A new cluster stops at the OSD stage. Monitors and mgr come up normally — they need no keys — while OSD creation does not go through. ElasticCluster stays in InProgress without naming the reason; look for it in the Rook operator’s log (d8 k -n d8-sds-elastic logs deploy/sds-elastic | grep vault), where failed to store secret. failed to init vault kms repeats.
• Recovery is automatic. Once the store is back, the cluster comes up on its own with no intervention: about three and a half minutes to restart the OSDs that were down, and about a minute to finish a cluster that had stalled during creation. The timing is set by the pod restart interval, not by the module.

Two cases worth knowing in advance:

• A sealed store is indistinguishable from an absent one. The OSDs get 503 Vault is sealed and behave exactly as they do against a completely unreachable store. A restarted Stronghold pod comes up on its own, but unsealing does not, so automatic unsealing after a restart is as mandatory as the store’s availability itself.
• A circular dependency needs a manual way out. If Stronghold’s volume is served by the cluster it protects, stopping both at once does not resolve itself. The Stronghold pod cannot even finish: unmounting its volume waits on dead OSDs, the pod stays in Terminating forever, and a replacement pod gets Multi-Attach error because the volume is formally still held by the previous one. The way out is to force-delete the wedged pod (--grace-period=0 --force) and move Stronghold onto storage this cluster does not serve.

Rotating the keys

spec.storage.keyRotation asks Rook to replace the key encryption key of every OSD on a schedule. It works with either key store — Secrets of the cluster’s namespace or Stronghold:

spec:
  storage:
    encrypted: true
    keyRotation:
      enabled: true
      schedule: "0 3 * * 0"   # cron, or a descriptor such as @weekly; unset means weekly

What is replaced is the key that unlocks the volume, not the key the data is encrypted with. Nothing is re-encrypted, the OSDs keep serving, and the rotation costs one write per OSD in the key store.

Unlike encrypted and encryptionKeyStore, this one is mutable: rotation is an operational policy, so it can be turned on, moved to another schedule, or turned off on a running cluster. Turning it off is a value the module writes through, not an omission — an ElasticCluster with enabled: false leaves the CephCluster with rotation explicitly disabled.

Rotation needs the key store reachable at the moment it runs. A rotation that cannot reach Stronghold fails and is retried at the next scheduled run; the OSDs already running are unaffected, because they hold their volume open.

BlockDevice Ownership

Once an ElasticCluster selects a BlockDevice for the first time, the controller patches it with the sds-elastic.deckhouse.io/cluster=<cluster-name> label. The label is the durable record of which cluster owns the device and drives several behaviors:

• Single owner per BlockDevice. If a BlockDevice matches the blockDeviceSelector of two ElasticCluster resources, the second one cannot adopt it. The controller refuses to overwrite the existing label and surfaces StorageReady=False with Reason=OwnershipConflict and a message listing each contested BD and its current owner. No LVMVolumeGroup, LVMLogicalVolume, or local PersistentVolume is created until every conflict is resolved — even free BDs in the selector remain unadopted while a conflict is pending.

  To resolve a conflict, decide which cluster should own the BD. If it should move to the other cluster, clear the current owner’s label from it:

  d8 k label blockdevice <bd-name> sds-elastic.deckhouse.io/cluster-

  Or remove the conflicting ElasticCluster entirely. The next reconcile picks the BD up.

• An adopted BlockDevice stays with the cluster. Once a BD has been labelled by the controller, it remains part of the cluster’s working set even if it later drifts out of blockDeviceSelector or nodeSelector (for example, the operator narrows the selector, the device’s labels change, or its node is relabelled). This is intentional: the OSD on top of it is already provisioned, the local PV is bound to a specific node, and dropping it from the working set would shrink CephCluster.spec.storageClassDeviceSets[0].count and risk data unavailability. The cluster’s OSD count is therefore monotonic for the lifetime of an ElasticCluster — it can grow when new BDs match the selector but never shrinks on its own.

  As a side effect, sds-node-configurator flips BlockDevice.status.consumable to false once a VG appears on the device. Because adoption is kept, this does not drop the BD from the working set on the next reconcile.

• Empty selectors — the whole cluster. Both selectors follow the standard LabelSelector semantics, where an empty selector matches everything. blockDeviceSelector: {} therefore stands for every consumable BlockDevice on the selected nodes, and nodeSelector: {} for every Node of the cluster, control-plane Nodes included — the pair is how a cluster is handed every free disk it can reach. Both fields are required, so the key cannot be left out: “everything” is written as an explicit {}.

  spec:
    storage:
      nodeSelector: {}
      blockDeviceSelector: {}

  The selectors are re-evaluated on every reconcile rather than frozen at creation, so the growth needs no further input: a disk plugged into a matching node, or a new Node joining the cluster, produces a BlockDevice that already matches {} and is adopted on the next pass. Only Nodes where the sds-node-configurator agent runs publish BlockDevice resources at all (see Selecting Data Nodes above), and that is what bounds the set in practice. Read together with the previous point, an empty pair means the working set grows on its own and never shrinks on its own — the same monotonic OSD count, reached from the widest selector pair there is.

• Releasing a BlockDevice. While the module is experimental there is no automatic way to release a BD. Deleting the ElasticCluster does NOT cascade to the objects created per device: the controller only removes the Rook CephCluster and the csi-ceph CephClusterConnection, leaving the LVMVolumeGroup / LVMLogicalVolume / local PersistentVolume and the BD label for you to clean up by label (see Deleting Resources below). To retire a single BD from a live cluster, manually delete the corresponding LVMLogicalVolume and LVMVolumeGroup, and only then clear the label:

  d8 k delete lvmlogicalvolume <name>
  d8 k delete lvmvolumegroup <name>
  d8 k label blockdevice <bd-name> sds-elastic.deckhouse.io/cluster-

  Doing this while pools still hold useful data risks losing replicas.

• Editing the selectors after creation. ElasticCluster.spec.storage.nodeSelector and spec.storage.blockDeviceSelector are editable after creation (d8 k -n <namespace> edit elasticcluster <name>). The validating webhook on UPDATE enforces two safety rails:

  • Keeping adopted BDs. If an edit would leave an already-adopted BD outside the new selector pair (its labels no longer match blockDeviceSelector, or its status.nodeName is no longer in the set produced by nodeSelector), the webhook rejects the request and lists the offending BDs. Adopted BDs cannot be released automatically — follow the manual procedure above first.
  • Conflict detection. If a widening edit would pull in a BD already labelled by another ElasticCluster, the webhook rejects the request and reports the contested BDs along with their current owners. Resolve the conflict (clear the label, or delete the other EC) before retrying.

  spec.network remains immutable on UPDATE: changing the public/cluster CIDRs on a live cluster invalidates mon endpoints and host-network bindings, and there is no safe automatic remediation. To change the network configuration, delete and re-create the ElasticCluster.

Declaring StorageClasses

Pools and the matching csi-ceph StorageClasses are declared per ElasticStorageClass. One ESC produces one Ceph pool + one CephStorageClass named after the ESC.

Create it in the namespace of the ElasticCluster it names, and give it a name no other namespace uses — see Namespaces and the number of Ceph clusters.

RBD pool with default replication (3 replicas)

d8 k apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: ElasticStorageClass
metadata:
  name: ceph-prod-rbd
  namespace: d8-sds-elastic
spec:
  clusterRef: ceph-prod
  type: RBD
  replication: ConsistencyAndAvailability
EOF

CephFS pool with default replication (3 replicas)

d8 k apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: ElasticStorageClass
metadata:
  name: ceph-prod-cephfs
  namespace: d8-sds-elastic
spec:
  clusterRef: ceph-prod
  type: CephFS
  replication: ConsistencyAndAvailability
EOF

The ErasureCodedCompact replication mode is temporarily disabled and cannot be selected.

Pool that survives two simultaneous host failures (HighRedundancy)

d8 k apply -f - <<EOF
apiVersion: storage.deckhouse.io/v1alpha1
kind: ElasticStorageClass
metadata:
  name: ceph-prod-rbd-hr
  namespace: d8-sds-elastic
spec:
  clusterRef: ceph-prod
  type: RBD
  replication: HighRedundancy
EOF

HighRedundancy produces a 4-replica pool (size=4, min_size=2, requireSafeReplicaSize=true):

• two simultaneous host failures keep I/O continuous (2 replicas equal min_size);
• a third simultaneous failure pauses I/O but does not lose data — Ceph backfills the surviving copy onto free cluster space and resumes;
• data loss only at the fourth simultaneous failure.

The mode requires at least 5 storage nodes (4 for the pool’s CRUSH placement at failureDomain=host and 5 to host a 5-mon quorum). The first time you create a HighRedundancy ESC against an ElasticCluster, the controller automatically promotes the underlying CephCluster to mon.count=5, mgr.count=3 (the standard topology is 3, 2). The promotion is sticky: deleting the last HighRedundancy ESC does NOT roll the counts back, because silently weakening a live cluster’s fault-tolerance guarantee is unsafe.

A validating webhook gates ESC creation on the same thresholds so the sticky promotion cannot fire on an undersized cluster. CREATE of an ESC with replication: HighRedundancy is rejected when:

• the parent ElasticCluster referenced by spec.clusterRef does not exist;
• fewer than 5 nodes match ElasticCluster.spec.storage.nodeSelector (the 5-mon quorum floor);
• adopted BlockDevice resources of the parent EC live on fewer than 4 distinct nodes (the 4-replica CRUSH placement floor).

So the creation order is fixed: create the ElasticCluster first, wait until at least four storage nodes have adopted BDs (check with d8 k get bd -l sds-elastic.deckhouse.io/cluster=<cluster-name> or ElasticCluster.status.phase=Ready), and only then create the HighRedundancy ESC. Creating both in one d8 k apply fails: the cluster is created first, but it has no adopted BDs yet when the ESC is validated.

The audit trail lives on ElasticCluster.status.cephTopology:

d8 k -n d8-sds-elastic get elasticcluster ceph-prod -o jsonpath='{.status.cephTopology}'
# {"monCount":5,"mgrCount":3,"reason":"HighRedundancyESCPresent","lastPromotedAt":"2026-…"}

Possible reason values: Standard, HighRedundancyESCPresent, StickyHighWaterMark. To force a recompute (for example, after deliberately scaling down to a smaller cluster), clear the field via the status subresource and trigger a reconcile:

d8 k -n d8-sds-elastic patch elasticcluster ceph-prod \
  --type=merge --subresource=status \
  -p '{"status":{"cephTopology":null}}'

Wait until each ESC reports Ready:

d8 k -n d8-sds-elastic get elasticstorageclass -w

The conditions transition is PoolReady → CsiStorageClassReady → aggregate Ready.

Verify the resulting csi-ceph objects and Kubernetes StorageClasses:

d8 k get cephclusterconnection
d8 k get cephstorageclass
d8 k get sc

A CephClusterConnection named after the parent ElasticCluster (ceph-prod) and one CephStorageClass per ElasticStorageClass (ceph-prod-rbd, ceph-prod-cephfs) are expected. Each csi-ceph CephStorageClass produces a Kubernetes StorageClass with the same name, ready to be consumed by PersistentVolumeClaim resources.

The internal helm-managed StorageClass sds-elastic-osd (provisioner kubernetes.io/no-provisioner, volumeBindingMode: WaitForFirstConsumer) backs OSD-local PersistentVolumes and is intentionally not user-facing — ElasticStorageClass resources cannot reuse this name (the validating webhook rejects them).

How much of a volume is reserved for the superuser?

Nothing, by default: a volume is formatted with -m0, so the whole volume is available to the workload.

Where the classic 5% ext4 reserve is wanted — for example to keep a filesystem writable for a privileged process after a workload has filled it — annotate the ElasticStorageClass with the percentage to reserve:

d8 k -n <namespace> annotate elasticstorageclass <name> storage.deckhouse.io/ext4-reserved-percent=5

Volumes of this module are formatted by csi-ceph, so the module copies the annotation onto the CephStorageClass it generates; the rest is csi-ceph’s doing. Clearing the annotation returns the class to the default, since the resolved value is written on every reconcile.

The value is a whole number of percent between 0 and 50; an invalid one leaves the ElasticStorageClass with its storage-class stage failing and the reason in its status, instead of breaking volume creation later.

The reserve applies only to volumes created after the annotation was set — filesystems that already exist keep the reserve they were created with. It is meaningful for type: RBD only: CephFS volumes are subdirectories of an existing filesystem and are never formatted.

Deleting Resources

Deleting an ElasticCluster

Deleting an ElasticCluster is reversible as long as no ElasticStorageClass still references it: the controller removes only the Rook CephCluster and the csi-ceph CephClusterConnection. The OSD disks and the mon store are left intact, so no data goes anywhere — but the cluster cannot be re-created on the same devices until the surviving mon store is removed, see step 4.

Order of operations:

1. Delete every dependent ElasticStorageClass first (see below). The controller refuses to start the cluster teardown while any ESC references it.

2. Note the host directory the cluster keeps its mon store in — step 4 needs it, and it is gone from the API together with the cluster — then delete the ElasticCluster:

   d8 k -n d8-sds-elastic get elasticcluster ceph-prod -o jsonpath='{.status.dataDirHostPath}{"\n"}'
   # /var/lib/deckhouse/sds/elastic/d8-sds-elastic/ceph-prod
   d8 k -n d8-sds-elastic delete elasticcluster ceph-prod

   Held by a finalizer, the controller deletes the CephCluster and CephClusterConnection, then releases the CR.

3. Clean up the remaining controller-labelled objects by hand — they are intentionally preserved (no automatic cascade).

   On an encrypted cluster, close the dm-crypt mappings first. Rook opens each OSD volume with cryptsetup luksOpen in an init container of the OSD Pod and never closes it, so deleting the ElasticCluster leaves the dm-crypt mapping on the node. While it is open, LVM refuses to remove the logical volume underneath — lvremove fails with Logical volume ... is used by another device — and the LVMVolumeGroup stays in Terminating indefinitely. Close the mappings on every storage node:

   # on each storage node, as root
   dmsetup ls --target crypt | awk '{print $1}' | grep -- '-block-dmcrypt$' | while read -r dm; do
     cryptsetup luksClose "$dm"
   done

   The names look like ceph-<fsid>-<pvc>-block-dmcrypt; the suffix is what tells an OSD mapping apart from any other crypt device on the node. Closing one does not touch the data — it only removes the mapping, and the volume can be opened again with the key while the key still exists.

   Then delete the objects:

   # inspect what is still labelled with the cluster name
   d8 k get pv,lvmlogicalvolume,lvmvolumegroup -l sds-elastic.deckhouse.io/cluster=ceph-prod
   
   d8 k delete pv -l sds-elastic.deckhouse.io/cluster=ceph-prod
   d8 k delete lvmlogicalvolume -l sds-elastic.deckhouse.io/cluster=ceph-prod
   d8 k delete lvmvolumegroup -l sds-elastic.deckhouse.io/cluster=ceph-prod
   # finally clear the cluster label from the BlockDevices
   d8 k label blockdevice -l sds-elastic.deckhouse.io/cluster=ceph-prod sds-elastic.deckhouse.io/cluster-

   Keep the ElasticClusterCredential if you plan to re-create the cluster with the same identity.

4. If you are going to create a cluster again under the same name, remove its dataDirHostPath on every storage node. That is the path noted in step 2. A cluster created by the current module version has /var/lib/deckhouse/sds/elastic/<namespace>/<cluster-name>, the module namespace included. A cluster created by an earlier version stays on the path it was created with, which the module never moves: /opt/deckhouse/sds/elastic/<cluster-name> in d8-sds-elastic, /opt/deckhouse/sds/elastic/<namespace>/<cluster-name> in any other namespace:

   # on every storage node, as root
   rm -rf /var/lib/deckhouse/sds/elastic/d8-sds-elastic/ceph-prod
   # a cluster created by an earlier module version
   rm -rf /opt/deckhouse/sds/elastic/ceph-prod

   The controller does not remove that directory: it holds the mon store, and keeping it is the reversibility that makes deletion erase nothing. A new cluster of the same name, however, comes up with a new fsid while its monitors start on the old store and look for the previous cluster’s monitors, which no longer exist. The cluster then never converges, and the monitor’s log says:

   mon.a@0(probing) e3 handle_auth_request failed to assign global_id

   The OSD disks do not need wiping: dropping the LVMVolumeGroup takes the whole LVM layer with it, BlueStore data included, and sds-node-configurator reports the BlockDevices free again (consumable: true) within seconds. Nor does the ownership label have to go if the name is the same — the new cluster reads its own name there and adopts the disks without a conflict.

While the teardown is in progress the ElasticCluster Ready condition explains what is blocking it:

Deleting an ElasticStorageClass

Delete an ElasticStorageClass to remove the corresponding pool and CephStorageClass:

d8 k -n d8-sds-elastic delete elasticstorageclass ceph-prod-rbd

Held by a finalizer, the controller runs an ordered teardown:

1. It refuses to delete anything while any PersistentVolume provisioned from this StorageClass is still Bound. Delete the consuming PersistentVolumeClaims first — this guard cannot be overridden.
2. Once nothing is bound, it removes the CephStorageClass and tears down the backing pool / filesystem.

For block (RBD) classes, a pool that still holds data is preserved by default. To permanently delete it (the data in the pool is lost), authorise the destructive purge with the force-deletion annotation:

d8 k -n d8-sds-elastic annotate elasticstorageclass ceph-prod-rbd sds-elastic.deckhouse.io/force-deletion=true

For shared-filesystem (CephFS) classes there is no force override: the filesystem is removed automatically once it is empty, which you achieve by deleting the remaining PersistentVolumes for the StorageClass.

While the teardown is in progress the ElasticStorageClass Ready condition explains what is blocking it:

Disabling the Module

A validating webhook on the sds-elastic ModuleConfig rejects setting spec.enabled: false while any ElasticCluster still exists. This prevents accidentally tearing down the controller and the Rook operator while a live Ceph cluster (OSD data on host disks) is still under management. Follow the ordered teardown below; the disable is accepted only once the last ElasticCluster is gone.

1. Delete every ElasticStorageClass and wait until the controller has removed the pools and csi-ceph StorageClasses:

   d8 k get elasticstorageclasses.storage.deckhouse.io -A

   Wait until the command returns No resources found.

2. Delete every ElasticCluster and wait for cluster teardown:

   d8 k get elasticclusters.storage.deckhouse.io -A

   Wait until the command returns No resources found.

3. Optionally remove the ElasticClusterCredential. It is a cluster-scoped identity backup and does not gate the disable (only a live ElasticCluster blocks it). Delete it unless you plan to re-create the cluster with the same identity:

   d8 k get elasticclustercredentials.storage.deckhouse.io
   d8 k delete elasticclustercredential <name>

4. Disable the module. Disabling requires the modules.deckhouse.io/allow-disabling: "true" annotation on the ModuleConfig:

   d8 k annotate moduleconfig sds-elastic modules.deckhouse.io/allow-disabling=true --overwrite
   d8 k patch moduleconfig sds-elastic --type=merge -p '{"spec":{"enabled":false}}'

Forcing the Module Off While ElasticClusters Remain

If you must disable the module without deleting the ElasticClusters first, set the sds-elastic.deckhouse.io/force-disable: "true" annotation on the ModuleConfig. With this annotation present, the webhook allows spec.enabled: false regardless of how many ElasticClusters exist:

d8 k annotate moduleconfig sds-elastic sds-elastic.deckhouse.io/force-disable=true --overwrite
d8 k annotate moduleconfig sds-elastic modules.deckhouse.io/allow-disabling=true --overwrite
d8 k patch moduleconfig sds-elastic --type=merge -p '{"spec":{"enabled":false}}'

Checking Cluster Health

The controller exposes coarse-grained progress on each CR through conditions. For an ElasticCluster:

d8 k -n <namespace> describe elasticcluster <cluster-name>

Useful conditions: NamespaceReady, StorageReady, CephClusterReady, CredentialsReady, CsiCephReady, UpgradeReady, UpgradeInProgress, and the aggregate Ready.

The UPGRADING column (shown by d8 k get elasticcluster -o wide) reflects the UpgradeInProgress condition. It is derived from the list of daemon versions Rook publishes in CephCluster.status.ceph.versions.overall. While the list holds more than one version the upgrade is still in progress, and UPGRADING stays True for the whole time Rook rolls through mon → mgr → OSD → MDS, including while CephCluster.status.phase=Progressing and the later stages of cluster processing wait for the upgrade to finish. UpgradeInProgress returns to False only once versions.overall holds a single version matching the target. During an upgrade the Ceph column (status.cephVersion.running) shows the oldest running version — usually that of the OSDs, which are upgraded last — not the target version Rook has already recorded.

For an ElasticStorageClass:

d8 k -n <namespace> describe elasticstorageclass <esc-name>

Useful conditions: PoolReady, CsiStorageClassReady, and the aggregate Ready.

Rook publishes the state of Ceph itself (health, reported problems, capacity) in the status of the CephCluster resource. It lives in the cluster’s namespace and has the same name as the ElasticCluster:

d8 k -n d8-sds-elastic get cephclusters.internal.sdselastic.deckhouse.io ceph-prod -o jsonpath='{.status.ceph}'

The ceph and rados CLIs run in the Rook operator Pod: it is built on the Ceph image and keeps the configuration and admin keyring of every cluster under /var/lib/rook/<namespace>/. The alert descriptions use the same commands:

d8 k -n d8-sds-elastic exec deploy/sds-elastic -c operator -- ceph --conf /var/lib/rook/d8-sds-elastic/d8-sds-elastic.config status
d8 k -n d8-sds-elastic exec deploy/sds-elastic -c operator -- ceph --conf /var/lib/rook/d8-sds-elastic/d8-sds-elastic.config osd tree

For a cluster in another namespace, put its name into the path of the configuration; the operator Pod is always in d8-sds-elastic.

Monitoring and Alerts

Two sources feed the module’s monitoring, and they cover different things:

• The active mgr publishes cluster-wide state — health, monitors, OSD up/in, placement groups, pools, capacity — on port 9283. It reaches Prometheus through the prometheus.deckhouse.io/custom-target: sds-elastic-ceph label the module puts on the mgr pods, under job="custom-sds-elastic-ceph", which needs the monitoring-custom module enabled (it is part of the default bundle).
• ceph-exporter publishes per-daemon counters — IOPS, throughput, operation latencies, per-OSD space — on port 9926. Automatic discovery skips that port because it is not named http-metrics, so the module ships its own ServiceMonitor for it. It looks for the rook-ceph-exporter Service labelled module: sds-elastic in every namespace, under job="sds-elastic-ceph-exporter".

Both are there for every ElasticCluster, whatever its namespace; the namespace label says which cluster a series belongs to.

Three dashboards land in the Storage folder in Grafana: SDS-ELASTIC / CEPH CLUSTER, SDS-ELASTIC / CEPH POOLS and SDS-ELASTIC / CEPH OSD. The last one is per-OSD: use it when the cluster dashboard shows the fleet average of apply and commit latency rising and the question is which disk is behind it.

The following alerts are configured on those metrics. They select the series by job, so a Ceph cluster someone else runs in the same Kubernetes cluster does not raise them. Each alert is about one cluster and carries the namespace of its ElasticCluster. None of them fires where the module is merely installed — a cluster without an ElasticCluster publishes none of these series.

State of the cluster:

The two health alerts leave out the AUTH_INSECURE_* checks, and so does the CephClusterReady condition of an ElasticCluster. From Ceph 19.2.6 on, Ceph calls cephx keys of the aes type insecure and reports them for as long as they exist, and this module relies on such a key: the Ceph user csi-ceph provisions and mounts with, client.sds-elastic-csi, has one, because the kernel client — krbd and the CephFS kernel mount — understands the newer aes256k type only from Linux 7.0. client.admin and the daemons of a cluster created on 19.2.6 or later keep aes256k keys. So ceph health detail shows AUTH_INSECURE_KEYS_ALLOWED, AUTH_INSECURE_KEYS_CREATABLE and AUTH_INSECURE_CLIENT_KEY_TYPE as warnings on every cluster; a cluster created before 19.2.6, whose daemons have aes keys, also shows AUTH_INSECURE_SERVICE_KEY_TYPE and AUTH_INSECURE_SERVICE_TICKETS as errors. None of them affects a volume.

State of the data:

Capacity, asked at the three levels that run out independently:

Capacity is measured raw, so replication is counted: a pool with size: 3 spends three raw bytes for every byte an application writes. Adding capacity means giving the cluster more disks — free or label BlockDevices on the nodes the ElasticCluster selects, and the controller provisions OSDs on them.