Release Notes

v0.1.0

Release date: 2026-10-01

ElasticCluster and ElasticStorageClass are namespaced, so one Kubernetes cluster can run several Ceph clusters, one per namespace (a breaking change of both resources, existing ones are migrated automatically); OSDs can be encrypted with keys kept in Deckhouse Stronghold, and the module ships Ceph dashboards and alerts.

Highlights

Changes in this release:

• Several Ceph clusters in one Kubernetes cluster: ElasticCluster and ElasticStorageClass are namespaced, a namespace holds one ElasticCluster, and Rook deploys that cluster into the namespace of the ElasticCluster itself. Existing resources are moved into d8-sds-elastic on upgrade, and the running Ceph cluster is adopted unchanged.
• OSD encryption: spec.storage.encrypted wraps every OSD in dm-crypt (LUKS2), spec.storage.encryptionKeyStore keeps the keys in Deckhouse Stronghold instead of Secrets of the cluster, and spec.storage.keyRotation replaces them on a schedule.
• Monitoring out of the box: three Grafana dashboards (cluster, pools, single OSD) and twelve alerts on the health, the data and the capacity of every Ceph cluster the module runs, in any namespace.

New features

This release adds:

• ElasticCluster and ElasticStorageClass are namespaced. A namespace holds at most one ElasticCluster; a second one is rejected at admission with a message naming the first, so several Ceph clusters mean several namespaces. An ElasticStorageClass resolves spec.clusterRef in its own namespace only.
• Rook deploys a Ceph cluster into the namespace of its ElasticCluster. The controller mirrors into that namespace the ServiceAccounts, Roles, RoleBindings and the registry pull Secret the Ceph daemons need (Rook ClusterRoles are copied into Roles of the same name, which a Project’s namespace accepts), and removes them when the cluster is deleted; the namespace itself is never created or deleted by the module. The namespace has to carry security.deckhouse.io/pod-policy: privileged, which exempts everything else in it as well, so a Ceph cluster is best given a namespace of its own. The namespace of a Project is not relabelled: the cluster waits at NamespaceReady=False until the Project grants the profile (podSecurityProfile: Privileged, or the label in spec.parameters.namespace.labels). A namespace outside any Project is labelled by the controller, which records that in the sds-elastic.deckhouse.io/added-labels annotation and removes the label on teardown. A namespace deliberately labelled baseline or restricted is refused, not overridden. Direct writes to the Rook resources of a cluster (internal.sdselastic.deckhouse.io) are refused in every namespace, as they were in d8-sds-elastic.
• ElasticCluster publishes a NamespaceReady condition, ahead of StorageReady, that reports whether the namespace can hold the Ceph cluster and names what is missing.
• spec.storage.encrypted: true on an ElasticCluster wraps every OSD in dm-crypt (LUKS2), so the data and Ceph metadata on a disk that leaves the cluster stay unreadable. It does not protect a running cluster or the network traffic. The field is immutable: it can only be set when the cluster is created. Measured cost: about 9% of the throughput on small random writes and up to 26% on large sequential reads, and OSDs take about twice as long to start. The nodes need the dm_crypt kernel module.
• spec.storage.encryptionKeyStore chooses where the OSD key encryption keys live: KubernetesSecret (default, Secrets in the cluster’s namespace) or Stronghold (Deckhouse Stronghold or any store serving the Vault API over KV v2), configured by stronghold.address, kvPath, role, authMountPath (default kubernetes_local), caCertSecretName or insecureSkipTLSVerify. It requires encrypted: true and is immutable. The module does not prepare Stronghold; the KV mount, the policy and the role are created by its administrator. The key is read on every OSD start, so an unreachable or sealed store keeps restarted OSDs from coming up, while running ones keep serving.
• spec.storage.keyRotation (enabled, optional cron schedule, Rook’s weekly default when unset) has Rook replace the key encryption key of every OSD on a schedule, for both key stores. Data is not re-encrypted and OSDs keep serving. It requires encrypted: true and, unlike the two fields above, can be turned on, rescheduled or turned off on a running cluster.
• Three Grafana dashboards in the Storage folder: SDS-ELASTIC / CEPH CLUSTER, SDS-ELASTIC / CEPH POOLS and SDS-ELASTIC / CEPH OSD (per OSD, to find the slow disk). Per-daemon counters are scraped from ceph-exporter by a ServiceMonitor the module ships; cluster-wide metrics come from the mgr through the monitoring-custom module.
• Twelve alerts on every Ceph cluster of the module, each carrying the namespace of its ElasticCluster: D8SdsElasticCephClusterHealthError, D8SdsElasticCephClusterHealthWarning, D8SdsElasticCephMonOutOfQuorum, D8SdsElasticCephOSDDown, D8SdsElasticCephPlacementGroupsUnavailable, D8SdsElasticCephPlacementGroupsDegraded, D8SdsElasticCephPlacementGroupsInconsistent, D8SdsElasticCephObjectsUnfound, and the capacity alerts D8SdsElasticCephClusterFillingUp (75%), D8SdsElasticCephClusterAlmostFull (85%), D8SdsElasticCephOSDNearFull (one OSD over 85%) and D8SdsElasticCephPoolAlmostFull (one pool over 85%). They select the module’s own scrape jobs, so a Ceph cluster run by something else does not raise them, and none fires where the module is only installed.

Improvements

This release improves:

• CephClusterReady of an ElasticCluster reports the reason Stalled, with the time spent, once the storage backend has not become ready for 15 minutes, so a cluster that will never assemble is told apart from a slow one. The condition stays False in both cases.
• A teardown that outlives its expected duration reports TeardownStalled with the elapsed time on the Ready condition: after 2 minutes for both an ElasticStorageClass and an ElasticCluster. Reasons that already say what to do (BoundVolumesExist, DataPresentInPool, FilesystemNotEmpty, StorageClassesExist, VolumesExist) are kept. An ElasticStorageClass with force deletion whose pool still refuses to go says so.
• While csi-ceph keeps the CephClusterConnection of a cluster being deleted for volumes or snapshots, the ElasticCluster teardown reports VolumesExist, names up to ten remaining PersistentVolumes and tells to delete them, or to delete the remaining VolumeSnapshots, instead of Terminating and later TeardownStalled. It needs csi-ceph v0.5.37 or later, which reports the PersistentVolumesExist and VolumeSnapshotContentsExist reasons; with an older one nothing changes.
• The host directory of a cluster’s mon store is pinned in the new ElasticCluster.status.dataDirHostPath field. A new cluster gets /var/lib/deckhouse/sds/elastic/<namespace>/<name>; an existing cluster keeps the path it runs on (/opt/deckhouse/sds/elastic/<name>), with no migration. A change of the recorded value is rejected; clearing it makes the controller record it again from the running cluster.
• csi-ceph provisions and mounts volumes as a Ceph user of its own, client.sds-elastic-csi, which the controller creates in every cluster with an aes cephx key instead of using client.admin. Ceph 19.2.6 creates new keys as aes256k, which the kernel client (krbd and the CephFS kernel mount) understands only from Linux 7.0; with this user, kernel mounts keep working on older kernels.
• The AUTH_INSECURE_* health checks that Ceph 19.2.6 raises for aes keys do not take a cluster out of CephClusterReady, do not block the next Ceph upgrade and do not fire the health alerts. None of them affects a volume.
• On Deckhouse 1.78 and later the module’s access roles follow the capability scheme: d8:system-capability:sds-elastic:view and d8:system-capability:sds-elastic:edit grant ElasticCluster and ElasticStorageClass in all namespaces and ElasticClusterCredential. Older Deckhouse versions keep the d8:manage:permission:module:sds-elastic:* roles.

Fixes

This release fixes:

• An ElasticCluster no longer reports CephClusterReady and Ready while its storage serves nothing. Readiness followed only the phase of the Rook CephCluster, which stays Ready even when every OSD is crash-looping; now any HEALTH_ERR check, or inactive placement groups (PG_AVAILABILITY), hold the condition back and are named in its message.
• An ElasticCluster re-created under the name of a deleted one serves volumes again. The ElasticClusterCredential left by the previous cluster kept its old fsid and admin key, which made every PVC fail with Permission denied while the cluster reported Ready; such a credential is now dropped and rebuilt for the new cluster. The credential of a deleted cluster is still kept as long as no cluster of that name comes back.
• Controller metrics are collected again. The kube-rbac-proxy in front of them could not authenticate scrapes, so the target stayed at up == 0 with a permanent TargetDown.
• The Warning event with which Rook reports a failed reconcile of a CephCluster is no longer lost on a large cluster: a message over the 1024-character API limit was rejected and the event was never recorded. It is now shortened for the event, and the full text stays in the Rook operator log.
• The ElasticCluster teardown deletes the Ceph user of csi-ceph and the Rook CephCluster only after the CephClusterConnection is gone. Deleted in the same pass, they left the PersistentVolumes that csi-ceph still had to detach or delete with no way to do it, and the ElasticCluster stayed in deletion forever.
• Deleting an ElasticStorageClass no longer removes its pool while a volume still needs it. The guard counted only Bound PersistentVolumes, so right after the last PVC was deleted the pool went while the volume was still attached to a node or waiting for its Delete reclaim: the detach then failed for good with pool ... not found, the VolumeAttachment stayed, the PersistentVolume could not be deleted, and the ElasticCluster deletion that followed never finished. The pool is now kept, with the reason BoundVolumesExist, while a PersistentVolume of the StorageClass is Bound, attached to a node, or Released/Failed with the Delete reclaim policy; a Retain volume attached nowhere does not hold it.

Security updates

Security updates in this release:

• golang.org/x/crypto is upgraded to v0.56.0 in the Rook operator and to v0.57.0 in the module hooks, closing CVE-2026-56854, CVE-2026-56855 and CVE-2026-78662 reported by image scanners against v0.53.0.

Breaking changes

Changes that affect backward compatibility:

• ElasticCluster and ElasticStorageClass are namespaced resources. A manifest kept for re-applying needs metadata.namespace (d8-sds-elastic for resources that exist before the upgrade), and kubectl get and edit need -n <namespace> or -A. Only one ElasticCluster is accepted per namespace.
• The mgr metrics of the module’s Ceph clusters are scraped under job="custom-sds-elastic-ceph" instead of job="custom-ceph": the mgr Pods carry prometheus.deckhouse.io/custom-target: sds-elastic-ceph instead of ceph.

Upgrade notes

Before upgrading, note the following:

• The upgrade migrates existing ElasticCluster and ElasticStorageClass resources into the d8-sds-elastic namespace by itself: the objects are saved into the sds-elastic-crd-scope-migration-backup Secret of that namespace, the CustomResourceDefinitions are re-created namespaced (their scope cannot be changed in place), and the objects are restored with their status. The running Ceph cluster is adopted, not rebuilt: in d8-sds-elastic every object name, the host path and the connection to csi-ceph stay as they were. Saving the resources beforehand (d8 k get elasticclusters,elasticstorageclasses -o yaml) is still a good idea.
• For the duration of the migration the controller is stopped and the ElasticCluster and ElasticStorageClass admission webhooks are removed; the Helm release that follows in the same converge restores both.
• Add metadata.namespace: d8-sds-elastic to the manifests of existing ElasticCluster and ElasticStorageClass resources kept in Git or elsewhere. Applied without it, a manifest creates a new resource in the current namespace instead of updating the migrated one.
• An ElasticStorageClass name has to be unique across the whole cluster, not just within its namespace, because it becomes the name of a cluster-wide StorageClass; a name already taken in another namespace is rejected.
• Queries and dashboards of your own that read the Ceph mgr metrics by job="custom-ceph" have to switch to job="custom-sds-elastic-ceph".
• Existing clusters are rolled from Ceph 19.2.5 to 19.2.6 by the controller’s upgrade stage, with no action needed. csi-ceph switches from client.admin to client.sds-elastic-csi, which the controller creates in existing clusters too; volumes that are already mounted are not affected.
• From Ceph 19.2.6 on, ceph health detail shows AUTH_INSECURE_KEYS_ALLOWED, AUTH_INSECURE_KEYS_CREATABLE and AUTH_INSECURE_CLIENT_KEY_TYPE warnings on every cluster, and a cluster created before 19.2.6 also shows AUTH_INSECURE_SERVICE_KEY_TYPE and AUTH_INSECURE_SERVICE_TICKETS errors for its daemons’ aes keys. They are expected; the module’s conditions, upgrade gate and alerts leave them out.
• The controller’s RBAC is wider: it reads and labels namespaces, manages ServiceAccounts, Roles, RoleBindings, Secrets and ConfigMaps cluster-wide, may bind and escalate roles and reads ClusterRoles, because it furnishes the namespace of every Ceph cluster the way the chart furnishes d8-sds-elastic.

Known issues

Known limitations of this release:

• The Pods of a Ceph cluster outside d8-sds-elastic are not covered by the Deckhouse ban on kubectl exec into d8-* namespaces: anyone allowed to exec in that namespace can exec into its Ceph daemons.

Docs

Documentation changes:

• New section on namespaces and the number of Ceph clusters: one ElasticCluster per namespace, what is mirrored into it, the privileged Pod Security profile and how to grant it through a Project, and what happens to clusters created before the change.
• New sections on OSD encryption: what it covers, the measured cost per load profile, the Stronghold preparation (KV mount, a policy that also grants sys/mounts, which Rook reads before writing the first key, and the role), what happens when the key store is unavailable or sealed, and key rotation.
• The deletion procedure for an ElasticCluster now closes the dm-crypt mappings of an encrypted cluster before its LVM volumes are removed, and says that re-creating a cluster under the same name requires removing its dataDirHostPath on every storage node, while the OSD disks need no wiping. The table of teardown reasons describes VolumesExist and TeardownStalled.
• spec.storage.nodeSelector and spec.storage.blockDeviceSelector document that an empty selector {} selects every Node, control-plane Nodes included, and every consumable BlockDevice on them, and that disks appearing later join the cluster on their own.
• New section on monitoring and alerts: where the metrics come from, the dashboards, and when each alert fires.
• The Russian usage guide is rewritten in plain language, and both locales use the Ceph CLI of the Rook operator Pod instead of a Rook toolbox, which the module does not deploy.

Dependencies

Dependency updates:

• Rook: 1.19.5 → 1.20.7 (changelog)
• Ceph (squid): 19.2.5 → 19.2.6 (changelog)
  • The version the controller deploys. Existing clusters are upgraded automatically.
• Ceph (tentacle): 20.2.3 → 20.2.4 (changelog)
  • The image is built and shipped; the controller does not deploy it.

v0.0.5

Release date: 2026-09-02

ElasticCluster networks are described as lists of CIDRs (a breaking change of the resource schema, migrated automatically), an ElasticStorageClass can reserve a share of the filesystem for the superuser, and release notes carry sections instead of one flat list.

Highlights

Changes in this release:

• ElasticCluster.spec.network.public and .cluster accept several CIDRs, so a Ceph cluster whose storage nodes sit in more than one host subnet can be described. Both fields are lists now, which is a breaking change of the resource schema; existing resources are migrated on upgrade.
• An ElasticStorageClass can ask for a share of the filesystem to be kept for the superuser — the classic ext4 reserve — with the storage.deckhouse.io/ext4-reserved-percent annotation. Volumes are formatted with no reserve unless it is asked for.
• A release now has a summary and per-kind sections — new features, improvements, fixes, security, breaking changes, upgrade notes — on the module’s release-notes page and in ModuleRelease.spec.changelog, where the Deckhouse console renders it.

New features

This release adds:

• ElasticCluster.spec.network.public and .cluster take one or more CIDRs each, which is what a cluster whose storage nodes span several host subnets needs: every Ceph daemon binds to a host interface and has to own an address inside one of the given ranges.
• storage.deckhouse.io/ext4-reserved-percent on an ElasticStorageClass sets the percentage of a volume kept for the superuser: a whole number from 0 to 50, default 0. The module copies the resolved value onto the CephStorageClass it generates, so clearing the annotation returns the class to the default. It applies to volumes created afterwards — an existing filesystem keeps the reserve it was created with — and only to type: RBD, because a CephFS volume is a subdirectory of an existing filesystem and is never formatted. An invalid value leaves the ElasticStorageClass with its storage-class stage failing and the reason in its status, instead of breaking volume creation later.

Improvements

This release improves:

• An ElasticCluster whose spec.storage.nodeSelector selects nodes that the module setting settings.dataNodes.nodeSelector excludes now says so: the StorageReady condition names the nodes that carry no sds-node-configurator agent and the setting to widen, instead of reporting that no BlockDevice matches the selectors. A selector that matches no node at all is reported separately from an empty BlockDevice set.
• Release notes are written per locale in .release-notes/<tag>.yaml and <tag>.ru.yaml with the sections summary, highlights, new_features, improvements, fixes, security, breaking, upgrade_notes, known_issues, docs and dependencies. Both locales reach the cluster, so the console shows the notes in the language of the reader.
• The release history before this version keeps the previous flat format and renders on the same page exactly as before; nothing had to be rewritten.

Fixes

This release fixes:

• A release tag with no release notes used to publish an empty changelog.yaml, leaving ModuleRelease.spec.changelog blank with a green build. Such a tag now fails the build.

Breaking changes

Changes that affect backward compatibility:

• ElasticCluster.spec.network.public and .cluster are lists of CIDRs, not single strings. A manifest kept for re-applying has to be updated to the list form; resources that already exist are migrated on upgrade.

Upgrade notes

Before upgrading, note the following:

• An ElasticCluster created before this version is migrated automatically on upgrade: each of spec.network.public and .cluster becomes a one-element list holding the same address. No daemon is restarted and nothing has to be done by hand.

Docs

Documentation changes:

• The section on reserving part of a volume for the superuser is new, with the annotation, its range and what it does not cover.
• The documentation of settings.dataNodes.nodeSelector now states what the data-node label actually controls — whether the sds-node-configurator agent runs on a node, and therefore whether that node has any BlockDevice — how it relates to ElasticCluster.spec.storage.nodeSelector, and that the empty default covers control-plane nodes too.

v0.0.4

• Fix: caBundle for webhooks is now sourced from the webhooks-https-certs secret, and the module’s service accounts are excluded from validation — a broken webhook no longer leaves ElasticCluster without a status and Ceph undeployed
• Fix: a fresh cluster no longer enters HEALTH_WARN RECENT_MGR_MODULE_CRASH — mgr modules that cannot load (dashboard, restful, cephadm, diskprediction_local, rook) have been removed from the image
• Controller metrics are now published via kube-rbac-proxy and collected by a ServiceMonitor
• The module has been migrated to the common logger; the logging level is taken from the module settings, as documented
• Fix: rebuilding the same tag no longer leaves an outdated changelog in the release image
• Base images updated to v2.1.7, Go to 1.27.0, and lib-helm to 1.72.18
• Internal changes to the module build and e2e tests

v0.0.3

• Build changes

v0.0.2

• Added the ability to control Rook object-store CRs from the sds-object controller (webhooks)
• ElasticClusterCredential publishes status.conditions with the Ready condition; the status is generated through the common conditions library, status.conditions in the CRD is declared as a keyed list for the correct server-side apply
• Fix: VPA gate for the presence of a CRD, and not for the list of enabled modules
• Fix: The Ceph cluster no longer goes into HEALTH_WARN - the non-working mgr module restful has been disabled, a dependency has been added to load the telemetry mgr module
• Build Ceph images moved to container-base + pm; supported versions - 19.2.5 (squid) and 20.2.3 (tentacle)
• Documentation clarified: spec.network.public/cluster should be host subnets
• Update base images to v2.1.2, Go to 1.26.6 and lib-helm to 1.72.14
• Internal changes in module assembly

v0.0.1

• Initial release of the module
• Added module setting settings.dataNodes.nodeSelector. The controller places the label storage.deckhouse.io/sds-elastic-node="" on each Kubernetes node that matches the selector and removes the label from nodes that no longer match it. Downstream components (sds-node-configurator agent, ElasticCluster hosting) can use this label as nodeAffinity (pod-node affinity rule) to select data nodes.