Usage

Enabling the module

d8 k apply -f - <<EOF
apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: sds-object
spec:
  enabled: true
  version: 1
EOF

Creating a store

A store is the data plane, and there is one Kind per backend. A SeaweedFS store on an existing StorageClass:

apiVersion: storage.deckhouse.io/v1alpha1
kind: SeaweedFSStore
metadata:
  name: default
spec:
  masters: 3
  volumeServers: 3
  replication: "001"        # one extra copy on another volume server
  storage:
    sizePerNode: 50Gi
    class: localpath

The counts and the replication code are SeaweedFS’s own settings, and they have to agree: the third digit of the code is how many copies land on other servers, so volumeServers must be at least one more than that or writes never complete. The admission webhook rejects a pair that cannot work rather than letting the store come up and stall.

More than one filer needs a shared metadata store — the default LevelDB lives on the filer’s own volume and cannot be shared:

spec:
  filers: 3
  metadataStore: Postgres   # requires the managed-postgres module

Creating a class

An ObjectStore is the class tenants name. It carries the reference to the store and the defaults buckets inherit:

apiVersion: storage.deckhouse.io/v1alpha1
kind: ObjectStore
metadata:
  name: standard
spec:
  storeRef:
    kind: SeaweedFSStore
    name: default
  reclaimPolicy: Retain
  quota:
    maxSize: 100Gi          # ceiling per bucket; a Bucket asking for more is rejected

Track readiness:

d8 k get seaweedfsstore
# NAME      VOLUMES   REPLICATION   PHASE   ENDPOINT                                     READY   AGE
# default   3         001           Ready   http://default-seaweedfs.d8-sds-object...    True    3m

d8 k get objectstore
# NAME       STORE-KIND       STORE     RECLAIM   PHASE   READY   AGE
# standard   SeaweedFSStore   default   Retain    Ready   True    2m

A class is Ready exactly when its store resolves and is itself Ready. An unknown storeRef.kind is rejected at admission: the set of store Kinds lives in the controller’s driver registry, not in the schema, so adding a backend does not mean editing the CRD every class is validated by.

Requesting a bucket

A Bucket is what you create to get a bucket. The controller provisions a cluster-scoped BucketContents for it — named after the Bucket, owned by it and private to the namespace — resolves the class to a store, and creates the bucket there:

apiVersion: storage.deckhouse.io/v1alpha1
kind: Bucket
metadata:
  name: app-data
  namespace: my-app
spec:
  objectStoreRef: standard
  accessPolicy: Private
  reclaimPolicy: Retain     # omit to take the class default

d8 k -n my-app get bucket app-data
# NAME       CONTENTS              PHASE   READY   AGE
# app-data   contents-1f2e3d-...   Ready   True    20s

The BucketContents it creates is visible cluster-wide (d8 k get bktc) but is not meant to be written by hand: the admission webhook admits it only from the module’s own service account, because contents without an owning Bucket are storage no namespace can reach and nobody is accountable for.

A bucket that is already there

The backend bucket name is derived from the BucketContents, so a bucket can already exist under that name for reasons that have nothing to do with the module: one made by hand, a leftover from a store that was rebuilt, or another cluster pointed at the same backend.

The module marks every bucket it creates with two tags:

storage.deckhouse.io/owned-by          sds-object
storage.deckhouse.io/bucket-contents   contents-1a2b3c4d5e-team-a-data

A bucket that carries neither is not taken over. The BucketContents stays not ready, BucketReady reports the reason BucketNotOwnedByModule, and nothing in the bucket is touched:

BucketReady   False   BucketNotOwnedByModule
  bucket "team-a-data" already exists in the backend and is not managed by this
  module (it carries no ownership tag); remove it or point this Bucket at
  another store

The decision is yours to make — remove that bucket, or point the Bucket at another store — so the controller reports it instead of retrying. Adopting an existing bucket was a feature once and was removed together with BucketClaimPolicy; what is left under that name is an accident, and handing a tenant credentials to somebody else’s data is not a recoverable one.

Buckets created before the module started marking them carry no tag either. Those are adopted and marked on the next reconcile, recognised by the module’s own record of having provisioned them (status.bucketName), so an upgrade does not turn every existing bucket into a refusal.

Requesting credentials

Each workload declares a BucketAccess referencing a Bound Bucket in its namespace. The controller mints a dedicated access key / secret key scoped to that bucket and writes a Secret (named <access>-s3-credentials by default) in the same namespace:

apiVersion: storage.deckhouse.io/v1alpha1
kind: BucketAccess
metadata:
  name: app-data
  namespace: my-app
spec:
  bucketRef: app-data
  permission: ReadWrite   # or ReadOnly

d8 k -n my-app get bucketaccess app-data
# NAME       BUCKET     PHASE   SECRET                    READY   AGE
# app-data   app-data   Ready   app-data-s3-credentials   True    20s

Consuming the credentials

The credentials Secret holds the standard S3 connection variables, ready to be mounted with envFrom:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: my-app
spec:
  template:
    spec:
      containers:
        - name: app
          image: my-app:latest
          envFrom:
            - secretRef:
                name: app-data-s3-credentials

Publishing a store outside the cluster

By default a store is reachable in-cluster only: status.endpoint carries just internal, and that is the address a Secret gets. To use a bucket from outside, publish the store through the alb module — a Gateway API implementation on top of Envoy.

What an administrator has to prepare before publishing:

1. a Gateway. The alb module creates it from an ALBInstance (a team’s own Gateway in its namespace) or a ClusterALBInstance (a shared one in the controller’s namespace). Take its name and namespace from that object’s status;
2. a DNS record for the hostname the endpoint will answer on;
3. a kubernetes.io/tls Secret with a certificate for that hostname, in the module namespace (d8-sds-object). The module issues no certificates: point this at the Secret cert-manager produces, or put your own there.

Then add a publish block to the store:

apiVersion: storage.deckhouse.io/v1alpha1
kind: SeaweedFSStore
metadata:
  name: default
spec:
  storage:
    class: linstor-thin-r2
  publish:
    hostname: s3.example.com
    gatewayRef:
      name: public-gw      # the Gateway the administrator created
      namespace: d8-alb
    tls:
      secretRef:
        name: s3-example-com-tls

The module creates a ListenerSet (the hostname, port 443, TLS terminated with that certificate) and an HTTPRoute to the store’s S3 port, after which status.endpoint.external carries the URL:

d8 k get seaweedfsstore default -o jsonpath='{.status.endpoint}'
# {"external":"https://s3.example.com","internal":"http://default-seaweedfs...:8333","region":"us-east-1"}

TLS is mandatory and cannot be turned off: S3 credentials travel in the Authorization header, so publishing over plain HTTP would hand the bucket to anyone who can see the traffic.

A note on ports: the module publishes a hostname, not a port — the listener is always 443, as the alb guide requires. If the ALBInstance or ClusterALBInstance uses a HostPort inlet on a non-standard port (8443, say), status.endpoint.external still reads https://<host> while a client on that stand needs https://<host>:8443. That is a property of the inlet rather than of the address: mapping a name to a port is the administrator’s business (a production LoadBalancer inlet listens on 443 and the address from the status is used as it is).

Choosing the address in the credentials

A Secret carries the in-cluster address by default — most consumers run in the same cluster, and there is no reason to send their traffic through an external load balancer. The external address is requested explicitly:

apiVersion: storage.deckhouse.io/v1alpha1
kind: BucketAccess
metadata:
  name: backup-writer
  namespace: my-app
spec:
  bucketRef: app-data
  permission: ReadWrite
  endpointScope: External

If the store is not published, the BucketAccess does not quietly get the in-cluster address: it stays not ready, and its condition says that the store needs spec.publish. A Secret holding an address that is unreachable from where it is about to be used is worse than no Secret at all.

When an endpoint moves — publishing switched on, the hostname changed — the Secrets of every affected BucketAccess are re-issued automatically.

Addressing limits

Path-style addressing is supported: https://s3.example.com/<bucket>/<key>. It needs one DNS name and one certificate.

Virtual-hosted addressing (https://<bucket>.s3.example.com/<key>) exists in the API as spec.publish.addressing: VirtualHosted but is currently refused by the webhook: it needs wildcard DNS and a wildcard certificate (issuable over DNS-01 only), the backends’ S3 gateways support it unevenly, and none has been measured against it yet. Clients that can only do virtual-hosted addressing will not work with a published endpoint — a deliberate limit of the first implementation rather than a defect.

What stays inside

Only the S3 port is exposed. For SeaweedFS a separate single-port Service is created for that: the store’s main Service also publishes the filer’s HTTP and gRPC APIs, and a route to it would carry metadata read and write out of the cluster without passing through S3 authorization.

Ceph RGW has no port split — its admin ops API is served on the same port as S3, under /admin/. Publishing an SDSElasticStore therefore publishes that path too, and it is worth being precise about what protects it.

The users this module issues hold no admin capabilities, so a tenant’s key cannot reach the admin API. But Rook creates an admin-capable user of its own on every object store — rgw-admin-ops-user, with capabilities buckets=*;users=* — and its keys live in a Secret in the sds-elastic namespace. What stands between the internet and full bucket and user administration is that credential, not the route.

Weigh that before publishing a Ceph store:

• if the platform team can deny /admin on the Gateway they own, do that;
• otherwise consider a separate CephObjectStore (its own RGW) for the public endpoint, so the admin API of the store your tenants use is not reachable from outside at all;
• and treat the rgw-admin-ops-user Secret as an internet-facing credential — because once the store is published, it is one.

A SeaweedFS store has no equivalent exposure: the published Service carries the S3 port only, and the filer’s own APIs are on ports that are not routed.

When publishing does not work

Publication state is a condition of its own, PublishedEndpointReady, and it does not take the store out of Ready: a broken external route stops neither creating buckets, nor deleting them, nor issuing keys.

d8 k get seaweedfsstore default -o jsonpath='{range .status.conditions[?(@.type=="PublishedEndpointReady")]}{.reason}: {.message}{end}'

Common reasons:

Public read

accessPolicy: PublicRead on a Bucket lets anyone fetch its objects without credentials. Both backends enforce it:

apiVersion: storage.deckhouse.io/v1alpha1
kind: Bucket
metadata:
  name: site-assets
  namespace: my-app
spec:
  objectStoreRef: standard
  accessPolicy: PublicRead

curl -s https://s3.example.com/my-app-site-assets/logo.png -o logo.png

What it grants is reading objects, and nothing else:

• Objects only, no listing. Anonymous callers can GetObject if they know the key; ListObjects on the bucket stays denied. A public listing turns the bucket into an index of everything in it, which is a separate decision — ask for it and it will be designed as one, rather than arriving as a side effect of “these files are public”.
• Reads only. Anonymous writes, deletes and tagging are denied on both backends.
• This bucket only. The grant names the bucket, so other buckets in the same store are unaffected.

Two things worth knowing before you use it:

• It only means something where the store is reachable. Inside the cluster the S3 endpoint answers on the module’s own Service; to serve the objects to the internet the store has to be published (spec.publish on the store), and then the whole endpoint is reachable — public bucket or not. Publishing is what exposes the endpoint; accessPolicy is what decides who may read through it.
• A Released bucket stays public. Deleting the Bucket under reclaimPolicy: Retain keeps the data and the BucketContents, including its accessPolicy — that is what lets the same Bucket, recreated, pick the data back up unchanged. To take public read away, set accessPolicy: Private on the BucketContents or delete it.

Going back to Private revokes it on the next reconcile, and does not disturb the access keys issued for the bucket.

Versioning and object lock

versioning: Enabled keeps every version of an object instead of overwriting it. objectLock goes further and makes the bucket write-once-read-many: a version under retention cannot be deleted — not by the tenant, not by the storage administrator, and not by this module:

apiVersion: storage.deckhouse.io/v1alpha1
kind: Bucket
metadata:
  name: audit-log
  namespace: my-app
spec:
  objectStoreRef: standard
  versioning: Enabled
  objectLock:
    mode: Compliance     # Governance | Compliance
    days: 365
  reclaimPolicy: Retain  # required with objectLock

Governance can be bypassed by a caller holding the bypass permission; Compliance cannot be bypassed by anyone until the retention expires, and an object’s retention can be extended but never shortened. Both are enforced by the backend, not by the module.

What the module reports back is what the backend has, not what was asked for:

d8 k get bktc -o custom-columns='NAME:.metadata.name,VERSIONING:.status.versioning,LOCK:.status.objectLock.enabled,MODE:.status.objectLock.mode'

Rules the API enforces

• objectLock requires versioning: Enabled. Object lock is built on versions.
• objectLock is immutable, and can only be set when the Bucket is created. Ceph RGW accepts object lock only in the bucket-creation call, so a bucket that exists without it can never be given one. The field is fixed on both backends so that the same manifest means the same thing on either — protecting an existing bucket means creating a new one and copying the data across.
• objectLock cannot be combined with reclaimPolicy: Delete, including when Delete is what the class hands out by default. A bucket holding protected objects cannot be removed, so that pair would leave the BucketContents in Terminating until the last retention expired. The request is refused rather than quietly switched to Retain.
• Versioning cannot be suspended once the bucket is locked. Both backends refuse it, and so does the API.

Legal hold

A legal hold blocks deletion of a version for as long as it is set, with no expiry. It is not configured here: it is an operation a client performs on an object, with the credentials it already has:

mc legalhold set myalias/my-app-audit-log/report.pdf
mc legalhold clear myalias/my-app-audit-log/report.pdf

While a hold is set, nothing can remove that version, and waiting does not help — only clearing it does.

Letting protected data go

Deleting the Bucket under reclaimPolicy: Retain leaves the data and the BucketContents behind in phase Released, retention intact. Deleting the BucketContents is the documented way to let retained data go — but on a locked bucket the backend refuses while anything in it is still protected. The object then keeps its finalizer and says why:

d8 k get bktc contents-... -o jsonpath='{.status.conditions[?(@.type=="Ready")].message}'
# the backend refuses to remove the bucket while its objects are protected; ...

The deletion resumes on its own once the last retention expires. It can be finished sooner only by removing the protected versions with credentials that may bypass the retention, which Compliance does not allow at all.

Rotating credentials

To rotate the access key of an BucketAccess, set or change the storage.deckhouse.io/rotate annotation. The controller issues a fresh key pair, updates the Secret, and revokes the previous key:

d8 k -n my-app annotate bucketaccess app-data \
  storage.deckhouse.io/rotate="$(date +%s)" --overwrite

Metadata in a database you already run

metadataStore: External points the filers at a PostgreSQL this module does not run. The capability is the same as Postgres — several filers sharing one database — and the ownership is not: its availability, its backups and its upgrades are yours. A store whose metadata database is down serves nothing, and no amount of replication in the store makes up for a metadata server that is not there.

apiVersion: v1
kind: Secret
metadata:
  name: media-metadata-db
  namespace: d8-sds-object
stringData:
  host: pg.example.internal
  port: "5432"
  database: seaweedfs_media
  username: seaweedfs
  password: <password>
  sslmode: verify-full
  ca.crt: |
    -----BEGIN CERTIFICATE-----
    ...
---
apiVersion: storage.deckhouse.io/v1alpha1
kind: SeaweedFSStore
metadata:
  name: media
spec:
  metadataStore: External
  externalMetadataStore:
    secretRef:
      name: media-metadata-db
  filers: 3
  storage:
    class: linstor-r2

What the database has to provide. PostgreSQL 12 or newer, an empty database, and a role that may create tables in it — the filer creates one table per bucket on first use and never migrates a schema, so nothing has to be prepared beyond the database itself. Give it its own database rather than a shared one: the table names come from bucket names, and a bucket called users would meet whatever else is in there.

TLS. sslmode defaults to require, which encrypts without checking who answers. Supply ca.crt and set sslmode: verify-full to check as well — the bundle is mounted into the filers and the connection is verified against it. sslmode: disable is refused: the password and every object name cross that connection. Any value libpq does not know is refused too, with the accepted list in the message — a misspelt mode passes the reachability check, because TCP is fine, and then leaves the filers crash-looping on a typo.

Changing the connection. Edit the Secret — move the database, rotate the password, replace the CA — and the filers restart to pick it up. They read the connection once, at startup, so a change that did not restart them would be a change that had not happened yet. Expect the S3 endpoint to be briefly unavailable on a single-filer store; a multi-replica set rolls one pod at a time.

Where the Pods run

spec.placement on the store carries a nodeSelector and tolerations, and they are applied to every Pod the module creates itself: masters, volume servers and filers.

Replicas of one component are kept on different nodes. For masters and volume servers this is a hard rule: three volume servers on one node survive no node failure, and a replication code promising copies “on other servers” would have placed them all in one failure domain — so a replica that cannot be given a node of its own stays Pending, where the unmet promise is visible. Filers prefer to spread and still run when they cannot: they front metadata that lives elsewhere, so losing several at once costs availability while they reschedule, not data.

The managed metadata database is placed differently, and spec.placement does not reach it. The Postgres resource this module creates has no scheduling fields at all — tolerations, nodeSelector and nodeAffinity belong to the PostgresClass. To put the database on particular nodes, create a PostgresClass that says so and name it in spec.postgresClassName; empty means the class called default.

spec:
  metadataStore: Postgres
  postgresClassName: storage-dedicated
  placement:
    nodeSelector:
      node-role/storage: ""
    tolerations:
      - key: storage.deckhouse.io/dedicated
        operator: Exists
        effect: NoSchedule

What the module will not do. It does not create the database or the role, does not run migrations, does not back anything up and does not watch the server. What it does check, before pointing the filers at it, is that something is listening — a host that does not resolve or a port nothing answers on is reported on the store:

BackendReady   False   Pending
  the external metadata database at pg.example.internal:5432 did not answer:
  dial tcp: lookup pg.example.internal: no such host

Losing that database loses the store: the objects are still on the volume servers, and nothing knows which bucket or which name any of them belongs to. Back it up like the rest of the store.

How much space is left

Every store reports its space in status.capacity, and kubectl shows the headline figures in the wide output:

d8 k get seaweedfsstore media -o wide

NAME    VOLUMES   REPLICATION   PHASE   ENDPOINT                     USED%   CAPACITY   READY
media   3         001           Ready   http://...svc:8333           37.24   300Gi      True

status:
  capacity:
    total: 300Gi
    used: 111Gi
    available: 189Gi
    usedPercent: "37.24"
    lastUpdated: "2026-08-30T09:12:04Z"

What the numbers are made of depends on the backend, and the difference matters.

• SeaweedFSStore — the disks under the store’s own volume servers, summed. This is the store’s space: the module provisioned those PVCs and nothing else writes to them. A volume server that does not answer the probe is left out of the sum rather than counted as empty, so the total shrinks while one is unreachable — better than a used fraction that quietly falls because a node went missing.
• SDSElasticStore — the raw, cluster-wide capacity of the Ceph cluster behind it, as the cluster itself reports it. It is not the store’s share and not what a client can write: replication divides raw capacity by the pool’s size, and the same cluster serves every other pool on it. Read it as “how much room does the cluster this store lives on have left”, not as a quota.

status.capacity is absent, rather than zero, while nothing has answered — a store that has not been measured says so instead of reporting itself full.

Metrics and alerts

The same numbers are exported per store, labelled store_kind and store:

A store that was not measured publishes no capacity series at all, so an alert on “nearly full” cannot fire on a store nobody probed. Readiness is the opposite: it is always published, because 0 and “no series” have to mean different things — otherwise uninstalling the module would read as an outage.

Three alerts come with them: D8SdsObjectStoreNotReady (not ready for half an hour — rollouts and PVC provisioning take minutes, this is for the store that did not come back), D8SdsObjectStoreFillingUp (over 85% for an hour) and D8SdsObjectStoreAlmostFull (over 95%). The dashboard is SDS Object — Stores.

Worth knowing where the second threshold comes from: on SeaweedFS the master stops placing new volumes on a disk past 90%, so a store can refuse a new bucket while still reporting Ready.

Data integrity

Every store reports what is known about its data being intact — when it was last checked, by whom, and what was found:

d8 k get seaweedfsstore,sdselasticstore \
  -o custom-columns='NAME:.metadata.name,INTEGRITY:.status.conditions[?(@.type=="IntegrityHealthy")].status,LAST SCRUB:.status.integrity.lastScrubTime,DAMAGED:.status.integrity.damaged'

IntegrityHealthy is Unknown until the first check completes — “we have not looked” and “we looked and it is fine” are different answers, and only one of them is the store’s own doing. A finding makes it False and puts the backend’s own words in the message, which is what names the disk to suspect.

It never makes the store not Ready. A store with one damaged volume keeps serving every other object, and taking those away would not repair anything.

Who checks, and when

• SeaweedFS — the module does. The engine verifies a checksum on every read and has a scrub that verifies every stored object, but it never runs that scrub on its own, so cold data would never be checked. Nothing rots faster than data nobody reads.

  apiVersion: storage.deckhouse.io/v1alpha1
  kind: SeaweedFSStore
  metadata:
    name: media
  spec:
    integrity:
      interval: 24h     # default 168h; values under 1h are raised to 1h
      mode: Full        # Full (default) reads every byte; Index only checks indexes
      enabled: true
    # ...

  Full is the default because Index cannot find a rotted object body — and that is the thing worth looking for. The interval default is deliberately conservative: a full scrub reads every stored byte.

  Because it reads every stored byte, the scrub runs in the background, not inside a reconcile. While a pass is under way IntegrityHealthy says so (ScrubInProgress), and the result lands in the status when the pass finishes — which on a large store is minutes or hours after it started. Restarting the controller cancels a pass in flight; the schedule simply starts another.

  A volume server that could not be asked is not counted as damage. It appears as status.integrity.unreachable, and a pass with unreachable nodes and no findings leaves the condition Unknown with reason ScrubIncomplete rather than True: part of the store went unchecked, and that is a different claim from “checked and clean”.

  A pass that found no volumes at all — a store that has not been written to yet — is not recorded as a check. The condition stays Unknown with reason NothingToCheck and status.integrity is not stamped, so the store remains due and is checked as soon as it holds something. Publishing that pass would say “checked and clean” over no data and buy a whole interval of silence over the first data written afterwards.

• Ceph RGW — the engine does, on its own schedule, and the module reports what it found by reading the ElasticCluster health checks (OSD_SCRUB_ERRORS, PG_DAMAGED, PG_NOT_DEEP_SCRUBBED, …). There is no spec.integrity on SDSElasticStore: the module never starts a deep-scrub and never turns on automatic repair, because that is configuration of a Ceph cluster it does not own.

status.integrity.source says which of the two produced the reported result.

Enough copies?

A missing copy is not damage, so it is reported separately — by the RedundancyHealthy condition and status.redundancy:

d8 k get seaweedfsstore -o custom-columns='NAME:.metadata.name,WANTED:.status.redundancy.copiesWanted,VOLUMES:.status.redundancy.volumes,SHORT:.status.redundancy.underReplicated'

The accounting comes from the master’s own topology, so it is refreshed on every reconcile rather than on the scrub schedule — it costs nothing to ask.

RedundancyHealthy going False while IntegrityHealthy stays True says exactly what it should: the data is intact, and there is less of a safety net under it than was asked for. A store with replication: "000" asks for one copy and has one, so it is never reported short.

Nothing puts the copy back on its own. SeaweedFS 4.39 has an admin server with a maintenance framework and workers that execute its tasks, and the protocol even carries a replication task message — but the tasks it ships are balance, vacuum, erasure coding, EC balance, S3 lifecycle and Iceberg. There is no replication task, so a volume that lost a replica stays short until somebody restores it:

# from a pod with the seaweedfs image
weed shell -master=<store>-seaweedfs-master:9333 <<'EOF'
lock
volume.fix.replication -apply
unlock
EOF

On SDSElasticStore the condition is Unknown with reason NotAccountedHere: how many copies Ceph keeps is the cluster’s own accounting, reported through its health, and a second opinion from this module would be exactly that.

What the module does with a finding

By default it reports it and stops there. With spec.integrity.autoRepair: true it also puts it right — on SeaweedFS, where a rotted copy is replaced by pulling a fresh one from a copy that is intact:

spec:
  replication: "001"      # more than one copy, or there is nothing to pull from
  integrity:
    autoRepair: true

It is off by default because it is the most destructive thing the module does: the repair deletes a copy of live data on an automatic decision. Two conditions have to hold before it touches anything, and both are checked per volume, per pass:

• the store keeps more than one copy. On replication: "000" the damaged copy is the only copy, and deleting it is deleting the data;
• another copy of that volume scrubbed clean in the same pass. “This copy is bad” is not “that one is good”, and a volume server that did not answer the scrub has told us nothing.

When either fails, the damage is reported, the copy is left alone, and status.integrity.details says which condition stopped it. The same applies when the copy back fails: the fresh copy is pulled onto the node the damaged one was deleted from, and if that does not complete the volume is one copy short — which RedundancyHealthy then reports, and which is a better state than a copy whose contents are wrong.

Repaired volumes are counted in status.integrity.repaired and stop counting as damaged, so a store that fixed itself does not keep alerting about it.

On Ceph nothing here applies: the cluster repairs its own placement groups, and this module neither starts a deep-scrub nor turns on osd_scrub_auto_repair.

Metrics and alerts

The findings above are also exported as metrics, one series per store, labelled store_kind and store:

Three alerts come with them, and the third is the one worth having:

• D8SdsObjectStoreDataDamaged — a scrub found damage. Fires immediately: the number comes from a check that already finished.
• D8SdsObjectStoreUnderReplicated — fewer copies than asked for, after 15 minutes so a rolling restart does not page anyone.
• D8SdsObjectStoreEncryptionKeyWithheld — the key in the spec is not the key the data was written with, and the module is refusing to apply it. The highest severity of the four precisely because nothing is failing yet: the running gateway still holds the working key, and the breakage is scheduled for its next restart.
• D8SdsObjectStoreNotScrubbed — the store has not been checked in three of its own intervals. This is the quiet failure the whole feature exists for: the IntegrityHealthy condition keeps carrying the verdict of the last completed scrub, so a store whose scrub stopped running reads as healthy indefinitely. The threshold follows spec.integrity.interval, so a daily store and a weekly one are each judged at their own pace, and a store with the scrub switched off publishes no interval and is never overdue.

A Grafana dashboard — SDS Object — Data integrity, folder Storage — puts the same series on one page: how many stores are damaged, short of copies or overdue, a table with a row per store, and the sawtooth of scrub age against each store’s own alerting threshold.

Encryption at rest

Without spec.encryption, objects are written to the volume servers as they arrive: anyone holding the disk reads them. Turning it on takes two steps and no change to the applications writing the data.

# 32 bytes of key material. Keep a copy somewhere that is not this cluster.
d8 k -n d8-sds-object create secret generic media-sse \
  --from-literal=kek="$(head -c 32 /dev/urandom | xxd -p -c 32)"

apiVersion: storage.deckhouse.io/v1alpha1
kind: SeaweedFSStore
metadata:
  name: media
spec:
  encryption:
    mode: ServerManaged
    keySecretRef:
      name: media-sse

The Secret may instead carry key with a passphrase, from which the key is derived — useful when the key material comes from somewhere that hands out strings rather than bytes.

From then on the module hands the S3 gateway the wrapping key and marks every bucket it manages so that objects written without any encryption header are encrypted anyway. Each object gets its own data key, wrapped with the key from the Secret. The wrapping key itself is never written to the filer’s metadata store, so holding that store — the Postgres database or the filer PVC — is not enough to read anything.

$ d8 k get swfsstore media -o jsonpath='{.status.encryption}' | jq
{
  "mode": "ServerManaged",
  "keyFingerprint": "sha256:9c1f4a0b7e2d8536",
  "since": "2026-08-25T09:12:04Z"
}

Do not lose the key, and do not change it

Nothing re-wraps existing objects. A gateway started with a different wrapping key answers every read of previously encrypted data with an internal server error — not with anything naming the key, and not with a hint that the key is the problem. The module therefore fingerprints the key and, when the fingerprint changes, refuses to apply the new one:

$ d8 k get swfsstore media -o jsonpath='{.status.conditions[?(@.type=="EncryptionActive")]}' | jq -r .message
the wrapping key changed (sha256:9c1f4a0b7e2d8536 -> sha256:22ba07f6c4e1d980) and was NOT applied: ...

While that condition is False the gateway is left exactly as it is running, on the key that can still read the data — so the fix is to put the old key material back. Only if the objects written under the previous key are genuinely expendable:

d8 k annotate swfsstore media \
  storage.deckhouse.io/encryption-key-change-acknowledged=sha256:22ba07f6c4e1d980

Note that the danger is latent without this guard: the gateway reads the key at startup, so rewriting the Secret breaks nothing today — it breaks everything at the next pod restart, which will happen weeks later for an unrelated reason.

spec.encryption.mode also cannot be turned back off. Switching it off would only stop encrypting new writes while every object already written still needed the key, so a store that says “disabled” would have to keep the key forever. The way back is a new store and a copy.

What is not encrypted, and what the client can do itself

• Object names, sizes and tags are not encrypted — only object contents.
• Objects written before encryption was turned on stay as they were. Nothing rewrites them; re-uploading them is the only way to encrypt them.
• A client that wants to hold its own key can (SSE-C) — pass the key with each request and the store never sees it. That needs no configuration here and works regardless of spec.encryption. Note that on the in-cluster HTTP endpoint the key travels unencrypted in a header, so SSE-C belongs on the published TLS endpoint.
• SSE-KMS is not available on SeaweedFS stores. The gateway reads KMS configuration only from a static credentials file, which is incompatible with the per-BucketAccess keys this module mints. SDSElasticStore (Ceph RGW) is where an external KMS belongs.

On Ceph RGW stores

The same field on an SDSElasticStore needs an external secrets store, and that is not a matter of taste: RGW’s server-managed encryption keeps its keys in a KMS and has no mode where a key can simply be handed to the daemon. On this platform that store is Deckhouse Stronghold, whose transit engine is exactly what RGW asks for.

# a token that may read and use the transit key, and nothing else
d8 k -n d8-sds-object create secret generic rgw-stronghold-token \
  --from-literal=token="$STRONGHOLD_TOKEN"

apiVersion: storage.deckhouse.io/v1alpha1
kind: SDSElasticStore
metadata:
  name: heavy
spec:
  elasticClusterRef: main
  encryption:
    mode: ServerManaged
    stronghold:
      address: https://stronghold.d8-stronghold.svc.cluster.local:8200
      tokenSecretRef:
        name: rgw-stronghold-token
      # for a Stronghold with a private certificate:
      # caSecretRef:
      #   name: stronghold-ca

Four things follow from how Ceph does this, and each is visible in the API:

• Stronghold is named, the Vault API is spoken. What reaches Rook is KMS_PROVIDER: vault and VAULT_ADDR, because Stronghold keeps the Vault API and RGW speaks it. Any other Vault-compatible store works for the same reason; it is simply not what this page walks you through.
• The transit mount path is not configurable. For SSE-S3 Rook builds the RGW prefix from the engine name alone (/v1/transit), so a transit mounted elsewhere would be accepted here and never reached. Mount it at transit.
• The token is copied into the sds-elastic namespace. Rook resolves the token Secret next to the CephObjectStore, not next to the store object, so the module keeps a copy there owned by the store. Deleting the store takes the copy with it. Rotating the token is a write to your Secret; the copy follows on the next reconcile. A CA bundle is copied the same way and is renamed on the way — you write ca.crt, Rook projects cert.
• There is no key fingerprint in status.encryption, and that is not an omission. The key never passes through the module — it lives in Stronghold — so there is nothing here to fingerprint, and no guard against changing it: what the data depends on is the key in Stronghold, not the token that reads it. Keeping that is Stronghold’s job, and losing it loses the objects just as surely.

status.encryption.message and the EncryptionActive condition report the configuration not reaching RGW — a missing token Secret, most often. While that is the case the store keeps reconciling everything else, and a store that has never been created is not created unencrypted: objects written before the Secret appears could never be encrypted afterwards.

Expiring objects on a schedule

A bucket that collects logs, exports or build artefacts grows until somebody remembers to empty it. spec.lifecycle puts the forgetting in the bucket:

apiVersion: storage.deckhouse.io/v1alpha1
kind: Bucket
metadata:
  namespace: team-a
  name: build-logs
spec:
  objectStoreRef: standard
  lifecycle:
    rules:
      - id: logs
        prefix: logs/
        expireAfterDays: 30
      - id: leftovers
        abortIncompleteUploadsAfterDays: 1

Every rule needs an id, and no two may share one. It is the name the rule carries in the backend, which S3 requires to be unique and which this module uses to tell whether anything needs writing — so it is given, never derived from the rule’s position, where inserting a rule would rename the ones after it.

• expireAfterDays deletes an object that many days after it was written. On a versioned bucket this only makes the current version non-current — the data is still there, and expireNoncurrentAfterDays is what reclaims the space.
• expireNoncurrentAfterDays deletes a version that many days after it stopped being current.
• abortIncompleteUploadsAfterDays removes the parts of multipart uploads that were never completed. Those parts occupy space and appear in no listing, so without this they are storage nobody can see and nobody frees.
• prefix limits a rule to keys starting with it. Absent, the rule applies to every object in the bucket — worth reading twice before applying.

The whole set is written to the backend on every pass, assembled from the spec alone: a rule removed from the Bucket is a rule gone from the bucket. Nothing is patched, so a rule nobody asks for any more cannot be left behind quietly deleting objects.

Object lock outranks expiry. A version under an unexpired retention is not deleted, whatever a rule says — the lock is the stronger statement, and the engine enforces it.

Expiry only, no tiering. Moving objects between storage tiers is a lifecycle transition, and it is not offered here because it would work on one backend and not the other: SeaweedFS 4.39’s lifecycle engine compiles six actions — expiration by days, expiration by date, noncurrent expiration, newer-noncurrent, abort of incomplete uploads and delete-marker cleanup — and has no transition action at all. A Transition rule would be accepted, stored, listed back, and never run. Ceph RGW does support transitions across storage classes; when tiering arrives it will arrive as a field that says which backends honour it.

Which S3 operations work

The two backends are not the same S3, and the difference is worth knowing before an application meets it rather than after.

Where this table comes from. For SeaweedFS it is read off the engine’s own router at 4.39 — every operation it registers a handler for, and nothing else. For Ceph RGW it is Ceph’s own feature-support table for the release sds-elastic vendors. Neither is a promise this module makes: both are what the engine does, recorded so nobody has to find out from a failing SDK call. It is revisited when either engine version changes, and the version it was read at is in the heading of each column for exactly that reason.

Answered but not implemented. SeaweedFS serves a handful of endpoints purely so an SDK’s startup probe does not fail: accelerate configuration always reports Suspended, bucket logging returns an empty status, and the analytics, inventory, intelligent-tiering and metrics configuration listings return empty lists. A 200 from any of those means the request was understood, not that anything was configured.

What the module manages, and what it only passes through. The module drives bucket creation, the bucket policy behind accessPolicy, quotas, versioning, object lock and default encryption. Everything else in the table above is between the application and the engine — the module neither sets nor watches it.

Where the module asks for something a backend cannot do, it says so on the BucketContents rather than failing the bucket:

FeaturesApplied   False   Unsupported
  backend SeaweedFS does not enforce: quota.maxObjects

That condition is the runtime half of this table: the table says what the engine serves, and FeaturesApplied says what was actually enforced for one bucket.

Reclaim policy

• Bucket reclaimPolicy: Retain (the class default) — deleting the Bucket keeps the bucket and its objects. The BucketContents stays behind in phase Released, still recording which Bucket it belonged to and which store holds the data:

  d8 k get bktc
  # NAME                  STORE-KIND       STORE     OWNER-NS   OWNER      BUCKET            PHASE      READY   AGE
  # contents-1f2e3d-...   SeaweedFSStore   default   my-app     app-data   my-app-app-data   Released   False   4h

  Create a Bucket named app-data in my-app again and it re-binds those contents, data and all. This is also what happens when the whole namespace is deleted: the Bucket goes with it, the contents do not.

  Delete removes the bucket and its objects instead, and the contents with them. A Bucket that names no policy takes the class’s; the class default is Retain.

• Deleting a BucketAccess always revokes its access key and removes its credentials Secret (it does not touch bucket data).

• Store reclaimPolicy: Retain (default) — deleting the store preserves persisted data: an SDSElasticStore keeps its Ceph RGW pools, a SeaweedFSStore leaves its PVCs in place. Delete destroys it.

• Deleting an ObjectStore class destroys nothing. It takes away the name tenants provision through; the store, and every bucket already in it, are untouched — which is why BucketContents records the store rather than the class.

To let go of retained data, delete the BucketContents object: with the owning Bucket gone, deleting it removes the bucket in the backend.

The reclaim policy decides what happens when the Bucket is deleted; deleting the BucketContents itself is a separate instruction, and it always means “let this data go” — it is the only handle left once the Bucket has gone. Deleting it while the Bucket still exists does not touch the data: the controller provisions the backing object again and re-adopts the bucket. A Bucket that is itself being deleted does not count as still existing — it will never re-adopt anything — so deleting the contents under one means the same as deleting them with the Bucket already gone.

Ceph RGW stores

An SDSElasticStore provisions a Ceph RADOS Gateway on top of an existing sds-elastic cluster, and takes Ceph’s own pool settings:

apiVersion: storage.deckhouse.io/v1alpha1
kind: SDSElasticStore
metadata:
  name: heavy
spec:
  elasticClusterRef: main
  dataPool:
    replicated:
      size: 3
    # or, instead of replication:
    # erasureCoded: { dataChunks: 4, codingChunks: 2 }

Then a class in front of it, exactly as above:

apiVersion: storage.deckhouse.io/v1alpha1
kind: ObjectStore
metadata:
  name: capacity
spec:
  storeRef:
    kind: SDSElasticStore
    name: heavy