The module lifecycle stage: Experimental

The module has requirements for installation

Before enabling the module, you need to allow the use of experimental modules in the deckhouse ModuleConfig. To do this, set the allowExperimentalModules parameter to true.

The module is in the Experimental stage. The API, configuration, and custom resources may change without notice; do not use it for production workloads.

The system-s3 module provides the system S3 object storage for Deckhouse components. It runs one Garage cluster on nodes selected by the system-s3 role, then system, then other eligible nodes. SystemBucket provisions the bucket and connection parameters; consumers obtain temporary credentials separately through STS using a ServiceAccount token and RBAC.

It is not a storage service for applications. Workloads get their object storage from the sds-object module, which offers the user-facing backends; this module exists so platform components have somewhere to put their data without waiting for that one, and it is deliberately the smaller of the two.

What it does and what it does not

What the module does:

  • gives a platform component a bucket for a SystemBucket, with a quota on bytes and on objects, and the connection parameters in a ConfigMap;
  • hands out short-lived credentials through STS for a ServiceAccount token, under ordinary RBAC;
  • runs one replica, or three in the highly available mode, and switches between them on the same data;
  • moves the store to another StorageClass with every object checked, holding writes only for the switch-over;
  • grows the volumes with the buckets’ sizes and when a replica fills up;
  • reports its state in the conditions of Module/system-s3, in metrics, alerts and Grafana dashboards, and on the console page.

What it does not do:

  • serve applications: their object storage comes from sds-object;
  • issue long-lived keys;
  • keep object versions or lock objects (Object Lock): Garage implements neither;
  • use an external S3 store: the data is always in the module’s own Garage cluster;
  • shrink volumes, or keep a bucket’s data once its SystemBucket is deleted;
  • cap the CPU, memory, disk or network the store uses on its nodes: give it nodes of its own with the system-s3 role instead (see The data plane).

Custom resource

Resource Scope Purpose
SystemBucket (sb) Namespaced Asks for a bucket in the system storage and receives a ConfigMap with the S3 connection parameters in the same namespace.

That is the whole API, and each omission is deliberate:

  • No resource for the storage itself. The module runs exactly one data plane; its settings live in ModuleConfig. There is nothing to reference and no name to know.
  • No separate access resource. RBAC on systembuckets/read and systembuckets/write controls access. STS issues temporary keys; this contract has no long-lived consumer keys or manual rotation annotation.
  • No reclaim policy. Deleting a SystemBucket deletes the bucket and every object in it. A platform component owns its bucket outright.

Who may use it

SystemBucket is admitted only in a namespace that both carries the label heritage: deckhouse — the label Deckhouse puts on the namespace of every module — and has a d8-* or kube-* name, which only Deckhouse may give a namespace (the system-ns.deckhouse.io admission policy). Either alone admitted too much: the label can be put on any namespace by whoever may label namespaces, and the alert rules of the module see only namespaces with such names.

No user-facing RBAC is granted for the resource: only cluster and storage administrators (the d8:system-capability:system-s3:* capabilities on DKP 1.78 and later, the d8:manage:permission:module:system-s3:* roles below 1.78) can see or edit it.

The data plane

The data plane is built when the first SystemBucket is created, not when the module is enabled: until a platform component asks for a bucket the module runs only its controller and the credentials endpoint, and Module/system-s3 reports StorageReady=False with the reason WaitingForFirstBucket. Once built, the data plane stays — deleting the last SystemBucket deletes that bucket, not the store; only deleting the namespace of the disabled module removes it (see Removing the module). The settings can be changed freely before the first bucket: they decide what is built, and no migration is started.

One Garage cluster, deployed by the controller:

  • three replicas with a replication factor of 3 in the platform’s high-availability mode. Required pod anti-affinity keeps them on distinct eligible hosts; fewer than three hosts leave replicas Pending rather than co-locating them;

  • how big it is is not a setting: every replica’s volume is sized from the sum of the SystemBucket sizes (plus 10% headroom and a 4 GiB metadata reserve, at least 10 GiB, rounded up to a multiple of 5 GiB) and grows with it, as long as the StorageClass allows volume expansion. A replica whose volume has less than 5% free (CapacityReady=Exhausted) also grows the volumes, one 5 GiB step at a time, because the metadata engine can outgrow any reserve;

  • storage is one PersistentVolumeClaim per replica, ordered from storageClass (falling back to global.modules.storageClass and then the cluster default). A StorageClass is required — without one no data plane is created, and every SystemBucket says so;

  • outside that mode, a single replica with replication factor 1 — a third of the footprint, no redundancy. That replica never relocates: on a node-pinned class relocating it would mean losing the whole store (there is no second copy), so when its node goes away it stays Pending with its data intact on that node’s disk — the storage is unavailable until the node comes back. On a class whose volumes follow the pod the outage is only the move. The mode follows highAvailability, then global.highAvailability, and with neither set, whether the control plane itself is highly available. In that last case a shrink is not performed: a cluster that loses masters reads as not highly available, and the store keeps its three replicas until highAvailability: false is written out loud;

  • what a replica’s move costs depends on the class. Where volumes are pinned to nodes (a local class), moving a replica leaves its copy behind and Garage rebuilds it from the others — the next replica moves only once that rebuild is complete, which takes at least ten minutes per move; where volumes follow the pod (a network class), the volume is simply attached elsewhere. The module reads which it is from the volume itself;

  • placement follows the platform’s rule for system components: the data plane runs on nodes with the node-role.deckhouse.io/system-s3 role if the cluster has any, on system nodes (node-role.deckhouse.io/system) if it has those instead, and on the masters otherwise; it tolerates the dedicated.deckhouse.io taint with the value system-s3 or system, and on the masters their node-role.kubernetes.io/control-plane and node-role.kubernetes.io/master taints. To give the store nodes of its own, give a NodeGroup the system-s3 role — with disks sized for it, since that is where its volumes are ordered on a local class. Required anti-affinity separates HA replicas by kubernetes.io/hostname, so the highly available mode needs three nodes of the chosen step. The step is chosen by whether the cluster has nodes with a role, not by whether there are enough of them: a single node with the system-s3 (or system) role takes the data plane to that role, and it does not move on to the next step however many replicas are left without a place. With fewer nodes than replicas:

    • each node gets one replica and the remaining Pods stay Pending; PlacementReady=False/PlacementBlocked says how many nodes are needed and how many there are. PVCs are not recycled to consolidate the replicas;
    • a store that already ran on three replicas keeps working on two: Garage holds its 2-of-3 write quorum, with no redundancy left. With one replica left there is no quorum and the store is unavailable;
    • a new installation does not start at all: Garage refuses a layout with fewer nodes than the replication factor, and StorageReady=False reports the failure to apply the layout;
    • with neither the system-s3 nor the system role the replicas go to the masters. The default mode follows the control plane, and Deckhouse counts it as highly available from two masters on. So with one master the store runs one replica, with three it runs one replica per master, and with two masters it gets three replicas, the third of which has nowhere to go. The same holds for highAvailability: true with one or two masters.

    In every case the way out is one of three: give three nodes the system-s3 role, bring the nodes of the chosen step up to three, or set highAvailability: false. During a migration two Garage clusters run on the same nodes side by side; each spreads itself and neither pushes the other off a node. The controller itself stays on the control-plane nodes and follows control-plane HA independently of the Garage replica mode;

  • resources: a Garage container requests 100m CPU and 256 MiB, its TLS terminator 20m and 32 MiB. With the vertical-pod-autoscaler module enabled, each Garage StatefulSet gets a VerticalPodAutoscaler in Initial mode: a replica receives the recommended requests (Garage between 100m/256 MiB and 2 CPU/4 GiB) when its Pod is created — on a module update, a configuration change, a replica move, or a node drain — and the autoscaler never evicts a running replica to resize it. Without the module the requests above stay as they are;

  • what this means for the StorageClass: it has to be able to give volumes on those nodes, and with volumeBindingMode: WaitForFirstConsumer — with Immediate the volume is allocated before a node is chosen, and the replica is bound to wherever it landed.

Changing the high-availability mode of an installation that already holds data is done on the same Garage cluster and the same volumes. Garage cannot change its replication factor while it runs, so the module stops the data plane, removes the saved cluster layout and starts it again with the other replica count; nothing is copied. Going from one replica to three, the store is unavailable from the stop until the three replicas are up — with duration depending on volumes and startup time — and the new replicas are filled in the background while it serves. Going from three to one, writes are refused (403) until every replica holds everything, while reads keep working; if that takes longer than 10 minutes, writes are given back and the change is tried again later. The store is then unavailable while it restarts on one replica, and the volumes of the two replicas that are gone are released. Temporary keys asked for with write access while writes are refused are refused as well.

Changing the StorageClass — alone or together with the mode — is a migration between two Garage clusters. Bound volumes cannot be re-ordered from another class, so the module builds a second cluster next to the first, copies the objects into it and moves the endpoint onto it. Depending on how much there is to copy, that takes minutes or hours; while the copy runs, MigrationReady says how much of the source the new cluster already holds (bytes, objects and a percentage, from the two clusters’ bucket totals).

During cutover writes are refused with a 403. The final pass copies the delta and removes destination objects absent from the frozen source, then checks aggregate object and byte counts, not every object’s SHA256. An in-flight multipart upload may need to be retried; the controller sweeps incomplete parts older than a day. Previously issued temporary keys are not imported: consumers must obtain new credentials even before the old Expiration. Expiry-based SDK refresh does not guarantee recovery from this early invalidation; test the application’s handling of that 403. The endpoint address is stable, but uninterrupted access is not guaranteed. Progress is reported in Module/system-s3.MigrationReady; HA transitions use ReplicationReady. After the switch the old cluster stays read-only only until every replica of the new one holds every block, and is then removed; there is no rollback to it once it is gone.

Cutover also restarts every source Garage Pod on its existing PVC, without purging the layout, so that every request admitted before the freeze has finished. In the highly available mode the Pods are restarted one at a time — the next only after the previous one is back, ready and the cluster has synchronized — so two replicas keep a quorum and the store keeps serving. A read can still stall for up to about a minute while the replica Garage asks first for that object’s blocks is away: Garage waits for a peer that has disappeared until it misses four pings, 15 seconds apart. A single-replica store has to be stopped for it, and reads are unavailable during that restart. The final copier starts only after every old Pod has been replaced and the cluster has synchronized; termination is never force-bypassed on timeout. Pending write-key grants are tracked in the migration ConfigMap across controller replicas and restarts. An ambiguous grant RPC retains its barrier until Garage confirms the key has expired; a missing record alone is not proof of expiration. Do not manually remove these tickets or force-delete source Pods to unblock migration. The controller normally abandons a cutover after 30 minutes, stopping copier Pods before allowing writes again; recovery still depends on the source volumes and Pods becoming available.

Objects are skipped only when size and a non-empty ETag agree. Different multipart ETags require copying again, regardless of timestamps, so large multipart datasets can exceed the cutover budget. A confirmed destination quota failure in bulk copy enters bounded frozen synchronization, which prunes stale destination keys before copying replacements. Authentication failures do not trigger this transition and destination quotas are not removed.

Both clusters exist at once for the length of the migration, so the storage it needs is up to twice the usual — more when moving from one replica to three. Putting the setting back before the switch happens calls the migration off and removes the half-built cluster.

One way to reach a bucket

There are no long-lived credentials, and no Secret anywhere holding a key for a bucket. A consumer presents its ServiceAccount token at the credentials endpoint and receives a key pair the backend stops honouring at most thirty minutes later. The endpoint speaks AWS STS AssumeRoleWithWebIdentity, so any S3 client already knows how to use it; see USAGE.

Who may ask, and for which bucket, is Kubernetes RBAC: a Role granting get on systembuckets/read or systembuckets/write for the named bucket. That includes the component that created the resource — owning a SystemBucket is permission to edit the resource, not to read the objects in its bucket.

This is what makes access revocable. A stored key is a credential nobody can account for, and taking it back means rotating it for everyone who holds a copy. A grant is a RoleBinding: deleting it stops the next request, and what was already issued expires on the backend’s clock — nothing has to be rotated, because nothing was durable in the first place.

The cost is that a component has to be able to do the exchange. Something that can only read a key from a file or an environment variable cannot use this storage.

Garage is used as-is, through its standard admin API. It is AGPL-licensed, which is why it backs the platform’s own storage only and is never offered as a service to users.

Removing the module

Disabling the module asks for confirmation, and it stops the system object storage without deleting it. What the chart rendered goes — the controller, its Services, the credentials endpoint — and the Garage workload is scaled to zero. The module namespace stays, and with it the StatefulSets, the claims of their volumes and the Secrets that make those volumes a cluster again. Platform components lose access while the module is disabled; enabling it again brings the store back with its data and its buckets.

SystemBucket resources are not deleted. The module releases its finalizer on them so that they can be deleted without it, and they stay with their connection ConfigMaps.

Deleting the data is a separate, explicit step: with the module disabled, delete the namespace (kubectl delete namespace d8-system-s3). Disabling the module removes the heritage: deckhouse label from the namespace and the objects in it, so a cluster administrator can do this without Deckhouse’s own rights; enabling the module gives the label back. What happens to the volumes behind the claims is then the StorageClass’s reclaim policy: Delete removes them with the objects in them, Retain keeps them, but a module enabled afterwards does not reattach retained volumes — it starts with an empty store.

Before disabling the module, list the buckets with kubectl get sb -A: every component listed keeps data in the system object storage and loses access to it until the module is back.