The registry module: Custom Resources

RegistryConfig

Short names: rcfg

Scope: Cluster
Version: v1alpha1

Defines the resolved registry configuration of the cluster.

In normal operation the object is rendered by the module from ModuleConfig/registry. It is a contract rather than Deckhouse property: an administrator may edit it directly when the Deckhouse operator is unavailable, and the change is applied on the fly by the registry controller and the storage syncer without a helm re-render.

Only one object named registry is used.

• spec

  object

  Required value

  • spec.mode

    string

    Whether the module manages the image pull path.

    • Managed — the module owns the pull path and creates its components.
    • Unmanaged — the module owns nothing; no components are created.

    Default: Managed

    Allowed values: Managed, Unmanaged

  • spec.primary

    object

    The single authoritative source of Deckhouse component images.

    Additional registries are declared as separate RegistryUpstream objects. More than one primary source is not supported: it would make both “is the cache complete” and “what fills the cache in an air-gapped cluster” ambiguous.

    • spec.primary.upstream

      object

      The registry to pull Deckhouse component images from.

      If omitted, the cluster is air-gapped: the in-cluster cache becomes authoritative and is populated out of band with d8 mirror push. Omitting it requires storage.cache: true.

      • spec.primary.upstream.auth

        object

        Credentials for the registry.

        Sensitive. Credentials are stored inline in the object rather than referenced from a Secret, so that the node agent stays self-sufficient while the API server is unreachable. Restrict access to registry resources accordingly.

        • spec.primary.upstream.auth.auth

          string

          Pre-encoded base64("username:password").

          Takes precedence over username and password when set.

        • spec.primary.upstream.auth.password

          string

          The password for basic authentication.

        • spec.primary.upstream.auth.username

          string

          The username for basic authentication.

      • spec.primary.upstream.ca

        string

        A PEM-encoded certificate authority bundle used to verify the registry when scheme is HTTPS.

        If omitted, the system trust store is used.

      • spec.primary.upstream.host

        string

        Required value

        The registry host, optionally with a port.

        Example:

        host: registry.deckhouse.io
        

      • spec.primary.upstream.mirrors

        array of objects

        Additional addresses serving the same content as the primary registry, used for failover and load balancing.

        Mirrors are not separate sources. The storage caches one de-duplicated set regardless of which mirror the images were pulled from.

        • spec.primary.upstream.mirrors.auth

          object

          Credentials for the mirror.

          Sensitive. See primary.upstream.auth.

          • spec.primary.upstream.mirrors.auth.auth

            string

            Pre-encoded base64("username:password").

          • spec.primary.upstream.mirrors.auth.password

            string

            The password for basic authentication.

          • spec.primary.upstream.mirrors.auth.username

            string

            The username for basic authentication.

        • spec.primary.upstream.mirrors.ca

          string

          A PEM-encoded certificate authority bundle used to verify the mirror.

        • spec.primary.upstream.mirrors.host

          string

          Required value

          The mirror host, optionally with a port.

          Example:

          host: registry-mirror.example.com
          

        • spec.primary.upstream.mirrors.path

          string

          The repository prefix inside the mirror.

        • spec.primary.upstream.mirrors.scheme

          string

          The protocol used to reach the mirror.

          Default: HTTPS

          Allowed values: HTTP, HTTPS

      • spec.primary.upstream.path

        string

        The repository prefix inside the registry.

        Example:

        path: "/deckhouse/ee"
        

      • spec.primary.upstream.scheme

        string

        The protocol used to reach the registry.

        Use HTTP only for insecure, trusted registries.

        Default: HTTPS

        Allowed values: HTTP, HTTPS

  • spec.storage

    object

    Configures the in-cluster registry cache.

    • spec.storage.cache

      boolean

      Whether pulls go through the in-cluster cache on the master nodes.

      This is the first of the two configuration axes. The second is whether primary.upstream is set. Together they replace the former mode state machine:

      • cache: false with an upstream — the agent forwards pulls straight to the upstream and no storage is needed.
      • cache: true with an upstream — a pass-through cache, filled from the upstream.
      • cache: true without an upstream — air-gap: the cache is authoritative and is filled with d8 mirror push.

      Default: false

    • spec.storage.garbageCollection

      object

      When the cache reclaims the disk taken by releases the cluster has moved past.

      Nothing else ever removes anything: every release adds a slice of the repository, so a cluster that lives for years fills its store and then stops being able to pull.

      A collection does not interrupt serving: the replica keeps serving every image it holds. It does not start while the store is being filled or a d8 mirror push is in progress, and while it runs the replica stops filling its own store. A d8 mirror push started during a collection can fail and has to be retried. Only one replica collects at a time, so the others are unaffected.

      • spec.storage.garbageCollection.enabled

        boolean

        Whether the cache reclaims its disk on its own: on the schedule, after a new image set, and when the store refuses writes. A RegistryOperation created by hand runs either way.

      • spec.storage.garbageCollection.schedule

        string

        A five-field cron expression, in UTC.

        Defaults to a night hour, or to the start of the master node group’s maintenance window when one is set.

    • spec.storage.size

      string

      The size of the persistent volume backing the cache.

      Pattern: ^[0-9]+(\.[0-9]+)?(E|P|T|G|M|k|Ei|Pi|Ti|Gi|Mi|Ki)?$

      Example:

      size: 50Gi
      

    • spec.storage.source

      object

      The image set the cache is expected to hold.

      Required in an air-gapped cluster: with no upstream to fall back on, completeness must be decidable before it is safe to serve from the cache alone.

      • spec.storage.source.bundleRef

        string

        The name of the image set, for example the bundle pushed with d8 mirror push.

        Example:

        bundleRef: d8-mirror-bundle
        

      • spec.storage.source.expectedDigests

        integer

        The number of distinct digests the set contains.

        Allowed values: 0 <= X

        Example:

        expectedDigests: 459
        

• status

  object
  • status.conditions

    array of objects

    The health summary written by the registry controller.

    • Valid — the resolved configuration is coherent.
    • UpstreamValid — the result of the preflight probe of the primary upstream: reachable, credentials accepted, sentinel content in place. False means the controller is holding the last known good upstream and has not switched over, so a broken license or a wrong address cannot take the cluster down.
    • StorageConverged — the storage has reached the state the configuration asks for.
    • status.conditions.lastTransitionTime

      string

      Required value

      When the condition last changed.

    • status.conditions.message

      string

      A human-readable explanation.

    • status.conditions.observedGeneration

      integer

      The spec generation the condition was set for.

    • status.conditions.reason

      string

      Required value

      A machine-readable reason for the condition.

    • status.conditions.status

      string

      Required value

      The condition status.

      Allowed values: True, False, Unknown

    • status.conditions.type

      string

      Required value

      The condition type.

  • status.effectiveUpstream

    object

    The upstream actually in effect, which is not necessarily the configured one.

    This is the controller’s own record of the last upstream that passed its preflight probe, and it is what makes a bad change survivable: entering credentials that do not work, or switching to a license without access, leaves this field — and the cluster — on the previous working upstream, while the UpstreamValid condition reports why.

    It also holds the upstream through the air-gap transition, where the configuration already says “none” but the cache is not complete yet.

    Empty means no upstream is in effect: either the cluster is air-gapped, or nothing has been configured yet.

    • status.effectiveUpstream.ca

      string

      A PEM-encoded certificate authority bundle used to verify the registry.

    • status.effectiveUpstream.host

      string

      The registry host, optionally with a port.

    • status.effectiveUpstream.mirrors

      array of objects

      Additional addresses serving the same content.

      • status.effectiveUpstream.mirrors.ca

        string

        A PEM-encoded certificate authority bundle used to verify the mirror.

      • status.effectiveUpstream.mirrors.host

        string

        Required value

        The mirror host, optionally with a port.

      • status.effectiveUpstream.mirrors.path

        string

        The repository prefix inside the mirror.

      • status.effectiveUpstream.mirrors.scheme

        string

        The protocol used to reach the mirror.

        Allowed values: HTTP, HTTPS

    • status.effectiveUpstream.path

      string

      The repository prefix inside the registry.

    • status.effectiveUpstream.scheme

      string

      The protocol used to reach the registry.

      Allowed values: HTTP, HTTPS

  • status.effectiveUpstreamAuthDigest

    string

    Identifies the credentials in effect without disclosing them.

    Needed because “has the configuration changed?” has to include the credentials: a rotated license under an unchanged address is exactly the change that must be probed before the cluster is switched onto it. With the credentials no longer recorded here, comparing this against a digest of the configured ones is what keeps that question answerable.

  • status.observedGeneration

    integer

    The spec generation the controller last acted on.

RegistryNode

Short names: rnode

Scope: Cluster
Version: v1alpha1

Defines the complete registry layout of a single node, as compiled for its agent.

One object per node, named after the node, with an ownerReference on it. Written by the registry controller; status is written by the agent only.

The object is flat and, apart from one deliberate reference, self-contained: certificate authority bundles and routes are inlined, and the agent keeps a full copy on disk so it keeps serving images while the API server is unreachable — at which point there is nothing left to dereference. The exception is credentials: they are named rather than held (auth.secretRef points at the Secret this module resolves them into), which is what lets every node read this object without also reading every registry credential in the cluster. For the same reason the controller compiles every accepted RegistryUpstream into spec.additionalRoutes instead of having the agent watch N objects: consistency across many objects cannot be guaranteed when the API is down.

The object carries no per-node secrets. Everything in it is global configuration, identical on every node. This is what makes it safe for the whole system:nodes group to read, which is how the agent gets access using the kubelet’s kubeconfig.

• spec

  object

  Required value

  • spec.additionalRoutes

    array of objects

    The accepted RegistryUpstream objects, compiled by the controller.

    Always transit, never cached: the agent proxies the pull and streams it through without writing to disk.

    • spec.additionalRoutes.auth

      object

      Where the credentials for this endpoint are kept.

      Named rather than carried. This resource is cluster-scoped and the node agent’s role is bound to the system:nodes group, so a credential written here would be readable through the API by every kubelet in the cluster — including credentials for registries that node never pulls from. The component that reads this resolves the reference once and keeps the result, so it still works when the API server does not.

      • spec.additionalRoutes.auth.secretRef

        object

        The Secret holding the credentials, and the key inside it.

        • spec.additionalRoutes.auth.secretRef.key

          string

          Required value

          Key inside it. One Secret holds every credential the module resolved, keyed by what it belongs to.

        • spec.additionalRoutes.auth.secretRef.name

          string

          Required value

          Name of the Secret in the d8-system namespace.

    • spec.additionalRoutes.ca

      string

      A PEM-encoded certificate authority bundle used to verify the upstream.

    • spec.additionalRoutes.host

      string

      Required value

      The upstream host, optionally with a port.

      Example:

      host: registry-vendor.example.com
      

    • spec.additionalRoutes.match

      string

      Required value

      The registry host intercepted on this node.

      Example:

      match: images.virtualization.example.com
      

    • spec.additionalRoutes.mirrors

      array of objects

      Additional addresses serving the same content.

      • spec.additionalRoutes.mirrors.auth

        object

        Where the credentials for this mirror are kept.

        Named rather than carried, for the same reason as the endpoint above: this resource is readable by every node in the cluster. A mirror can need credentials of its own, so it can name a key of its own.

        • spec.additionalRoutes.mirrors.auth.secretRef

          object

          The Secret holding the credentials, and the key inside it.

          • spec.additionalRoutes.mirrors.auth.secretRef.key

            string

            Required value

            Key inside it.

          • spec.additionalRoutes.mirrors.auth.secretRef.name

            string

            Required value

            Name of the Secret in the d8-system namespace.

      • spec.additionalRoutes.mirrors.ca

        string

        A PEM-encoded certificate authority bundle used to verify the mirror.

      • spec.additionalRoutes.mirrors.host

        string

        Required value

        The mirror host, optionally with a port.

      • spec.additionalRoutes.mirrors.path

        string

        The repository prefix inside the mirror.

      • spec.additionalRoutes.mirrors.scheme

        string

        The protocol used to reach the mirror.

        Default: HTTPS

        Allowed values: HTTP, HTTPS

    • spec.additionalRoutes.path

      string

      The repository prefix inside the upstream.

    • spec.additionalRoutes.scheme

      string

      The protocol used to reach the upstream.

      Default: HTTPS

      Allowed values: HTTP, HTTPS

  • spec.backends

    array of objects

    The sources of the primary image set, in priority order.

    With the cache on, this is the storage followed by the upstream as a fallback while the cache is still filling; the upstream entry disappears from the layout when the cluster goes air-gap. With the cache off, it is the upstream alone.

    • spec.backends.auth

      object

      Where the credentials for this endpoint are kept.

      Named rather than carried. This resource is cluster-scoped and the node agent’s role is bound to the system:nodes group, so a credential written here would be readable through the API by every kubelet in the cluster — including credentials for registries that node never pulls from. The component that reads this resolves the reference once and keeps the result, so it still works when the API server does not.

      • spec.backends.auth.secretRef

        object

        The Secret holding the credentials, and the key inside it.

        • spec.backends.auth.secretRef.key

          string

          Required value

          Key inside it. One Secret holds every credential the module resolved, keyed by what it belongs to.

        • spec.backends.auth.secretRef.name

          string

          Required value

          Name of the Secret in the d8-system namespace.

    • spec.backends.ca

      string

      A PEM-encoded certificate authority bundle used to verify the backend.

    • spec.backends.host

      string

      Required value

      The backend host, optionally with a port.

      Example:

      host: registry.d8-system.svc:5001
      

    • spec.backends.mirrors

      array of objects

      Other addresses of the same backend. The agent recognises requests that name them, but pulls the primary image set only from the endpoint.

      • spec.backends.mirrors.auth

        object

        Where the credentials for this mirror are kept.

        Named rather than carried, for the same reason as the endpoint above: this resource is readable by every node in the cluster. A mirror can need credentials of its own, so it can name a key of its own.

        • spec.backends.mirrors.auth.secretRef

          object

          The Secret holding the credentials, and the key inside it.

          • spec.backends.mirrors.auth.secretRef.key

            string

            Required value

            Key inside it.

          • spec.backends.mirrors.auth.secretRef.name

            string

            Required value

            Name of the Secret in the d8-system namespace.

      • spec.backends.mirrors.ca

        string

        A PEM-encoded certificate authority bundle used to verify the mirror.

      • spec.backends.mirrors.host

        string

        Required value

        The mirror host, optionally with a port.

      • spec.backends.mirrors.path

        string

        The repository prefix inside the mirror.

      • spec.backends.mirrors.scheme

        string

        The protocol used to reach the mirror.

        Default: HTTPS

        Allowed values: HTTP, HTTPS

    • spec.backends.name

      string

      Required value

      The backend’s role.

      • Storage — the in-cluster cache.
      • Upstream — the primary upstream, used either as the only backend or as a fallback while the cache fills.

      Allowed values: Storage, Upstream

    • spec.backends.path

      string

      The repository prefix inside the backend.

    • spec.backends.scheme

      string

      The protocol used to reach the backend.

      Default: HTTPS

      Allowed values: HTTP, HTTPS

  • spec.cache

    boolean

    The cache axis for this node.

    true sends the agent to the in-cluster storage, false straight to the primary upstream.

• status

  object

  Written by the agent only, reporting containerd-side facts.

  It carries no storage completeness: that is a property of the storage and lives in RegistryStorage .status.replicas. Any node can patch any RegistryNode status, because RBAC cannot scope list/watch by field selector, so nothing here may be load-bearing for the pull path — it is observability.

  • status.activeBackends

    array of strings

    The backends the agent is currently able to use.

  • status.conditions

    array of objects

    The detailed state reported by the agent.

    • Reconciled — whether the spec has been applied. The reason distinguishes a normal reconcile from one applied out of the on-disk copy (AppliedFromCache): the agent could not get a usable spec from the API server. Most often the credentials it references could not be read; a read of the spec that failed while the status write after it went through, as on a short API server outage, looks the same. The agent reports the fallback by writing this status, so an agent that cannot write it at all, such as one on a node cut off from the API server, leaves the previous condition in place.
    • status.conditions.lastTransitionTime

      string

      Required value

      When the condition last changed.

    • status.conditions.message

      string

      A human-readable explanation.

    • status.conditions.observedGeneration

      integer

      The spec generation the condition was set for.

    • status.conditions.reason

      string

      Required value

      A machine-readable reason for the condition.

    • status.conditions.status

      string

      Required value

      The condition status.

      Allowed values: True, False, Unknown

    • status.conditions.type

      string

      Required value

      The condition type.

  • status.observedGeneration

    integer

    The spec generation the agent last applied.

  • status.proxyListening

    boolean

    Whether the agent is serving the registry endpoint.

  • status.reconciled

    boolean

    Whether the agent has applied the spec it currently sees.

  • status.staleStorageDataBytes

    integer

    How much of the in-cluster cache’s data is still on this node while no cache is configured.

    The blobs are left behind deliberately when the cache is turned off: turning it back on then refills from what is already there rather than from scratch, and over a slow link that difference is hours. What would not be acceptable is keeping them silently, since nothing else in the module would ever mention the disk they occupy — so the agent measures them and an alert says so.

    To reclaim the space, remove the contents of the cache directory on the node, keeping the directory itself, which the agent has mounted: /var/lib/deckhouse/registry on a node of a NodeGroup with systemType: DeckhouseEngine, /opt/deckhouse/registry on other nodes. Absent while a cache is configured, and on a node that never held one.

RegistryOperation

Short names: rop

Scope: Cluster
Version: v1alpha1

An operation every replica of the registry cache performs on its own store, one replica at a time.

Images keep being served throughout.

The only type is GarbageCollection: it removes from the cache what the releases the cluster has moved past left there. The module creates these operations itself:

• on the schedule set in storage.garbageCollection of the module configuration;
• once the cache holds a new image set — after a Deckhouse Platform update, or after a module is enabled or updated;
• when the store refuses writes and so can never take in a new set — once per deployed Deckhouse Platform release.

With storage.garbageCollection.enabled: false the module creates none of them. An operation created by hand runs either way.

A replica starts only when its store is not being written to: while the cache is being filled or an image set is being pushed with d8 mirror push, the replica waits. A store that refuses writes — over its budget, or down to the space kept free on its filesystem — is not waited for: a fill cannot go on until something frees room. A replica that has not picked the operation up, or is still held back by its store, an hour after the operation was accepted fails with the reason DeadlineExceeded. Once its turn comes, a replica looks at its store again: if a fill or a push has started meanwhile, it gives the turn back and waits for the store to go quiet.

Only one operation runs at a time. An operation created while an older one is not finished fails with the reason AnotherOperationActive.

A replica that is gone fails with the reason ReplicaLost: one that was collecting, when it has not held the Lease registry-storage-gc in the d8-system namespace for five minutes; one that was waiting, when its node has not been Ready for ten minutes.

To cancel an operation, set the label registry.deckhouse.io/cancel: "true" on it. A replica that has not started does not start, and a replica that is running stops. What it has already removed stays removed. Deleting the object cancels an operation too, but does not stop a replica that has already started.

A finished operation is deleted a day after it finished.

apiVersion: deckhouse.io/v1alpha1
kind: RegistryOperation
metadata:
  generateName: gc-manual-
spec:
  type: GarbageCollection
  cause: Free space before pushing a new release.

• spec

  object

  Required value

  Does not change once the operation is created.

  • spec.cause

    string

    Why the operation was created, for a person reading it.

    The module sets it on the operations it creates itself.

    Maximum length: 1024

  • spec.type

    string

    Required value

    What the operation does.

    • GarbageCollection — remove from the cache what the releases the cluster has moved past left there. The deployed release, the previous one, anything newer and every tag that is not a version are kept.

    Allowed values: GarbageCollection

• status

  object
  • status.acceptedAt

    string

    When the operation was accepted.

  • status.deadline

    string

    When a replica held back by a store that is being written to gives up.

    An hour after acceptedAt.

  • status.finishedAt

    string

    When the operation reached a terminal phase.

    The operation is deleted a day later.

  • status.garbageCollection

    object

    What a GarbageCollection removed, summed over the replicas.

    • status.garbageCollection.reclaimedBytes

      integer

      How many bytes the store shrank by: its blobs and uploads in progress, measured before the collection and after.

      Absent when the store could not be measured.

    • status.garbageCollection.removedBlobs

      integer

      How many blobs the sweep removed, as the registry counts them.

      A sweep also removes what an earlier one failed to, so these blobs need not belong to the images deleted in this run. Absent when the sweep did not run.

    • status.garbageCollection.removedManifests

      integer

      How many images were deleted by digest — the images the fill writes with no tag, where nearly all of a store’s weight is.

    • status.garbageCollection.removedVersions

      array of strings

      The Deckhouse Platform releases removed.

  • status.message

    string

    A human-readable explanation of the reason.

  • status.phase

    string

    Where the operation is.

    • Pending — no replica has started yet.
    • Running — at least one replica has started, and not every one has finished.
    • Succeeded — every replica finished the operation.
    • Failed — the operation was refused, or a replica did not finish it.
    • Canceled — the operation was canceled before every replica finished.

    Allowed values: Pending, Running, Succeeded, Failed, Canceled

  • status.reason

    string

    A machine-readable reason for the phase.

    • Pending: Accepted.
    • Running: InProgress.
    • Succeeded: Completed.
    • Failed: NoCache (the cache is turned off), NoReplicas (no replica of the storage has reported yet), AnotherOperationActive, ReplicaFailed.
    • Canceled: Canceled.
  • status.replicas

    array of objects

    One entry per replica the operation was handed to.

    Every replica of the storage that had reported when the operation was accepted is included.

    • status.replicas.finishedAt

      string

      When the replica finished.

    • status.replicas.garbageCollection

      object

      What a GarbageCollection removed from the replica’s store.

      • status.replicas.garbageCollection.reclaimedBytes

        integer

        How many bytes the store shrank by: its blobs and uploads in progress, measured before the collection and after.

        Absent when the store could not be measured.

      • status.replicas.garbageCollection.removedBlobs

        integer

        How many blobs the sweep removed, as the registry counts them.

        A sweep also removes what an earlier one failed to, so these blobs need not belong to the images deleted in this run. Absent when the sweep did not run.

      • status.replicas.garbageCollection.removedManifests

        integer

        How many images were deleted by digest — the images the fill writes with no tag, where nearly all of a store’s weight is.

      • status.replicas.garbageCollection.removedVersions

        array of strings

        The Deckhouse Platform releases removed.

    • status.replicas.message

      string

      A human-readable explanation of the reason.

    • status.replicas.node

      string

      Required value

      The node the replica runs on.

    • status.replicas.phase

      string

      Required value

      Where the replica is.

      • Pending — the replica has not picked the operation up yet.
      • Waiting — the replica is held back; reason says by what.
      • Running — the replica is running the operation.
      • Succeeded — the replica finished the operation.
      • Failed — the replica did not finish the operation.
      • Skipped — the replica left the storage before it started.
      • Canceled — the operation was canceled before the replica finished.

      Allowed values: Pending, Waiting, Running, Succeeded, Failed, Skipped, Canceled

    • status.replicas.reason

      string

      A machine-readable reason for the phase.

      • Waiting: Filling (the cache is being filled), Pushing (an image set is being pushed with d8 mirror push), Turn (another replica is running the operation; or the replica waited a whole window for its turn and waits for the store to go quiet before waiting again).
      • Failed: OperationFailed, DeadlineExceeded (by deadline, the replica had not picked the operation up, or its store was still being written to), TimedOut (the replica did not report a result within 24 hours), ReplicaLost (the replica is gone: it stopped holding the collection lease while it collected, or its node was not Ready for ten minutes while it waited).
      • Skipped: ReplicaGone, NoCache.
      • Canceled: Canceled.
    • status.replicas.startedAt

      string

      When the replica started.

  • status.startedAt

    string

    When the first replica started.

RegistryStorage

Short names: rstor

Scope: Cluster
Version: v1alpha1

Defines how the in-cluster registry storage operates, and reports the state it is actually in.

The object is written by the registry controller and read by the syncer sidecar, which reconfigures distribution on the fly. This is the path that makes a credential or certificate change take effect while the Deckhouse operator is down: it never goes through a helm re-render.

It carries a real spec rather than being a status-only mirror, because distribution has to be told what to do, not merely observed.

Only one object named registry is used.

• spec

  object

  Required value

  • spec.airGapRequested

    boolean

    Whether the configuration asked to run without an upstream, whether or not the upstream has been dropped yet.

    This is the transition window: air-gap asked for, the upstream still held so the cluster keeps working, images arriving through d8 mirror push that the syncer never sees. It is what decides that completeness must be measured by reading the store rather than by counting what a copy wrote — a distinction that matters because a pass-through cache writes into its own store on every miss, so the same reading on a caching cluster would count images it merely fetched.

  • spec.auth

    string

    The bearer token guarding the storage.

    Sensitive. Authentication is never disabled, not even in an air-gapped cluster where the cache only serves reads.

  • spec.garbageCollection

    object

    When and whether a replica reclaims the disk taken by releases the cluster has moved past.

    Needed because nothing else ever removes anything: every release adds a slice of the repository, so a cluster that lives for years fills its store and then stops being able to pull.

    A collection does not interrupt serving: the replica keeps serving every image it holds. It does not start while the store is being filled or a d8 mirror push is in progress, and while it runs the replica stops filling its own store. A d8 mirror push started during a collection can fail and has to be retried. Only one replica collects at a time, so the others are unaffected.

    • spec.garbageCollection.enabled

      boolean

      Whether the controller creates garbage collections on its own: on the schedule, after a new image set, and when the store refuses writes.

      A RegistryOperation created by hand runs either way.

    • spec.garbageCollection.schedule

      string

      A five-field cron expression, in UTC.

      Defaults to a night hour, or to the start of the master node group’s maintenance window when one is set — that being the time an operator has already declared safe for disruption.

  • spec.needSync

    boolean

    How the controller hands the syncers a fill or replication task.

    Set while any replica is short of the expected set: the leader filling from the upstream, or a follower that has not caught up with the leader. Cleared once every replica holds the set; nothing is copied until the set changes again.

  • spec.publish

    boolean

    Whether to expose an authenticated write endpoint (Ingress) for d8 mirror push.

    Write credentials are separate from read credentials. The endpoint is a second listener of the same registry, served without the cache in front of the store, so publishing it costs the address the cluster pulls through nothing — one listener cannot both proxy an upstream and accept a push.

  • spec.source

    object

    The image set that must end up in the storage.

    • spec.source.bundleRef

      string

      The name of the image set.

    • spec.source.expectedDigests

      integer

      The number of distinct digests the set contains.

      Allowed values: 0 <= X

  • spec.store

    object

    Where distribution keeps its blobs.

    • spec.store.path

      string

      Path on the node. Set by the module, not by the user: /var/lib/deckhouse/registry when the master NodeGroup has systemType: DeckhouseEngine, /opt/deckhouse/registry otherwise.

      Example:

      path: "/opt/deckhouse/registry"
      

    • spec.store.size

      string

      The budget of each replica’s store, taken from storage.size of the ModuleConfig: how much its blobs and uploads in progress may take. Unset means no budget; the reserve kept free on the store’s filesystem (see status.replicas[].store.reserveBytes) applies either way.

      Pattern: ^[0-9]+(\.[0-9]+)?(E|P|T|G|M|k|Ei|Pi|Ti|Gi|Mi|Ki)?$

  • spec.upstream

    object

    The primary upstream, projected here by the controller.

    Set means a pass-through cache. Absent means the cache is authoritative (air-gap).

    The controller removes this field only after the leader is full, so nodes are never left without a source of images — even if the upstream has already been removed from the configuration by the user.

    • spec.upstream.auth

      object

      Where the credentials for this endpoint are kept.

      Named rather than carried. This resource is cluster-scoped and the node agent’s role is bound to the system:nodes group, so a credential written here would be readable through the API by every kubelet in the cluster — including credentials for registries that node never pulls from. The component that reads this resolves the reference once and keeps the result, so it still works when the API server does not.

      • spec.upstream.auth.secretRef

        object

        The Secret holding the credentials, and the key inside it.

        • spec.upstream.auth.secretRef.key

          string

          Required value

          Key inside it. One Secret holds every credential the module resolved, keyed by what it belongs to.

        • spec.upstream.auth.secretRef.name

          string

          Required value

          Name of the Secret in the d8-system namespace.

    • spec.upstream.ca

      string

      A PEM-encoded certificate authority bundle used to verify the registry.

    • spec.upstream.host

      string

      Required value

      The registry host, optionally with a port.

    • spec.upstream.mirrors

      array of objects

      Additional addresses serving the same content. The syncer pulls from whichever is reachable and stores one de-duplicated set.

      • spec.upstream.mirrors.auth

        object

        Where the credentials for this mirror are kept.

        Named rather than carried, for the same reason as the endpoint above: this resource is readable by every node in the cluster. A mirror can need credentials of its own, so it can name a key of its own.

        • spec.upstream.mirrors.auth.secretRef

          object

          The Secret holding the credentials, and the key inside it.

          • spec.upstream.mirrors.auth.secretRef.key

            string

            Required value

            Key inside it.

          • spec.upstream.mirrors.auth.secretRef.name

            string

            Required value

            Name of the Secret in the d8-system namespace.

      • spec.upstream.mirrors.ca

        string

        A PEM-encoded certificate authority bundle used to verify the mirror.

      • spec.upstream.mirrors.host

        string

        Required value

        The mirror host, optionally with a port.

      • spec.upstream.mirrors.path

        string

        The repository prefix inside the mirror.

      • spec.upstream.mirrors.scheme

        string

        The protocol used to reach the mirror.

        Default: HTTPS

        Allowed values: HTTP, HTTPS

    • spec.upstream.path

      string

      The repository prefix inside the registry.

    • spec.upstream.scheme

      string

      The protocol used to reach the registry.

      Default: HTTPS

      Allowed values: HTTP, HTTPS

• status

  object
  • status.allReplicasFull

    boolean

    Whether every replica holds the expected set, so losing the leader costs nothing.

  • status.authoritative

    boolean

    Whether the upstream has been dropped and the cache is now the only source of images.

  • status.conditions

    array of objects

    The detailed state of the storage.

    • status.conditions.lastTransitionTime

      string

      Required value

      When the condition last changed.

    • status.conditions.message

      string

      A human-readable explanation.

    • status.conditions.observedGeneration

      integer

      The spec generation the condition was set for.

    • status.conditions.reason

      string

      Required value

      A machine-readable reason for the condition.

    • status.conditions.status

      string

      Required value

      The condition status.

      Allowed values: True, False, Unknown

    • status.conditions.type

      string

      Required value

      The condition type.

  • status.fill

    object

    Progress of the current fill task.

    • status.fill.filled

      integer

      The number of digests written so far.

    • status.fill.total

      integer

      The number of digests expected.

  • status.leader

    string

    The node currently holding the storage leader lease.

    Not a plain election. Leadership is not a symmetric role — the leader is the replication source every follower copies from, and the one whose completeness gates going air-gap. Once some replica holds the whole expected set, a leader that cannot complete the set by itself steps aside for it: one with no upstream, one whose last pass failed, one whose store refuses writes, or one that has stopped renewing the lease. A leader still completing the set from the upstream keeps the lease even if a follower finishes first. While no replica is full, which is where every cluster starts, the lease does not move; with no leader at all, it goes to the replica holding the most verified images.

    Without that rule an air-gapped cluster can stall. d8 mirror push arrives through the publication endpoint and always lands on the leader; an incomplete leader beside a full follower (for example, one whose store was lost) has no upstream to fill itself from, the followers replicate its gaps, and the replica holding the images sits idle.

  • status.observedGeneration

    integer

    The spec generation the controller last acted on.

  • status.phase

    string

    A coarse summary of the fill state.

    • Idle — no fill or verification task is running.
    • Filling — the leader is being populated.
    • Verifying — the contents are being checked against spec.source.
    • Ready — the leader holds the expected set.
    • Failed — a fill or verification task failed. The upstream is not dropped in this state.

    Allowed values: Idle, Filling, Verifying, Ready, Failed

  • status.replicas

    array of objects

    The per-replica fill state.

    This is the only place storage completeness is reported. RegistryNode deliberately carries none of it, since completeness is a property of the storage and not of a node.

    • status.replicas.address

      string

      Where this replica answers, as host:port.

      Reported by the replica itself so that a follower can replicate from the leader without resolving a node name into an address.

    • status.replicas.aheadSet

      string

      The parts of what is ahead of the set that this replica holds as well — the platform version the cluster updates to next, and each module update it is about to apply — as their keys, sorted and comma-separated.

      A part is reported once a copy ahead left it whole. Separate from verifiedSet because it is copied separately — after the set, in the background, best-effort — and by part, so that one module that never arrives does not keep the rest from counting. A follower copies the parts its leader holds and it does not, so an update brought to the leader alone, such as one pushed with d8 mirror push, reaches the other replicas. Empty when nothing is pending or nothing pending has been copied.

    • status.replicas.collectedAt

      string

      When this replica last reclaimed its disk.

      Absent until it has. Reported per replica because each collects its own store, one at a time: a cluster where one replica has not collected for a month while the others have is a different problem from one where none has.

    • status.replicas.collectionError

      string

      Why the last attempt to reclaim did not finish, empty when it did.

      Separate from error, which is about filling. A store that cannot be reclaimed still serves every image it holds, so the two say different things about how worried to be.

    • status.replicas.declaredDigests

      integer

      How big the set is — the number verifiedDigests is out of.

      Reported by the replica because only the replica computing the set knows its size, and nothing is decided from it. Before it existed the only denominator the status could use was spec.source.expectedDigests, which an operator states when declaring air-gap and nobody states otherwise — so a cluster with an upstream showed filled: 0, total: 0 throughout its longest phase while gigabytes were being copied. Completeness is still full, and the air-gap transition is still decided from spec.source.expectedDigests: a declaration by an operator is not replaced by a number the cluster computes about itself.

    • status.replicas.error

      string

      Why the last fill or verification task failed, as reported by this replica’s syncer. Empty when the last task succeeded.

      While the last task did not fail, it also carries what healthError reports.

      Safety does not depend on this field: a failed replica is simply not full, and the upstream is only ever dropped once the leader is full. It exists so that an operator can tell “still filling” from “broken”, which the fill counters alone cannot express.

    • status.replicas.full

      boolean

      Whether this replica holds the expected set.

      Derived from what the syncer actually wrote, never from probing the serve path: on a miss a pass-through cache would fetch the image from the upstream and report success on an empty store. In an air-gapped cluster self-filling is impossible, so reading the catalogue and comparing against spec.source.expectedDigests is honest there.

    • status.replicas.healthError

      string

      Why this replica cannot serve what it holds: its registry has not answered for five minutes, or its store has refused writes for two. Empty while both work.

      The syncer checks both every half a minute on its own, whatever the fill is doing, and clears the error once the registry answers and the store accepts writes again; full is left as it was and counts again from then on. A leader reporting it hands the lease over to a replica that holds the whole expected set, if there is one; otherwise it keeps the lease and only reports the error.

      While no replica holds the whole set and none leads, the replicas’ syncers choose among themselves which one leads and begins filling: the one that holds the most. They skip a replica reporting this error. They also skip one whose store is at its budget or at the node’s reserve (store.writable is false), unless every replica’s store is. An error alone, such as a failed copy from a leader that is gone, does not keep a replica from being chosen.

    • status.replicas.node

      string

      Required value

      The node the replica runs on.

    • status.replicas.role

      string

      The replica’s role in replication.

      • Leader — filled from the upstream, or by d8 mirror push in an air-gapped cluster, and hands out replication tasks.
      • Follower — filled from the leader, ahead of time, so a leader loss does not mean filling from scratch.

      Allowed values: Leader, Follower

    • status.replicas.source

      string

      The replica this one is filled from. Empty for the leader.

    • status.replicas.store

      object

      How much of its node’s disk this replica’s store takes, and whether it still accepts writes. Absent until the replica has measured it.

      The registry enforces both limits itself, on every write; this is the syncer’s account of the same figures, for the status and the alerts. Sizes are rounded to 16 MiB — used up, free and capacity down — so that a filesystem shared with the rest of the node does not rewrite the status on every pass.

      • status.replicas.store.budgetBytes

        integer

        storage.size of the module configuration. Absent when none is set, and then only the reserve applies.

      • status.replicas.store.dedicatedFilesystem

        boolean

        True when the store has a filesystem of its own: a disk mounted at the store’s directory or at the store path above it, holding neither the kubelet’s nor containerd’s directory. The replica finds this from its mounts when it starts. Absent when the store shares a filesystem with the node, or when that could not be told: such a filesystem is treated as shared.

      • status.replicas.store.filesystemCapacityBytes

        integer

        Required value

        Size of the filesystem the store is on, as measured now: a filesystem grown under a running store shows its new size, and the reserve follows it.

      • status.replicas.store.filesystemFreeBytes

        integer

        Required value

        Free space on the filesystem the store is on. On a node without a dedicated disk it is the filesystem the kubelet and containerd use too.

      • status.replicas.store.reason

        string

        What writes are refused on, when they are.

        • BudgetExhausted — the store holds as much as storage.size allows;
        • ReserveExhausted — the filesystem under the store is down to the space kept free on it, see reserveBytes;
        • NotWritable — the registry cannot create files in its tree in the store: docker/registry/v2 and the directories under it, or the store’s directory while that tree does not exist yet. Their owner or mode, the immutable attribute, a read-only mount or a security module stands in the way. No limit is involved, and neither a garbage collection nor a larger storage.size helps. The replica’s error names the directory and the command that fixes it. Takes precedence over either limit.

        Allowed values: BudgetExhausted, ReserveExhausted, NotWritable

      • status.replicas.store.reserveBytes

        integer

        Required value

        How much of the filesystem the store leaves free, taken of the filesystem’s current size rounded up to a whole GiB. On a filesystem shared with the node it is the kubelet’s soft eviction threshold (10%, at most 40 GiB) plus a margin for other writers (10%, at most 20 GiB). On a filesystem of the store’s own (see dedicatedFilesystem) only the margin is kept.

      • status.replicas.store.usedBytes

        integer

        Required value

        What the store’s blobs and uploads in progress take.

      • status.replicas.store.writable

        boolean

        Required value

        False while the store refuses new writes. Reads are never refused: a full store serves everything it holds, and a pull it cannot keep is streamed from the upstream. The exception is reason: NotWritable, where the registry fails such a pull: its directories refuse it before any limit is checked.

    • status.replicas.syncerStartedAt

      string

      When the syncer process that wrote this entry started, to the second. Set only once that process has itself counted, copied into or rechecked its store; absent before.

      An entry outlives the process that wrote it. A leader whose pod is recreated on the same node takes the lease back at once, and until the new process reads its store, its entry still says what the previous process found, full included, even if the store was wiped with the pod. So the upstream is dropped only on a lease holder’s entry stamped no earlier than the start of the syncer container running in its Ready pod. Both times come from the clock of that node.

    • status.replicas.totalDigests

      integer

      How many distinct digests this replica holds altogether.

      A different population from verifiedDigests, and nothing is decided from it. It is here so that a store holding more than this cluster needs — a bundle pushed with other modules, other editions, attestations — looks like what it is rather than like a shortfall against spec.source.expectedDigests.

    • status.replicas.verifiedDigests

      integer

      How many digests of the set were confirmed present.

      The set is what the cluster’s releases and the modules it keeps declare — the digests deckhouse-controller knows about — so this number is directly comparable with full and is what the air-gap transition is decided from. Anything else the store happens to hold is counted in totalDigests.

    • status.replicas.verifiedSet

      string

      A fingerprint of the set this replica was last found full against: the platform’s versions and the modules the cluster keeps, each with its version and source.

      full is only as good as the set it was measured against, and the set moves — a module is installed, the platform is updated. Every replica reports it with its full. The leader compares its own with the set the cluster needs now; when they differ it takes full back and verifies before an air-gap transition may rely on it. A full for another set counts for nothing when the replicas decide who may lead. Empty when full is not set.

  • status.safeToDropUpstream

    boolean

    Whether the leader holds the expected set and nothing on the platform still names the upstream, which together are the gate for going air-gap.

    Deliberately the leader alone and not every replica: followers are filled ahead of time and in parallel, so waiting for all of them would delay the transition without adding safety.

    A full cache does not make up for a remaining reference: an image reference that names the upstream by its own name is not answered by the cache once the upstream is gone. For details, see upstreamReferences.

    The leader’s full counts only when the syncer running on its node now wrote it, after reading its own store (see replicas[].syncerStartedAt). After the leader’s syncer restarts, this stays false until its first pass, normally within two minutes.

  • status.upstreamReferences

    integer

    Number of objects in platform namespaces whose images still reference the upstream the storage is holding.

    Each object is counted separately: Deployment, DaemonSet, StatefulSet and CronJob templates, pods not started yet, pods that would pull again on their next start, and replicas of the storage itself, which cannot pull their own image from the storage once the upstream is gone. A Deployment in the middle of a rollout adds one for its template and one for each such pod.

    Only the kube-system and d8-* namespaces are counted. Workloads in other namespaces whose images reference the upstream address do not hold the transition, and once the upstream is removed they have nowhere to pull from. Move them onto the in-cluster address before removing the upstream.

    Counted while the leader is full and the upstream is still held. If the field is absent, the count was not taken, and safeToDropUpstream is false, as with a nonzero count.

RegistryUpstream

Short names: rup

Scope: Cluster
Version: v1alpha1

Declares an additional registry the cluster routes to, on top of the primary one.

This is the supported way for a module, an administrator or a user to bring their own registry. A module creates the object with an ownerReference on itself, so the object is garbage-collected together with the module; a GitOps tool owning ModuleConfig never competes for field ownership, because every upstream is a separate object.

Traffic through these upstreams is always transit: the agent proxies the pull, substituting credentials and the certificate authority, and streams it through without writing to disk or to the cache. Cache space is finite and reserved for Deckhouse components, all the more so in an air-gapped cluster where the cache is the only source of images.

A match already claimed by another object, or one that would shadow the primary path, is rejected rather than silently merged. See status.conflict.

Objects are accepted in creation order, starting with the earliest created object. Settings from no more than 256 objects are applied at a time. The combined size of their spec fields in JSON format must not exceed 256 KiB. If adding the next object exceeds either limit, its settings are not applied. Creating a new object does not displace a previously accepted object from the node configuration.

With the module disabled or in Unmanaged mode, there is no agent on the nodes. The objects are then arbitrated by the same rules by node-manager, which writes status and hands the accepted ones to the nodes. The nodes apply them directly to the container runtime configuration: the runtime pulls from the upstream and its mirrors without a proxy. A match equal to the registry of a ModuleSource is accepted: on Immutable nodes the object takes over that registry together with its certificate authority, while nodes configured by bashible, apart from those with containerd v1 described below, keep the registry as the ModuleSource configures it. While the module configures no node — it is disabled or in Unmanaged mode — nodes with containerd v1 (Containerd) do not apply an upstream with path, or one whose registry is already configured on the node or named twice in the object. Such an object stays accepted, and the message of its Accepted condition names the node groups that do not apply it and why.

• spec

  object

  Required value

  • spec.match

    string

    Required value

    The registry host intercepted on the nodes.

    Pattern: ^[a-zA-Z0-9]([-a-zA-Z0-9]*[a-zA-Z0-9])?(\.[a-zA-Z0-9]([-a-zA-Z0-9]*[a-zA-Z0-9])?)*(:[0-9]{1,5})?$

    Length: 1..255

    Example:

    match: images.virtualization.example.com
    

  • spec.upstream

    object

    Required value

    Where the intercepted pulls are actually sent.

    • spec.upstream.auth

      object

      Credentials for the registry.

      Sensitive. Credentials are stored inline in the object rather than referenced from a Secret, so that the node agent stays self-sufficient while the API server is unreachable.

      • spec.upstream.auth.auth

        string

        Pre-encoded base64("username:password").

        Takes precedence over username and password when set. When both forms are set, they must name the same credentials.

        Maximum length: 4096

      • spec.upstream.auth.password

        string

        The password for basic authentication.

        Maximum length: 4096

      • spec.upstream.auth.username

        string

        The username for basic authentication.

        Maximum length: 1024

    • spec.upstream.ca

      string

      A PEM-encoded certificate authority bundle used to verify the registry when scheme is HTTPS.

      Only CERTIFICATE blocks are accepted, and each of them must parse.

      Maximum length: 65536

    • spec.upstream.host

      string

      Required value

      Registry URL, optionally with a port.

      An IPv6 address is written in brackets, followed by a port.

      Length: 1..255

      Example:

      host: registry-vendor.example.com
      

    • spec.upstream.mirrors

      array of objects

      Additional addresses serving the same content as the registry, used for failover.

      • spec.upstream.mirrors.auth

        object

        Credentials for the mirror.

        Sensitive. See upstream.auth.

        • spec.upstream.mirrors.auth.auth

          string

          Pre-encoded base64("username:password").

          Maximum length: 4096

        • spec.upstream.mirrors.auth.password

          string

          The password for basic authentication.

          Maximum length: 4096

        • spec.upstream.mirrors.auth.username

          string

          The username for basic authentication.

          Maximum length: 1024

      • spec.upstream.mirrors.ca

        string

        A PEM-encoded certificate authority bundle used to verify the mirror.

        Maximum length: 65536

      • spec.upstream.mirrors.host

        string

        Required value

        Mirror URL, optionally with a port.

        An IPv6 address is written in brackets, followed by a port.

        Length: 1..255

      • spec.upstream.mirrors.path

        string

        The repository prefix inside the mirror.

        Maximum length: 1024

      • spec.upstream.mirrors.scheme

        string

        The protocol used to reach the mirror.

        Default: HTTPS

        Allowed values: HTTP, HTTPS

    • spec.upstream.path

      string

      The repository prefix inside the registry.

      Maximum length: 1024

      Example:

      path: "/virt"
      

    • spec.upstream.scheme

      string

      The protocol used to reach the registry.

      Default: HTTPS

      Allowed values: HTTP, HTTPS

• status

  object
  • status.accepted

    boolean

    Whether the controller compiled this upstream into the per-node layout.

  • status.conditions

    array of objects

    The arbitration result with a machine-readable reason.

    • Accepted — whether the upstream is part of the per-node layout.
    • status.conditions.lastTransitionTime

      string

      Required value

      When the condition last changed.

    • status.conditions.message

      string

      A human-readable explanation.

    • status.conditions.observedGeneration

      integer

      The spec generation the condition was set for.

    • status.conditions.reason

      string

      Required value

      A machine-readable reason for the condition.

    • status.conditions.status

      string

      Required value

      The condition status.

      Allowed values: True, False, Unknown

    • status.conditions.type

      string

      Required value

      The condition type.

  • status.conflict

    string

    Reason for rejecting the object: the match value is already used by another object, match intercepts requests to the primary image registry (Primary), a field in spec is invalid, or adding the object exceeds the limit on the number of accepted objects or the combined size of their spec fields.

    Empty when the object is accepted.

  • status.observedGeneration

    integer

    The spec generation the controller last acted on.