Module registry: configuration

Schema version: 1

• settings

  object
  • settings.https

    object

    What certificate type to use.

    This parameter completely overrides the global.modules.https settings.

    Examples:

    https:
      mode: Disabled
    

    https:
      mode: OnlyInURI
    

    https:
      mode: CustomCertificate
      customCertificate:
        secretName: foobar
    

    https:
      mode: CertManager
      certManager:
        clusterIssuerName: letsencrypt
    

    • settings.https.certManager

      object

      Parameters for certmanager.

      • settings.https.certManager.clusterIssuerName

        string

        What ClusterIssuer to use for getting an SSL certificate (currently, letsencrypt, letsencrypt-staging, selfsigned are available; also, you can define your own).

        Default: letsencrypt

        Example:

        clusterIssuerName: letsencrypt
        

    • settings.https.customCertificate

      object

      Parameters for custom certificate usage.

      • settings.https.customCertificate.secretName

        string

        The name of the secret in the d8-system namespace to use with the registry ingress.

        This secret must have the kubernetes.io/tls format.

    • settings.https.mode

      string

      The HTTPS usage mode:

      • CertManager: The registry ingress is accessed over HTTPS using a certificate obtained from a clusterIssuer specified in the certManager.clusterIssuerName parameter.
      • CustomCertificate: The registry ingress is accessed over HTTPS using a certificate from the d8-system namespace.
      • Disabled: In this mode, the registry ingress can only be accessed over HTTP.
      • OnlyInURI: The registry ingress will work over HTTP (thinking that there is an external HTTPS load balancer in front of it that terminates HTTPS traffic). Load balancer should provide a redirect from HTTP to HTTPS.

      Default: CertManager

      Allowed values: Disabled, CertManager, CustomCertificate, OnlyInURI

  • settings.ingressClass

    string

    The class of the Ingress controller used for the registry.

    Optional. By default, the modules.ingressClass global value is used.

    Pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$

  • settings.mode

    string

    Whether the module manages how the cluster pulls images.

    • Unmanaged — the module manages nothing. No components are created and no node configuration is written; the cluster keeps pulling from the registry it was installed with. This is the default, so enabling the module changes nothing on its own.

      Asking for Unmanaged on a cluster the module was managing does not take effect instantly: image references all over the cluster name the in-cluster registry, and they move to the upstream one only as each module’s release is rendered again. So the module stops advertising its address at once and keeps serving it until nothing names it any more, which normally takes a few minutes. On a cluster that was installed with the module already managing, it also writes the registry it was fetching from back into the cluster’s registry credentials — otherwise nothing would remember where images come from, since the installer had put the in-cluster registry there. Until then its components are still running, and D8RegistryDrainStuck reports a withdrawal that cannot finish.

    • Managed — the module owns the pull path: it configures the container runtime on every node through its node agent, and optionally runs an in-cluster cache.

    There is no choice of implementation. A cluster that has never run the v1 implementation of this module (DP up to and including 1.77) always uses the current one; a cluster that has runs the v1 implementation until it is brought to its Unmanaged state, after which the migration completes on its own.

    Default: Unmanaged

    Allowed values: Managed, Unmanaged

  • settings.primary

    object

    The single authoritative source of Deckhouse component images. Applies only when mode is Managed.

    Additional registries are declared as separate RegistryUpstream resources rather than here: a module or a user bringing their own registry must not have to edit a ModuleConfig owned by someone else.

    • settings.primary.upstream

      object

      The registry to pull Deckhouse component images from.

      If omitted, the cluster is air-gapped: the in-cluster cache becomes authoritative and is populated with d8 mirror push. Omitting it requires storage.cache: true and storage.source.

      Example:

      upstream:
        host: registry.deckhouse.io
        path: "/deckhouse/ee"
        scheme: HTTPS
        auth:
          license: DECKHOUSE_LICENSE_KEY
      

      • settings.primary.upstream.auth

        object

        Credentials for the registry.

        • settings.primary.upstream.auth.license

          string

          The Deckhouse license key.

          A shorthand for the username/password pair used with registry.deckhouse.io. Mutually exclusive with them.

          Changing the license changes the registry credentials, so it goes through the same preflight probe as an address change: the new credentials are verified before the cluster is switched over, and the last known good ones are kept if the probe fails.

        • settings.primary.upstream.auth.password

          string

          The password for basic authentication.

        • settings.primary.upstream.auth.username

          string

          The username for basic authentication.

      • settings.primary.upstream.ca

        string

        A PEM-encoded certificate authority bundle used to verify the registry when scheme is HTTPS.

        If not specified, the system trust store is used.

      • settings.primary.upstream.host

        string

        Required value

        The registry host, optionally with a port.

        Examples:

        host: registry.deckhouse.io
        

        host: my-private-registry.com:5000
        

      • settings.primary.upstream.mirrors

        array of objects

        Additional addresses serving the same content as the primary registry, used for failover and load balancing.

        Mirrors are not separate sources: the cache holds one de-duplicated set regardless of which mirror the images came from. A registry with different content is an additional upstream and belongs in a RegistryUpstream resource.

        • settings.primary.upstream.mirrors.auth

          object

          Credentials for the registry.

          • settings.primary.upstream.mirrors.auth.license

            string

            The Deckhouse license key.

            A shorthand for the username/password pair used with registry.deckhouse.io. Mutually exclusive with them.

            Changing the license changes the registry credentials, so it goes through the same preflight probe as an address change: the new credentials are verified before the cluster is switched over, and the last known good ones are kept if the probe fails.

          • settings.primary.upstream.mirrors.auth.password

            string

            The password for basic authentication.

          • settings.primary.upstream.mirrors.auth.username

            string

            The username for basic authentication.

        • settings.primary.upstream.mirrors.ca

          string

          A PEM-encoded certificate authority bundle used to verify the mirror.

        • settings.primary.upstream.mirrors.host

          string

          Required value

          The mirror host, optionally with a port.

        • settings.primary.upstream.mirrors.path

          string

          The repository prefix inside the mirror.

        • settings.primary.upstream.mirrors.scheme

          string

          The protocol used to reach the mirror.

          Default: HTTPS

          Allowed values: HTTP, HTTPS

      • settings.primary.upstream.path

        string

        The repository prefix inside the registry.

        Example:

        path: "/deckhouse/ee"
        

      • settings.primary.upstream.scheme

        string

        The protocol used to reach the registry.

        Use HTTP only for insecure, trusted registries.

        Default: HTTPS

        Allowed values: HTTP, HTTPS

  • settings.storage

    object

    The in-cluster registry cache. Applies only when mode is Managed.

    • settings.storage.cache

      boolean

      Whether pulls go through the in-cluster cache on the master nodes.

      This is one of the two configuration axes; the other is whether primary.upstream is set. Together they cover every supported layout:

      • cache: false with an upstream — nodes pull straight from the upstream, no cache is deployed.
      • cache: true with an upstream — a pass-through cache, filled from the upstream.
      • cache: true without an upstream — air-gap: the cache is the only source of images and is filled with d8 mirror push.

      Turning the cache on or off is a safe, idempotent reconfiguration. Removing the upstream while the cache is on is the one transition with a condition: it takes effect only once the cache leader holds the whole expected image set, so nodes are never cut off.

      Default: false

    • settings.storage.garbageCollection

      object

      When the cache reclaims the disk taken by releases the cluster has moved past.

      Needed because nothing else ever removes anything: every release adds a slice of the repository, so a cluster that lives for years fills its store and then stops being able to pull.

      A collection does not interrupt serving: the replica keeps serving every image it holds. It does not start while the store is being filled or a d8 mirror push is in progress, and while it runs the replica stops filling its own store. A d8 mirror push started during a collection can fail and has to be retried. Only one replica collects at a time, so the others are unaffected.

      • settings.storage.garbageCollection.enabled

        boolean

        Whether the cache reclaims its disk on its own: on the schedule, once it holds a new image set, and when it refuses writes.

        Turning this off leaves the store to grow without bound, which only makes sense with a disk large enough that it never matters. A collection created by hand as a RegistryOperation object still runs.

        Default: true

      • settings.storage.garbageCollection.schedule

        string

        A five-field cron expression, in UTC.

        Defaults to a night hour. If the master node group has a maintenance window, the start of that window is used instead — that being a time the operator has already declared safe for disruption.

        Pattern: ^\s*\S+\s+\S+\s+\S+\s+\S+\s+\S+\s*$

        Examples:

        schedule: 17 3 * * *
        

        schedule: 0 2 * * Sun
        

    • settings.storage.size

      string

      How much the cache may hold on each master: the budget of its store, which the registry enforces on every write.

      Once the store holds this much, it refuses new writes — a fill, a replication, a d8 mirror push — and keeps serving everything it holds. It never deletes anything to make room; that is the garbage collection’s job.

      Independently of this value, the store never takes the space kept free on its filesystem: on a filesystem shared with the node, the kubelet’s soft eviction threshold (10% of the filesystem, at most 40 GiB) plus a margin (10%, at most 20 GiB); on a disk of the store’s own, the margin alone. So an unset or too large value cannot fill a master’s disk — but it leaves the reserve as the only limit, which the D8RegistryStoreUnbounded alert reports.

      The reserve is taken from the full size of the filesystem, while free space does not include the blocks the filesystem keeps for root. On ext4 these are 5% by default, so on a filesystem under 200 GiB the store and everything else on it get about 75% of the disk rather than 80%.

      Size it for three releases of the image set (the deployed one, the previous one kept for a rollback, and the next one) together with the modules the cluster runs and the catalog of modules the module sources offer. A replica copies the next release ahead of the update only while the room it has left, in the budget and on the filesystem above the reserve, is at least a fifth of the budget (or of the filesystem, when storage.size is not set); with an upstream this applies to every replica. In an air-gapped cluster the next release arrives on the leader with d8 mirror push and needs that room, and the other replicas copy it from the leader under the same rule. The store holds the union of the releases’ images: a layer they share is stored once. Make sure the value fits on the disk under the store above the reserve.

      Pattern: ^[0-9]+(\.[0-9]+)?(E|P|T|G|M|k|Ei|Pi|Ti|Gi|Mi|Ki)?$

      Example:

      size: 50Gi
      

    • settings.storage.source

      object

      The image set the cache is expected to hold.

      Required in an air-gapped cluster: with no upstream to fall back on, completeness must be decidable before the cache can be trusted as the only source.

      • settings.storage.source.bundleRef

        string

        The name of the image set, for example the bundle pushed with d8 mirror push.

        Example:

        bundleRef: d8-mirror-bundle
        

      • settings.storage.source.expectedDigests

        integer

        The number of distinct digests the set contains.

        Allowed values: 0 <= X

        Example:

        expectedDigests: 459
        

  • settings.whitelistSourceRanges

    array of strings

    A list of CIDR-formatted addresses allowed to connect to the registry’s publication endpoint, the Ingress that d8 mirror push writes through. If not specified, connections from any address are allowed.

    Applied by the Ingress only. The write port of the in-cluster cache behind it (4283/TCP on the control-plane nodes) does not filter by source address: it refuses any connection without a client certificate that the module’s certificate authority issued to the ingress controller or to the cache’s own pods. The cache’s read port (4281/TCP) does not accept writes at all, so the write port is the only one a push can land on.

    The ingress controller presents its certificate for any Ingress that names the module’s ingress client secret. If the controller lets an Ingress use a secret of another namespace (allow-cross-namespace-resources, enabled by default in ingress-nginx before 1.12), an Ingress of any namespace can reach the write port this way, outside this list; the write credentials are still checked. See What protects the ports on control-plane nodes.

    Example:

    whitelistSourceRanges:
    - 10.0.0.0/10
    - 192.168.0.0/16
    

    • Element of the array

      string

      Pattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(3[0-2]|[1-2][0-9]|[0-9]))?$