In clusters fully managed by Deckhouse Platform (Deckhouse Platform), you can configure the download paths:
- images of Deckhouse Platform component containers;
- images of containers from additional repositories (vendor repositories or your own).
The pull paths for container images are managed using the registry module.
Implementations of container image pull path management
Deckhouse Platform supports two implementations of the registry module for managing container image download paths:
- The current implementation. Used starting with Deckhouse Platform 1.78. Container image pull paths are configured through the
registryModuleConfig. The implementation is described in the “Current implementation” section. - The v1 implementation. Used in Deckhouse Platform up to and including 1.77. The Deckhouse Platform component registry is configured through the
deckhouseModuleConfig. The implementation is described in the “V1 implementation” section.
The two implementations never manage a cluster at the same time: only one of them is active at any given moment. A cluster that has never run the v1 implementation uses the current one right away. A cluster running the v1 implementation keeps using it until the conditions for switching to the current one are met — control is then handed over to the current implementation automatically, with no separate command. The switching conditions and step-by-step instructions for each mode of the v1 implementation are given in the “Migrating from the v1 implementation to the current one” section below.
Starting from version 1.78 Deckhouse Platform no longer uses the v1 implementation of the module. The cluster switches to the current implementation after the upgrade to Deckhouse Platform 1.78, and the preparation for that is done before the upgrade, on Deckhouse Platform 1.77.
Prepare your cluster in advance for the transition to the current implementation. The update to Deckhouse Platform 1.78, which no longer contains the v1 implementation, is blocked as long as the cluster is running in Proxy or Local mode of the v1 implementation, or in Direct mode without a configured registry in ModuleConfig.
You can find out which implementation of the module the cluster uses by checking whether the registry-v2-switch secret exists. To do this, follow the instructions.
Current implementation
In the current implementation, container image pull paths are configured through the registry ModuleConfig.
Modes of operation
In the current implementation, Deckhouse Platform supports the following container image pull path management modes (the mode is set with the mode parameter in the registry ModuleConfig):
Unmanaged(the default mode). Theregistrymodule does not manage the pull paths of Deckhouse Platform component images: the cluster pulls images from the registry it was installed with (the registry is specified during cluster bootstrap in thedeckhouseparameter in InitConfiguration).-
Managed. In this mode, theregistrymodule manages the pull paths for container images. The following independent settings are used in this mode:primary.upstream— the container image registry DP component images are pulled from. If the parameter is not set, the cluster is considered isolated (air-gapped). In an isolated cluster, the only source of container images is the in-cluster cache, which is filled with thed8 mirror pushcommand.storage.cache— controls the in-cluster cache on the master nodes. If the parameter is enabled, a container image registry is deployed on the master nodes, and all nodes pull images from it, with the upstream registry staying a fallback path while the cache is being filled.
The primary.upstream and storage.cache parameters together cover every supported configuration:
primary.upstream |
storage.cache |
What the cluster does |
|---|---|---|
| Set | false |
Nodes pull images directly from the upstream registry. Nothing is deployed on the master nodes |
| Set | true |
The cache acts as a pass-through: filled from the upstream registry on demand and ahead of time |
| Not set | true |
The cluster is isolated (air-gapped). The cache is the only source. The d8 mirror push command is used to fill it |
| Not set | false |
The configuration is rejected: nodes would have nowhere to pull images from |
You can reconfigure cache enablement and disablement, change the image storage location, and change your credentials at any time. However, when removing an upstream registry to transition to an isolated state, the module waits until the entire expected set of images appears in the intra-cluster cache—only then does it stop using the upstream registry. If the upstream registry stopped being used before the in-cluster cache held the whole expected image set, nodes could be left with no image source while the cache is being filled.
For more details on the architecture of the current implementation (the node agent, the in-cluster cache, leader replica election, how isolated clusters work), see the registry module documentation.
Installer behavior when no container image registry is configured
The installer creates the registry ModuleConfig with mode: Managed and primary.upstream pointing to the public Deckhouse Open/CE registry registry.deckhouse.io/deckhouse/ce, and warns about it in its output when the installation configuration specifies no container image registry in any of the following places:
- InitConfiguration
- The
registrysection of thedeckhouseModuleConfig - The
registryModuleConfig (the ModuleConfig is absent)
This applies only when the installation configuration contains a ClusterConfiguration and the container runtime is managed by Deckhouse Platform (defaultCRI is Containerd or ContainerdV2). In a Managed Kubernetes cluster (without ClusterConfiguration) and with defaultCRI: NotManaged, the installer does not create this ModuleConfig.
Under the same conditions, the installer rejects a configuration in which the registry ModuleConfig sets mode: Unmanaged (or omits mode, which means the same) or enabled: false and no registry is specified elsewhere. In this case, the nodes would have nowhere to pull images from. Specify the registry from which nodes pull images directly in one of the following ways:
- Via the
deckhouseparameter in InitConfiguration (imagesRepoandregistryDockerCfg). - By setting
mode: Managedwithprimary.upstreamin theregistryModuleConfig. A private or mirror registry populated withd8 mirror pushcan serve asprimary.upstream. To install an air-gapped cluster from an image bundle instead, setenabled: trueandmode: Managedwithstorage.cache: trueandstorage.source, withoutprimary.upstream, and pass the bundle to the installer with the--img-bundle-pathflag.
The installer also rejects a registry ModuleConfig that specifies primary or storage without mode: Managed, as the module schema does: in the default Unmanaged mode, these settings have no effect.
Garbage collection
Deckhouse Platform has a garbage collection mechanism for the container image registry. Each registry replica runs the operation on its own store. Garbage collection does not block image pulls from the store.
Garbage collection starts in the following cases:
- Once a day at night (at 3:17 a.m. Moscow Standard Time). Garbage collection is enabled by default. It is scheduled at night because filling the replica’s store is paused during the operation. A
d8 mirror pushcommand started during garbage collection can therefore fail. - When all replicas have received a new image set: after a Deckhouse Platform update or after the
registrymodule is enabled or updated. - When the image store refuses writes because it has exceeded its designated storage limit or reached the reserved free space threshold on its filesystem.
For details about when garbage collection starts and what happens during the operation, see the documentation for the registry module.
You can also start garbage collection manually.
If you need to disable or configure garbage collection, use the storage.garbageCollection parameter in the registry ModuleConfig.
Disabling garbage collection only makes sense if the storage uses a disk with sufficient capacity to accommodate unlimited storage growth.
To view the status and schedule of garbage collection, use the commands below.
-
List the recent collections — what started them, how they went, and how much they freed. A finished collection is kept for a day:
d8 k get registryoperation -o wide -
View the time of the last garbage collection on each replica and any errors, if any:
d8 k get registrystorage registry -o jsonpath='{.status.replicas}' | jq \ 'map({node, collectedAt, collectionError})' -
View the current garbage collection schedule:
d8 k get registrystorage registry -o jsonpath='{.spec.garbageCollection}' | jq
Garbage collection is the only mechanism that removes data from the registry. Each Deckhouse Platform release adds new images, so without garbage collection, the storage of a cluster that has been running for a long time will eventually fill up, and new images will no longer be added to it. However, for example, an isolated cluster with a full storage cannot be updated.
Collection removes the images of the releases the cluster has moved past. It keeps:
- the deployed release and the previous one, so that a rollback does not have to re-download anything;
- everything newer than the deployed release — an update in progress or, in an air-gapped cluster, a release pushed on purpose;
- every tag that is not a version: release channel names such as
stable, floating tags, anything pushed by hand. The collector cannot know what these mean, so it does not touch them.
It also removes blobs that no image references, such as those left by an interrupted fill or by a collection whose cleanup failed. A collection removes them even when there are no images to delete.
The collector is deliberately cautious. In an isolated cluster, deleting a blob that is still needed is unrecoverable without another d8 mirror push, while keeping an unneeded one merely costs disk space. So a run that cannot determine what to keep — for example, when no deployed release is found — does nothing at all.
For more information about how the garbage collector works, see the registry module documentation.
Running garbage collection manually
Garbage collection created manually runs even when automatic collection is disabled.
Before starting garbage collection manually, assess whether it is needed. See the “Is a collection worth running now?” section of the registry module FAQ.
To start garbage collection manually, create a RegistryOperation object of type GarbageCollection:
d8 k create -f - <<EOF
apiVersion: deckhouse.io/v1alpha1
kind: RegistryOperation
metadata:
generateName: gc-manual-
spec:
type: GarbageCollection
cause: Free space before pushing a new release.
EOF
It runs even when automatic collection is disabled. Replicas collect one at a time, exactly as on schedule. A replica starts only when its store is not being written to: while the cache is being filled or a d8 mirror push is in progress, the replica waits, and the reason is shown in status.replicas[].reason. A store that refuses writes is not waited for: a fill cannot go on until the collection frees room. A replica still held back an hour after the operation was accepted fails with the reason DeadlineExceeded. An operation created while another one is running fails with the reason AnotherOperationActive.
Follow the operation and see what it removed:
d8 k get registryoperation <NAME> -o jsonpath='{.status}' | jq \
'{phase, reason, message, garbageCollection}'
To cancel an operation, including one a replica is already running, set the cancel label on it:
d8 k label registryoperation <NAME> registry.deckhouse.io/cancel=true
What was deleted before the cancellation stays deleted.
If a replica’s syncer container stops answering while the replica collects or waits for its turn, for example because its master went NotReady, the replica fails with the reason ReplicaLost, so that the operation does not hold every collection after it for a day:
- A replica that was collecting, once it has not held the Lease
registry-storage-gcin thed8-systemnamespace for five minutes. - A replica that was waiting, once its node has not been
Readyfor ten minutes. A master that is back sooner keeps its replica’s turn.
A replica that waits on a node that stays Ready while its syncer does not answer is released by the cancel label, by the 24-hour timeout (the reason TimedOut), or by deleting the operation:
d8 k delete registryoperation <NAME>
Additional container image registries
primary.upstream is the only image registry configured in the registry ModuleConfig itself. It is used to pull DP component images. If you need additional registries (a vendor’s, one needed by some module, or your own), add them using the RegistryUpstream custom resource.
An example of adding an additional registry is given in the “Adding an additional container image registry” section.
Requirements for using the current implementation
Before enabling container image pull path management with the current implementation of the registry module, make sure the cluster meets the following requirements:
- Nodes use containerd or containerd v2, set with the
defaultCRIparameter in ClusterConfiguration. - The cluster is fully managed by Deckhouse Platform. The current implementation of the module does not work in Managed Kubernetes clusters — use switching to a third-party registry instead.
Examples of configuring the current implementation
If a module’s image wasn’t pulled again and the module wasn’t reinstalled while switching to the current implementation, follow the instructions to resolve the issue.
Configuring pull path management for Deckhouse Platform component images
To enable pull path management for DP component container images with the registry module, enable the module in Managed mode and specify the image registry details in primary.upstream. Example of the registry ModuleConfig:
apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
name: registry
spec:
version: 1
enabled: true
settings:
mode: Managed
primary:
upstream:
host: registry.deckhouse.io
path: /deckhouse/ee
scheme: HTTPS
auth:
license: <LICENSE_KEY> # Replace with your license key.
Instead of creating the ModuleConfig from scratch, you can use a ready-made one. The module publishes a ready-made configuration for your cluster, with the address, path, scheme, certificate authority, and credentials of the registry the cluster is already pulling images from.
To obtain a preconfigured ModuleConfig, run the following command:
d8 k -n d8-system get secret registry-suggested-config -o jsonpath='{.data.registry-mc\.yaml}' | base64 -d
Review it and apply it — this avoids mistakes that are easy to make when retyping these values by hand (a truncated path, the wrong certificate authority, and so on).
To see how the change takes effect, use the commands below.
-
View the state of the image registry configuration in Deckhouse Platform:
d8 k get registryconfig registry -o jsonpath='{.status}' | jq -
Verify the implementation of the registry configuration on the nodes to ensure that everything has been successfully configured and to determine which backends are currently active:
d8 k get registrynodes -o custom-columns=\ NODE:.metadata.name,APPLIED:.status.observedGeneration,OK:.status.reconciled,BACKENDS:.status.activeBackends
Enabling the in-cluster cache
The in-cluster cache uses the disks of the master nodes and works in both static and cloud clusters managed by Deckhouse Platform. In a cloud cluster, that is the master node’s root disk unless a separate disk is mounted at the cache directory (/opt/deckhouse/registry, or /var/lib/deckhouse/registry when the master NodeGroup has systemType: DeckhouseEngine). Size that disk for storage.size and the reserve the store keeps free for the node, on top of what the node itself uses. For details, see “Requirements for using the module” in the registry module documentation.
To enable the in-cluster cache, add the storage.cache setting to the registry ModuleConfig and specify the storage size:
spec:
settings:
mode: Managed
primary:
upstream:
host: registry.deckhouse.io
path: /deckhouse/ee
auth:
license: <LICENSE_KEY> # Replace with your license key.
storage:
cache: true
size: 50Gi
Nothing on the nodes is reconfigured: the container runtime already turns to the agent for any registry, and the agent starts using the cache first. The upstream registry is used as a fallback path. So missing data in the cache right after the cache is enabled means a slower pull, not a failed one.
To check how full the in-cluster cache is, use the following command:
d8 k get registrystorage registry -o jsonpath='{.status}' | jq '{phase,fill,leader,allReplicasFull}'
Turning the cache off is the same change in reverse, and just as safe. The blobs on disk are left untouched, and when the cache is turned on again, it is refilled from the data already accumulated. If you turn the in-cluster cache off and do not plan to turn it on again, remove the leftover cache data from the cluster nodes. For detailed instructions, see the registry module FAQ.
Switching the cluster into an air-gapped state
An isolated cluster has no upstream registry (it is not set in primary.upstream). For such a cluster, the cache is the only source of images. The d8 mirror push command is used to fill the storage. To verify that all required images are present in the cache, specify the expected set of images in the storage.source parameter
To move the cluster to air-gapped, follow these steps:
-
Pull the images on a machine that has internet access:
d8 mirror pull --license <LICENSE_KEY> ./d8-bundle -
Push them into the cluster through the publication endpoint:
PUSH_SECRET=$(d8 k -n d8-system get secret registry-storage-push -o json) d8 mirror push ./d8-bundle registry.example.com/system/deckhouse \ --registry-login "$(echo "$PUSH_SECRET" | jq -r .data.username | base64 -d)" \ --registry-password "$(echo "$PUSH_SECRET" | jq -r .data.password | base64 -d)"The publication endpoint (the
registrydomain built frompublicDomainTemplate,registry.example.comhere) exists only while the cache is enabled,publicDomainTemplateis set and thecert-managermodule is enabled. For details, see “Air-gapped clusters” in theregistrymodule documentation. -
Describe the expected image set in the cache and remove the
upstreamfrom theregistryModuleConfig:spec: settings: mode: Managed storage: cache: true size: 50Gi source: bundleRef: d8-mirror-bundle expectedDigests: 459
The upstream registry is removed from the nodes not at the moment the configuration is edited, but once the cache leader holds the whole expected image set — otherwise all nodes could be left with no image source. When the upstream is removed, the registry restarts, and the RegistryStorage resource may briefly report Failed: this is expected, wait for it to return to Ready. To check the status of the transition, use the following commands.
Check whether the cache leader holds the whole expected image set:
d8 k get registrystorage registry -o jsonpath='{.status}' | jq '{safeToDropUpstream,fill}'
Check whether the cluster is isolated (while effectiveUpstream is set, the cluster is using it; if it is empty, the cluster is isolated):
d8 k get registryconfig registry -o jsonpath='{.status.effectiveUpstream}' | jq
Adding an additional container image registry
Requests to additional registries are always routed through the node agent (credentials and the certificate authority are kept in one place), and images from them are not cached.
To add an additional image registry (not for DP system components), use the RegistryUpstream resource. Example:
apiVersion: deckhouse.io/v1alpha1
kind: RegistryUpstream
metadata:
name: virtualization-images
spec:
match: images.virtualization.example.com
upstream:
host: vendor.example.com
path: /virtualization
auth:
username: robot
password: <PASSWORD>
After creating the RegistryUpstream, check that it has been accepted. The object is rejected in the following cases:
- It conflicts with the primary registry or another object claiming the same
matchvalue. The object is rejected rather than merged. - It contains an invalid field, such as a
cathat is not a PEM certificate. - Adding the object exceeds either of the following limits:
- Settings from no more than 256 RegistryUpstream objects can be applied at a time.
- The combined size of the accepted objects’
specfields in JSON format must not exceed 256 KiB.
To check whether the object has been accepted, run the following command:
d8 k get registryupstreams -o custom-columns=\
NAME:.metadata.name,MATCH:.spec.match,ACCEPTED:.status.conditions[0].status,REASON:.status.conditions[0].reason
In the example above, after the additional registry is added, image pull requests for images.virtualization.example.com are routed by the agent on every node to vendor.example.com/virtualization, with the credentials and certificate authority held by the cluster rather than by each workload individually. Nothing on the nodes is reconfigured for this.
Pulling from a private container image registry without declaring it
To load images from a storage module that is unknown to the module, no additional declarations are required. The node agent proxies such requests untouched, along with whatever credentials the request already carried. So an ordinary imagePullSecret works exactly as it does in a cluster where the module was never enabled.
To create a secret with the credentials of the private container image registry, run:
d8 k create secret docker-registry my-private-registry \
--docker-server=private.example.com \
--docker-username=robot \
--docker-password=<PASSWORD>
To use this secret when pulling images, specify it in the pod’s imagePullSecrets:
apiVersion: v1
kind: Pod
metadata:
name: example
spec:
imagePullSecrets:
- name: my-private-registry
containers:
- name: app
image: private.example.com/team/app:v1
It is worth creating a RegistryUpstream for the registry only if you want the cluster to hold the credentials instead of every workload doing so individually, or if the registry needs a certificate authority the nodes do not have.
Turning off pull path management
To turn off pull path management, set mode: Unmanaged in the registry ModuleConfig:
spec:
settings:
mode: Unmanaged
After mode: Unmanaged is set, pull path management through the registry module is turned off. The cluster resumes downloading images from the container registry specified in the deckhouse-registry secret—that is, from the same source from which images were downloaded before the module was enabled. Cache data on the master nodes is deliberately preserved: when pull path management is turned on again, the cache is topped up only with the images missing from the container image registry rather than filled from scratch. If the cache is no longer needed, free up the space it occupies by following the instructions in the registry module documentation.
V1 implementation
The v1 implementation of the registry module described below was used in Deckhouse Platform up to and including 1.77. Deckhouse Platform 1.78 no longer contains it. For new clusters, and for existing ones at the first opportunity, use the current implementation.
In the v1 implementation, Deckhouse Platform supports several management modes for the platform component registry settings: Unmanaged, Direct, Proxy, and Local. In Direct, Proxy, and Local modes, a fixed virtual address is used to access the Deckhouse Platform component registry. This eliminates the need to restart all control plane components and re-download images when the platform component registry changes.
In Unmanaged mode, no virtual address is used. The external registry is accessed directly: if the registry changes, all Deckhouse Platform components are restarted.
Switching between modes and registries is done via the deckhouse ModuleConfig. The switch occurs automatically (see the switching examples below for more details).
The architecture of the v1 implementation’s modes is described in the “Registry module” section.
Features of the Deckhouse Platform component registry settings management modes:
Direct: Provides direct access to an external registry via the fixed virtual addressregistry.d8-system.svc:5001/system/deckhouse. The fixed address prevents images from being re-downloaded and components from being restarted when registry parameters change.Proxy: Uses an internal caching proxy registry that accesses an external registry. The caching proxy registry runs on control-plane (master) nodes. This mode reduces the number of requests to the external registry by caching images. The internal registry is accessed via the fixed virtual addressregistry.d8-system.svc:5001/system/deckhouse, similar to theDirectmode.Local: Uses a local internal registry, with the registry running on control-plane (master) nodes. This mode allows the cluster to operate in an isolated environment. The internal registry is accessed via the fixed virtual addressregistry.d8-system.svc:5001/system/deckhouse, similar to theDirectandProxymodes.Unmanaged(configurable mode): Works without an internal registry. No virtual address is used. The external registry is accessed directly.
This document refers to the configurable Unmanaged mode (of the v1 implementation of the registry module) and the non-configurable Unmanaged mode (the fully deprecated format, without the registry module).
The non-configurable Unmanaged mode does not use the registry module at all (for example, in Managed Kubernetes clusters). The Deckhouse Platform component registry configuration parameters are set during cluster installation or, in a deployed cluster, using the helper change registry utility (deprecated).
Restrictions on Deckhouse Platform component registry settings
There are a number of restrictions and considerations regarding cluster installation, the v1 implementation’s usage conditions, and switching between its modes.
Restrictions during cluster installation
The following restrictions apply when installing a cluster:
- A new cluster cannot be installed with the v1 implementation: the installer refuses
ProxyandLocalin thedeckhouseModuleConfig. ADirectsection there is read only as the registry address, and the cluster is installed inUnmanagedmode. To install a cluster whose pull path theregistrymodule manages, use the current implementation. - To launch a cluster in the non-configurable
Unmanagedmode (Legacy, without using theregistrymodule), registry parameters must be specified in InitConfiguration.
Operating conditions restrictions
To manage Deckhouse Platform component registry settings using the v1 implementation, the following conditions must be met:
- CRI containerd or containerd v2 must be used on cluster nodes. To configure CRI, refer to the ClusterConfiguration configuration.
- The cluster must be fully managed by Deckhouse Platform. In Managed Kubernetes clusters, managing Deckhouse Platform component registry settings via the
registrymodule is not available.
Restrictions on mode switching
Mode switching restrictions are as follows:
- Changing registry parameters and switching modes is only available after the bootstrap phase is fully complete.
- For the first switch, you must migrate user registry configurations. For more details, see the “Registry Module: FAQ” section.
- Switching to the non-configurable
Unmanagedmode (without using theregistrymodule) is only available from the configurableUnmanagedmode. For more details, see the “Registry Module: FAQ” section.
Examples of switching between the v1 implementation’s modes
If a module’s image wasn’t pulled and the module wasn’t reinstalled during the switch, follow the instructions to resolve the issue.
Switching to Unmanaged mode
To switch an already running cluster to Unmanaged mode, follow these steps:
Changing the registry in Unmanaged mode will restart all Deckhouse Platform components.
-
Make sure the
registrymodule is enabled and running. To do this, execute the following command:d8 k get module registry -o wideExample output:
NAME WEIGHT ... PHASE ENABLED DISABLED MESSAGE READY registry 38 ... Ready True True -
Ensure the Deckhouse Platform queue is empty and has no errors:
d8 system queue listExample output:
Summary: - 'main' queue: empty. - 107 other queues (0 active, 107 empty): 0 tasks. - no tasks to handle. -
Set the
Unmanagedmode settings in the ModuleConfigdeckhouse. If you are using a registry other thanregistry.deckhouse.io, refer to thedeckhousemodule configuration for correct setup.Configuration example:
apiVersion: deckhouse.io/v1alpha1 kind: ModuleConfig metadata: name: deckhouse spec: version: 1 enabled: true settings: registry: mode: Unmanaged unmanaged: imagesRepo: registry.deckhouse.io/deckhouse/ee scheme: HTTPS license: <LICENSE_KEY> # Replace with your license key. -
Check the registry mode switch status in the
registry-statesecret using the instruction.Example output:
conditions: # ... - lastTransitionTime: "..." message: "" reason: "" status: "True" type: Ready hash: .. mode: Unmanaged target_mode: Unmanaged -
If you need to disable automatic Deckhouse Platform updates from the configured image registry, remove the
releaseChannelparameter from thedeckhousemodule configuration. After that, automatic platform updates will be disabled, and you will need to manage the Deckhouse Platform version manually.
If you need to switch to the legacy registry management method (without the registry module), see the instruction.
Managing the Deckhouse Platform component image registry without the registry module is a deprecated management format.
Migrating from the v1 implementation to the current one
This section describes the transition from the v1 implementation of the registry module to the current one. A cluster that does not use the registry module at all (the fully deprecated format) has nothing to hand over, since the v1 implementation never configured its nodes: it needs no preparation for the upgrade to Deckhouse Platform 1.78 and keeps pulling images from the registry it was installed with.
The migration is the handover of the pull path from the v1 implementation to the current one.
It does not need to be started manually, and there is no separate command for it: the registry module takes over control automatically after the cluster is upgraded to Deckhouse Platform 1.78. The only condition is that by then the v1 implementation must have released the pull path. Both implementations configure the same thing on every node — which container image registry the container runtime pulls images from and with which credentials — so they never manage a cluster at the same time.
Determining the migration method and steps
What needs to be done to hand over pull path management to the current implementation of the registry module, and when, depends on the mode the v1 implementation is running in. The mode is set in the settings.registry.mode parameter of the deckhouse ModuleConfig. To view the mode, use the following command:
d8 k get mc deckhouse -o jsonpath='{.spec.settings.registry.mode}'
Empty output means Unmanaged mode: the registry settings of this cluster have never been set. Such a cluster needs no preliminary actions for the migration — the handover happens automatically.
The actions required to migrate from the v1 implementation to the current one are described in the table:
| Mode of the v1 implementation | What to do | When |
|---|---|---|
Unmanaged |
Nothing — the handover happens on its own | — |
Direct |
Configure the registry ModuleConfig: set mode: Managed and primary.upstream |
Before upgrading to Deckhouse Platform 1.78 |
Proxy |
Switch the cluster to Unmanaged |
Before upgrading to Deckhouse Platform 1.78 |
Local |
Follow the separate procedure for air-gapped clusters | Before upgrading to Deckhouse Platform 1.78 |
Deckhouse Platform 1.78 no longer contains the v1 implementation of the module. The cluster switches to the current implementation after the upgrade to Deckhouse Platform 1.78, and the preparation for that is done before the upgrade, on Deckhouse Platform 1.77.
Prepare your cluster in advance for the transition to the current implementation. The update to Deckhouse Platform 1.78, which no longer contains the v1 implementation, is blocked as long as the cluster is running in Proxy or Local mode of the v1 implementation, or in Direct mode without a configured registry in ModuleConfig.
When the v1 implementation is used in Direct mode, no mode switch is needed. The registry ModuleConfig settings are deliberately accepted already in Deckhouse Platform 1.77: written before the upgrade, they are stored without effect and take effect on the module’s first reconciliation after the upgrade.
Checking which implementation the cluster uses
When the registry module takes over pull path management, it records that in the registry-v2-switch secret. Check whether the secret exists:
d8 k -n d8-system get secret registry-v2-switch >/dev/null 2>&1 \
&& echo "current implementation" || echo "v1 implementation"
If the cluster is still on the v1 implementation, the module reports the reason on every reconciliation, and the D8RegistryMigrationPending alert fires in the cluster. To see which actions are required, use the following command:
d8 k -n d8-system get secret registry-state -o jsonpath='{.data.state}' | base64 -d | head
Migrating from Unmanaged mode
In Unmanaged mode, the v1 implementation does not manage the pull path, so no preliminary preparation for switching to the current implementation is required — the handover happens automatically.
The migration procedure to the current implementation in this case is as follows:
-
If the cluster on the v1 implementation is being switched to
Unmanagedfrom another mode, wait for the transition to complete. The status must showmode: Unmanagedwith no pending target mode.To check, use the following command:
d8 k -n d8-system get secret registry-state -o jsonpath='{.data.state}' | base64 -d | head -
Upgrade the cluster to Deckhouse Platform 1.78 (or, if it has already been upgraded, wait for the next reconciliation). The module takes over image pull path management automatically, and the pull source does not change. The default mode of the
registrymodule in the current implementation is alsoUnmanaged, so the cluster keeps pulling images from the same registry as before. On containerd v1 nodes, however, the takeover restarts containerd node by node if the v1 implementation left a node registry configuration, and enablingmode: Managedin the next step restarts it (a second time, if the takeover already did; for details, see “Does the migration restart containerd?”). -
To have the module manage the pull paths, set
mode: Managedin theregistryModuleConfig and specify the registry to pull images from. A ready-made configuration for your cluster is published in theregistry-suggested-configsecret. For a configuration example, see the “Configuring pull path management for Deckhouse Platform component images” section.
Migrating from Direct mode
In Direct mode of the v1 implementation, nodes pull images through an in-cluster address served by the v1 implementation’s proxy. The current implementation serves the same address, so the migration is a direct handover of the address: no switch through Unmanaged is needed, and no components are restarted.
The migration procedure to the current implementation in this case is as follows:
- Configure the
registryModuleConfig before the upgrade — without this configuration the upgrade is blocked. To fill in theregistryModuleConfig, take the values from theregistry.directsection of thedeckhouseModuleConfig:- split the
imagesRepovalue from thedeckhouseModuleConfig intohostandpathand specify them in the corresponding fields of theregistryModuleConfig; - use the credentials specified in the
deckhouseModuleConfig: the samelicense(orusername/password), andcaif the registry requires one.
Example of the
registryModuleConfig for the switch:apiVersion: deckhouse.io/v1alpha1 kind: ModuleConfig metadata: name: registry spec: enabled: true version: 1 settings: mode: Managed primary: upstream: scheme: HTTPS host: registry.deckhouse.io path: /deckhouse/ee auth: license: <LICENSE_KEY> # Replace with your license key.The v1 implementation stores these settings but does not act on them, so nothing changes in the cluster until the upgrade.
- split the
-
Upgrade the cluster to Deckhouse Platform 1.78. The handover of container image pull path management happens on the next reconciliation, and image pulls keep working throughout: the Service and the proxy of the v1 implementation keep serving the in-cluster address until the agent of the current implementation takes it over on every node, and only then are they removed.
-
Track the progress of the handover using the following commands.
-
Check whether path management has been handed over to the current implementation:
d8 k -n d8-system get secret registry-v2-switch >/dev/null 2>&1 && echo "handed over" -
Check the state of the image registry nodes:
d8 k get registrynode -o custom-columns='NODE:.metadata.name,READY:.status.reconciled,SERVING:.status.proxyListening'
The handover is recorded a few minutes after the new version starts, the agent appears on the nodes a few minutes later, and the objects of the v1 implementation are removed shortly after that. Image pulls keep working at each of these stages.
-
Migrating from Proxy mode
When the v1 implementation is used in Proxy mode, a proxy of its own, with its own certificates, is deployed on every node. The current implementation cannot adopt this state. So the cluster must first be switched to Unmanaged, and this can only be done before the upgrade.
The migration procedure to the current implementation in this case is as follows:
-
In the
deckhouseModuleConfig, setregistry.mode: Unmanaged, keeping the same registry address and credentials (for an example of thedeckhouseModuleConfig, see the “Switching to Unmanaged mode” section). All nodes are reconfigured to pull directly from the external registry, so the caching provided inProxymode is lost until the last step. -
Wait for the transition to complete —
mode: Unmanagedwith no pending target mode. A cluster caught mid-transition cannot be migrated; the status shows which mode it is still switching to:d8 k -n d8-system get secret registry-state -o jsonpath='{.data.state}' | base64 -d | head -
Upgrade the cluster to Deckhouse Platform 1.78. The handover happens on the next reconciliation, and the pull source does not change: in
Unmanagedmode, the current implementation does not manage the pull path either. On containerd v1 nodes, however, the takeover restarts containerd node by node, and enablingmode: Managedin the next step restarts it again (for details, see “Does the migration restart containerd?”). -
To get in-cluster caching back, set
mode: Managedwithstorage.cache: trueand the same external registry. Note that the cache of the current implementation is built differently fromProxymode of the v1 implementation: a single store with replicas on the master nodes instead of a proxy on every node. Before enabling it on a cluster with little free disk space on the master nodes, read how the cache is filled and cleaned up.
Migrating an air-gapped cluster from Local mode
In Local mode of the v1 implementation, the cluster has no external image registry — its role is played by a registry inside the cluster itself. So the standard migration procedure does not work here, since it goes through Unmanaged mode, where every node pulls images directly from an external registry, and such a cluster has nowhere to pull them from.
An external registry has to be connected to the cluster for the duration of the migration. It runs in the same cluster, but outside the Deckhouse Platform namespaces. The cluster is switched to it and upgraded, and once the registry of the current implementation is filled with images, the temporary registry is removed.
Before starting, make sure the master nodes have free disk space for four image sets: at the peak of the migration, three copies of the set exist at the same time — the Local store, the temporary registry, and the registry of the current implementation being filled — and the space for the fourth set is headroom the node must not run out of. The detailed disk space calculation and the verification procedure are described in the “Disk requirements” section of the registry module documentation.
Migration procedure:
-
Run a temporary external registry in its own namespace, outside Deckhouse Platform management. Any container image registry implementation will do, but it must serve TLS with a certificate the cluster can verify, and Deckhouse Platform must not manage it: whatever happens to the module’s objects, the temporary container image registry must keep working.
In addition, the container image registry must be reachable at the same address from two places: in step 3, nodes pull images from it, and in step 6, the
registrymodule’s syncer running in a pod reads it. The certificate must cover the chosen address. Options tested on a single cluster:Address From a node From a pod A hostNetworkport on the node’s own IP (<NODE_IP>:5000)works works A Service name ( <SERVICE>.<NAMESPACE>.svc:<PORT>)does not resolve works A NodePort on the node’s IP works does not work: operation not permittedUse the first option: run the container image registry with
hostNetwork: trueon one node, address it by that node’s IP, and add that IP to the certificate’s SAN — then the whole migration goes through a single address. A Service name looks neater, but stops working in step 3, because the node’s container runtime does not resolve names through the cluster DNS. - Load the image set of the current version into the temporary registry:
d8 mirror pullon a machine with internet access, thend8 mirror pushinto the temporary registry. - In the
deckhouseModuleConfig, setregistry.mode: Unmanagedwith the address, CA certificate, and credentials of the temporary registry. The data of theLocalstore stays where it is, in the/opt/deckhouse/registrydirectory on the master nodes. -
Make sure the transition is complete and images are really pulled from the temporary registry (
mode: Unmanagedwith no pending target mode in theregistry-statesecret):d8 k -n d8-system get secret registry-state -o jsonpath='{.data.state}' | base64 -d | head - Upgrade the cluster to Deckhouse Platform 1.78. The handover happens on the next reconciliation, and the cluster keeps pulling images from the temporary registry throughout. On containerd v1 nodes the takeover restarts containerd node by node, and enabling
mode: Managedin the next step restarts it again (see Does the migration restart containerd?). - Enable the current implementation: set
mode: Managedwithstorage.cache: true, specify the temporary registry asprimary.upstream— and addstorage.sourcein the same edit (without it, the next step is rejected). The registry of the current implementation uses the same host path thatLocalused, so the images already on disk are not downloaded again. -
Wait until the storage reports that it holds the whole image set (
phase: ReadyandsafeToDropUpstream: true), and removeprimary.upstreamfrom theregistryModuleConfig. The cluster is isolated again — now on the current implementation:d8 k get registrystorage registry -o jsonpath='{.status.phase} {.status.safeToDropUpstream}{"\n"}'Removing the upstream restarts the registry, and the RegistryStorage resource may briefly report
Failed: this is expected, wait for it to return toReady. - Delete the temporary registry and reclaim its disk space.
Migration to registry management format using the registry module
This procedure applies to Deckhouse Platform 1.77 and earlier only: it moves a cluster from the deprecated format (without the registry module, configured only via InitConfiguration) to the v1 implementation of the module, which Deckhouse Platform 1.78 no longer contains. It is not a step of the migration to the current implementation: a cluster in the deprecated format needs no preparation for the upgrade to Deckhouse Platform 1.78. See the “Migrating from the v1 implementation to the current one” section.
During the migration, containerd v1 will be switched to the new registry configuration scheme. containerd v2 uses the new scheme by default. For more details, see the section describing configuration methods.
To view the container runtime type used by default on cluster nodes (in NodeGroups), use the following command:
d8 system edit cluster-configuration
The container runtime type is specified in the defaultCRI parameter.
If containerd v2 is used as the container runtime, follow the containerd v2 migration instructions.
If containerd v1 is used as the container runtime, you can choose one of the following methods to migrate to the registry module’s image registry management format:
- Without switching to containerd v2. In this case, follow the instructions for containerd v1.
-
With switching to containerd v2. In this case:
- Verify whether you can switch to containerd v2, and if so, perform the switch.
- After migrating, follow the instructions for containerd v2.
For containerd v2
-
Switch to using the
registrymodule. To do this, specify theUnmanagedmode parameters in thedeckhouseModuleConfig. If you are using a registry other thanregistry.deckhouse.io, refer to thedeckhousemodule configuration for correct setup.You can view the current registry settings using the following command:
d8 k -n d8-system exec -it svc/deckhouse-leader -c deckhouse -- deckhouse-controller global values | yq e '.modulesImages.registry' -Specify these settings when configuring the
Unmanagedmode:apiVersion: deckhouse.io/v1alpha1 kind: ModuleConfig metadata: name: deckhouse spec: version: 1 enabled: true settings: registry: mode: Unmanaged unmanaged: imagesRepo: registry.deckhouse.io/deckhouse/ee scheme: HTTPS license: <LICENSE_KEY> # Replace with your license key. -
Wait for the switch to complete. Example status output:
conditions: # ... - lastTransitionTime: "..." message: "" reason: "" status: "True" type: Ready hash: .. mode: Unmanaged target_mode: Unmanaged
If you need to add configurations for an additional image repository, refer to the section “New way to add configuration for an additional registry”.
For containerd v1
- During the switch, containerd v1 will be restarted.
- During the switch, containerd v1 will be migrated to the new Deckhouse Platform component registry configuration scheme.
- During the switch, custom registry configurations for containerd v1 will be temporarily unavailable.
-
Make sure that nodes with containerd v1 do not have any custom registry configurations located in the
/etc/containerd/conf.ddirectory. -
If configurations are present, you need to migrate to the new registry configuration format in containerd. To do this, add new configuration files to the
/etc/containerd/registry.ddirectory. These configurations will take effect after switching to theregistrymodule. To add configurations, prepare a NodeGroupConfiguration. For more details, see the section with a description of configuration methods. Example:apiVersion: deckhouse.io/v1alpha1 kind: NodeGroupConfiguration metadata: name: containerd-additional-config-auth.sh spec: # The step can be arbitrary, as restarting the containerd service is not required. weight: 0 bundles: - '*' nodeGroups: - "*" content: | # Copyright 2023 Flant JSC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. REGISTRY_URL=private.registry.example mkdir -p "/etc/containerd/registry.d/${REGISTRY_URL}" bb-sync-file "/etc/containerd/registry.d/${REGISTRY_URL}/hosts.toml" - << EOF [host] [host."https://${REGISTRY_URL}"] capabilities = ["pull", "resolve"] [host."https://${REGISTRY_URL}".auth] username = "username" password = "password" EOF -
Apply the NodeGroupConfiguration. Wait until the configuration files appear in the
/etc/containerd/registry.ddirectory on all nodes. -
Verify that the configurations are working correctly. To do this, use the following command:
# For HTTPS. ctr -n k8s.io images pull --hosts-dir=/etc/containerd/registry.d/ private.registry.example/registry/path:tag # For HTTP. ctr -n k8s.io images pull --hosts-dir=/etc/containerd/registry.d/ --plain-http private.registry.example/registry/path:tag -
Switch to using the
registrymodule. To do this, specify theUnmanagedmode parameters in thedeckhouseModuleConfig. If you are using a registry other thanregistry.deckhouse.io, refer to thedeckhousemodule configuration for correct setup.You can view the current registry settings using the following command:
d8 k -n d8-system exec -it svc/deckhouse-leader -c deckhouse -- deckhouse-controller global values | yq e '.modulesImages.registry' -Specify these settings when configuring the
Unmanagedmode:apiVersion: deckhouse.io/v1alpha1 kind: ModuleConfig metadata: name: deckhouse spec: version: 1 enabled: true settings: registry: mode: Unmanaged unmanaged: imagesRepo: registry.deckhouse.io/deckhouse/ee scheme: HTTPS license: <LICENSE_KEY> # Replace with your license key. -
After applying, wait for the following message in the switch status:
Example output:
conditions: # ... - lastTransitionTime: "2025-08-13T15:22:34Z" message: | Check current nodes configuration 2/2 node(s) Unready: - master-0: has custom toml merge containerd configuration - worker-5e389be0-578df-s5sm5: has custom toml merge containerd configuration reason: Processing status: "False" type: ContainerdConfigPreflightReadyThis message means that there are old registry configurations on the nodes located in the
/etc/containerd/conf.ddirectory. The switch to the new containerd configuration is currently blocked. To allow the switch, you need to remove the old configuration files. -
Remove the old configuration files to allow switching to the
registrymodule. To do this, create a NodeGroupConfiguration. Example of a NodeGroupConfiguration manifest:apiVersion: deckhouse.io/v1alpha1 kind: NodeGroupConfiguration metadata: name: containerd-additional-config-auth-delete.sh spec: # This step must be completed before '032_configure_containerd.sh'. weight: 0 bundles: - '*' nodeGroups: - "*" content: | # Copyright 2023 Flant JSC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. file="/etc/containerd/conf.d/old-config.toml" [ -f "$file" ] && rm -f "$file" -
After removing the old configurations, make sure that the switch has resumed. Example of the switch status:
conditions: # ... - lastTransitionTime: "2025-08-13T16:42:09Z" message: "" reason: "" status: "True" type: ContainerdConfigPreflightReady -
Wait for the switch to complete. Example of the switch status:
conditions: # ... - lastTransitionTime: "..." message: "" reason: "" status: "True" type: Ready hash: .. mode: Unmanaged target_mode: Unmanaged -
Delete the NodeGroupConfiguration created in the step for deleting old configuration files:
d8 k delete nodegroupconfiguration containerd-additional-config-auth-delete.shTo verify that NodeGroupConfiguration has been deleted, use the command:
d8 k get nodegroupconfigurationThe list should not contain the NodeGroupConfiguration to be deleted (for this example,
containerd-additional-config-auth-delete.sh).
Migration to the deprecated registry management format (without the registry module)
- This is a deprecated Deckhouse Platform component registry management format.
- During the switch, containerd v1 will be restarted.
- During the switch, containerd v1 will be migrated to the legacy registry configuration scheme.
- During the switch, custom registry configurations for containerd v1 will be temporarily unavailable.
-
Switch the
registrymodule to configurableUnmanagedmode. If you are using a registry other thanregistry.deckhouse.io, refer to thedeckhousemodule configuration for correct setup.Example configuration:
apiVersion: deckhouse.io/v1alpha1 kind: ModuleConfig metadata: name: deckhouse spec: version: 1 enabled: true settings: registry: mode: Unmanaged unmanaged: imagesRepo: registry.deckhouse.io/deckhouse/ee scheme: HTTPS license: <LICENSE_KEY> # Replace with your license key. -
Check the switch status using the instruction. Example output:
conditions: # ... - lastTransitionTime: "..." message: "" reason: "" status: "True" type: Ready hash: .. mode: Unmanaged target_mode: Unmanaged -
Switch the registry to the non-configurable
Unmanagedmode (withoutregistrymodule). Example configuration:apiVersion: deckhouse.io/v1alpha1 kind: ModuleConfig metadata: name: deckhouse spec: version: 1 enabled: true settings: registry: mode: Unmanaged -
Check the switch status using the instruction. Example output:
conditions: # ... - lastTransitionTime: "..." message: "" reason: "" status: "True" type: Ready hash: .. mode: Unmanaged target_mode: Unmanaged -
If containerd v1 is used and custom registry configurations are applied in the cluster, they must be replaced with the old format. To do this, prepare the registry configurations in the old format. These configurations do not need to be applied at this stage. Example configuration:
apiVersion: deckhouse.io/v1alpha1 kind: NodeGroupConfiguration metadata: name: containerd-additional-config-auth.sh spec: # To add a file before the '032_configure_containerd.sh' step. weight: 31 bundles: - '*' nodeGroups: - "*" content: | # Copyright 2023 Flant JSC # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. REGISTRY_URL=private.registry.example mkdir -p /etc/containerd/conf.d bb-sync-file /etc/containerd/conf.d/additional_registry.toml - << EOF [plugins] [plugins."io.containerd.grpc.v1.cri"] [plugins."io.containerd.grpc.v1.cri".registry] [plugins."io.containerd.grpc.v1.cri".registry.mirrors] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."${REGISTRY_URL}"] endpoint = ["https://${REGISTRY_URL}"] [plugins."io.containerd.grpc.v1.cri".registry.configs] [plugins."io.containerd.grpc.v1.cri".registry.configs."${REGISTRY_URL}".auth] username = "username" password = "password" # OR auth = "<BASE64_AUTH_STRING>" EOF -
Delete the
registry-bashible-configsecret. This will trigger containerd v1 to switch back to the legacy registry format:d8 k -n d8-system delete secret registry-bashible-config -
After deletion, wait for the switch to complete. Use the instruction to track the progress. Example output:
conditions: # ... - lastTransitionTime: "..." message: "" reason: "" status: "True" type: Ready hash: .. mode: Unmanaged target_mode: Unmanaged -
If containerd v1 is used, apply the previously prepared NodeGroupConfiguration with custom registry configurations.
-
Disable the
registrymodule. Example:apiVersion: deckhouse.io/v1alpha1 kind: ModuleConfig metadata: name: registry spec: enabled: false settings: {} version: 1
Checking the registry mode switch status in the v1 implementation
The status of the Deckhouse Platform component registry mode switch in the v1 implementation can be retrieved using the following command:
d8 k -n d8-system -o yaml get secret registry-state | yq -C -P '.data | del .state | map_values(@base64d) | .conditions = (.conditions | from_yaml)'
Example output:
conditions:
- lastTransitionTime: "2025-07-15T12:52:46Z"
message: 'registry.deckhouse.io: all 157 items are checked'
reason: Ready
status: "True"
type: RegistryContainsRequiredImages
- lastTransitionTime: "2025-07-11T11:59:03Z"
message: ""
reason: ""
status: "True"
type: ContainerdConfigPreflightReady
- lastTransitionTime: "2025-07-15T12:47:47Z"
message: ""
reason: ""
status: "True"
type: TransitionContainerdConfigReady
- lastTransitionTime: "2025-07-15T12:52:48Z"
message: ""
reason: ""
status: "True"
type: InClusterProxyReady
- lastTransitionTime: "2025-07-15T12:54:53Z"
message: ""
reason: ""
status: "True"
type: DeckhouseRegistrySwitchReady
- lastTransitionTime: "2025-07-15T12:55:48Z"
message: ""
reason: ""
status: "True"
type: FinalContainerdConfigReady
- lastTransitionTime: "2025-07-15T12:55:48Z"
message: ""
reason: ""
status: "True"
type: Ready
mode: Direct
target_mode: Direct
The output displays the status of the switch process. Each condition can have a status of True or False, and may contain a message field with additional details.
Description of conditions:
| Condition | Description |
|---|---|
ContainerdConfigPreflightReady |
State of the containerd configuration preflight check. Verifies there are no custom containerd auth configurations on the nodes |
TransitionContainerdConfigReady |
State of preparing the containerd configuration for the new mode. Verifies that the configuration contains both the old and new mode settings |
FinalContainerdConfigReady |
State of finalizing the switch to the new containerd mode. Verifies that the containerd configuration has been successfully applied and contains only the new mode settings |
DeckhouseRegistrySwitchReady |
State of switching Deckhouse Platform and its components to use the new registry. True means Deckhouse Platform successfully switched and is ready to operate |
InClusterProxyReady |
State of In-Cluster Proxy readiness. Checks that the In-Cluster Proxy has started successfully and is running |
CleanupInClusterProxy |
State of cleaning up the In-Cluster Proxy if it is not needed in the selected mode. Verifies that all related resources have been removed |
NodeServicesReady |
State of Node Services Manager and Static-Pod registry readiness. Verifies that the Node Services Manager is successfully launched and operational, and that the Static-Pod registry has been successfully deployed by it |
CleanupNodeServices |
State of cleaning up the Node Services Manager and Static-Pod registry if they are not needed in the selected mode. Verifies that all related resources have been removed |
RegistryContainsRequiredImages |
State of checking the registry for the presence of required images |
Ready |
Overall state of registry readiness in the selected mode. Indicates that all other conditions are met and the module is ready to operate |