The page contains frequently asked questions about configuring and using the Deckhouse Platform.

Common

How can I apply an update for a specific module immediately?

To apply an update for a specific module immediately, set the modules.deckhouse.io/apply-now: "true" annotation on the corresponding ModuleRelease resource.

This annotation applies the release immediately without waiting for the update window. The requirements from spec.requirements still apply. If they are not met, the release will not be applied.

Example of setting the annotation for the console module:

d8 k annotate mr console-v1.43.3 modules.deckhouse.io/apply-now="true"

This can also be done using the d8 CLI for convenience (module names and versions are autocompleted):

d8 system module apply-now console v1.43.3

Example resource with the annotation set:

apiVersion: deckhouse.io/v1alpha1
kind: ModuleRelease
metadata:
  name: console-v1.43.3
  annotations:
    modules.deckhouse.io/apply-now: "true"
...

How can I apply an update DP immediately, bypassing update windows, canary releases, and manual update mode?

To apply a Deckhouse Platform (DP) update immediately, add the annotation release.deckhouse.io/apply-now: "true" to the corresponding DeckhouseRelease resource.

This will bypass update windows, canary release settings, and the manual cluster update mode. The update will be applied immediately after the annotation is set.

Example command to set the annotation and skip update windows for version v1.56.2:

d8 k annotate deckhousereleases v1.56.2 release.deckhouse.io/apply-now="true"

Example of a resource with the annotation set:

apiVersion: deckhouse.io/v1alpha1
kind: DeckhouseRelease
metadata:
  annotations:
    release.deckhouse.io/apply-now: "true"
...

What to do if, with VPN enabled, the container with the installer cannot access the network?

If a VPN is installed on the computer from which the cluster bootstrap is performed, there may be a problem with the container with the installer (or graphical installer) accessing the network. Because of this, for example, the graphical installer may not be displayed in the browser.

The problem can be solved in one of the following ways:

  • Disable VPN on your computer and restart the container with the installer.
  • If you cannot disable VPN (for example, if the cluster bootstrap is running on a VPN network), use the --network host parameter when starting the container with the installer (for Docker Desktop on Mac OS, the parameter is available starting with version 4.34.0). This will allow the container to access the network.

    Example of launching a container with an installer with VPN enabled, using the --network host parameter:

    docker run --network host --pull=always -it -v "$PWD/config.yml:/config.yml" -v "$HOME/.ssh/:/tmp/.ssh/" -v "$PWD/dhctl-tmp:/tmp/dhctl" registry.deckhouse.ru/deckhouse/ce/install:early-access bash
    

    Example of running the graphical installer with VPN enabled, using the --network host parameter:

    docker run --network host --rm --pull always -v $HOME/.d8installer:$HOME/.d8installer -v /var/run/docker.sock:/var/run/docker.sock -p 127.0.0.1:8080:8080 registry.deckhouse.ru/deckhouse/installer:latest -r $HOME/.d8installer
    

How do I change the release channel for a module?

A module can be built-in to DP or connected from a module source (defined using ModuleSource). Built-in modules have a common release cycle with DP and are updated together with DP. The release channel of a built-in module always matches the DP release channel. A module connected from a source has its own release cycle, which is independent of the DP release cycle. The release channel of a module connected from a source can be changed.

Below is the process of changing the release channel for a module connected from a source.

By default, the release channel for modules is inherited from the DP release channel (specified in the releaseChannel parameter of the deckhouse ModuleConfig). For more information about release channels, see the Release Channels section.

For modules from a source, the release channel is specified using ModuleUpdatePolicy, which is then linked to the module via the updatePolicy parameter in ModuleConfig.

To change the release channel for a module from a source, follow these steps:

  1. Define the module update policy.

    Create a ModuleUpdatePolicy where you specify the release channel in the releaseChannel parameter.

    Example ModuleUpdatePolicy:

    apiVersion: deckhouse.io/v1alpha2
    kind: ModuleUpdatePolicy
    metadata:
      name: my-module-policy
    spec:
      releaseChannel: Alpha
      update:
        mode: Auto
    

    If necessary, change the update mode and configure update windows.

    Ensure that the policy has been created:

    d8 k get mup my-module-policy
    

    Output example:

    NAME               RELEASE CHANNEL   UPDATE MODE
    my-module-policy   Alpha             AutoPatch
    
  2. Link the update policy to the module.

    Specify the name of the created update policy in the updatePolicy parameter of the corresponding module’s ModuleConfig.

    To edit the ModuleConfig, use the command (specify the module name):

    d8 k edit mc my-module
    

    Example ModuleConfig:

    apiVersion: deckhouse.io/v1alpha1
    kind: ModuleConfig
    metadata:
      name: my-module
    spec:
      enabled: true
      # ModuleUpdatePolicy name.
      updatePolicy: my-module-policy
    

When you change the module’s release channel, its version will change according to the configured update mode.

To view the current release channel of the module and other information about the module’s state in the cluster, use the corresponding Module object.

Example command to get information about the module:

d8 k get module my-module -o yaml

The update policy used will be specified in the properties.updatePolicy field, and the current release channel in the properties.releaseChannel field. Example output:

apiVersion: deckhouse.io/v1alpha1
kind: Module
metadata:
  name: my-module
  # ...
properties:
  # ...
  releaseChannel: Alpha # Module release channel.
  updatePolicy: my-module-policy # Module update policy.
  version: v1.16.10  # Module version.
  # ...

How do I configure an external module to update to patch versions only?

By default, if a module has no dedicated update policy, the update mode and windows are inherited from the DP settings.

If DP is set to the AutoPatch mode, the external module will also automatically receive only patch versions within the current minor version. Moving to a new minor version will require manual approval. In this case, no additional configuration is needed.

For more information about DP update modes, see Configuring updates.

If you need to manage the module update mode independently of DP, create a ModuleUpdatePolicy with update.mode: AutoPatch and link it to the module via the updatePolicy parameter in ModuleConfig:

  1. Create an update policy.

    Example ModuleUpdatePolicy:

    apiVersion: deckhouse.io/v1alpha2
    kind: ModuleUpdatePolicy
    metadata:
      name: my-update-policy
    spec:
      releaseChannel: Stable
      update:
        mode: AutoPatch
    

    Ensure that the policy has been created:

    d8 k get mup my-update-policy
    
  2. Link the policy to the module.

    Specify the policy name in the updatePolicy parameter of the module’s ModuleConfig:

    d8 k edit mc module
    

    Example ModuleConfig:

    apiVersion: deckhouse.io/v1alpha1
    kind: ModuleConfig
    metadata:
      name: module
    spec:
      enabled: true
      updatePolicy: my-update-policy
    

In the AutoPatch mode, module patch versions (for example, from v1.16.1 to v1.16.2) are applied automatically, taking update windows into account if they are configured. To move to a new minor version (for example, from v1.16.* to v1.17.*), approve the corresponding ModuleRelease:

d8 k annotate mr module-v1.17.0 modules.deckhouse.io/approved="true"

Or using the d8 CLI:

d8 system module approve module v1.17.0

What should I do if the module image did not download and the module did not reinstall?

In some cases, there may be a problem with automatically downloading the image and reinstalling the module. These cases include:

  • Damage to the file system or other problems that have rendered the module image invalid.
  • Switching to a different registry.
  • Switching from one DP edition to another.

In this case, the module may be in the Ready state. The error occurs in the module’s pods. To find the problematic pod, use the command:

d8 k -n d8-<module-name> get pods

A problematic pod will have a status other than Running.

To view information about a pod, use the command:

d8 k -n d8-<module-name> describe pod <pod-name>

Example of an error message in the pod when there is a problem with downloading the image and reinstalling the module:

Failed to pull image "registry.deckhouse.ru/deckhouse/ce/modules/console@sha256:a12b4f8de1d997005155d0ba0a7c968a015dd8d18bb5d54645ddb040ddab1ef4": rpc error: code = NotFound desc = failed to pull and unpack image "registry.deckhouse.ru/deckhouse/ce/modules/console@sha256:a12b4f8de1d997005155d0ba0a7c968a015dd8d18bb5d54645ddb040ddab1ef4": failed to resolve reference ...

To download the image and reinstall the module that caused the problem:

  1. Get a list of module releases:

    d8 k get mr -l module=my-module
    

    Output example:

    NAME               PHASE        UPDATE POLICY   TRANSITIONTIME   MESSAGE
    my-module-v3.7.4   Superseded                   5d23h
    my-module-v3.7.5   Deployed                     5d23h
    

    Find the module release deployed in the cluster in the list (it should have the status Deployed).

  2. Add the annotation modules.deckhouse.io/reinstall=true to the expanded release:

    d8 k annotate mr my-module-v3.7.5 modules.deckhouse.io/reinstall=true
    

After adding the annotation, the module image is re-downloaded from the registry, the module is validated with the current settings from ModuleConfig, and installed in the cluster. After successful reinstallation, the annotation is automatically removed from ModuleRelease.

To verify that the module has been successfully reinstalled and all module pods are working, use the command:

d8 k -n d8-<module-name> get pods

All pods in the module must have the status Running. Example:

NAME                                READY   STATUS    RESTARTS   AGE
backend-567d6c6cdc-g5qgt            1/1     Running   0          2d2h
frontend-7c8b567759-h8jdf           1/1     Running   0          2d2h
observability-gw-86cf75f5d6-7xljh   1/1     Running   0          2d2h

Cluster and Infrastructure

What to do if you encounter problems updating Kubernetes components on cluster nodes, synchronizing nodes, or applying NodeGroup Configuration?

If Kubernetes components are not updated on the cluster node, the NodeGroup configuration is not applied, and not all NodeGroup nodes are synchronized (have the UPTODATE status), perform the following steps:

  1. Check the bashible logs on the node where the problems are occurring. The bashible mechanism is used to keep cluster nodes up to date. It is started by the bashible.timer timer at regular intervals as a service on the cluster nodes. This involves restarting, synchronizing scripts, and executing them (if necessary).

    To check bashible logs, use the command:

    journalctl -u bashible
    

    If the response contains the message Configuration is in sync, nothing to do, the node is synchronized and there are no problems. The absence of this message or the presence of errors indicates a problem.

  2. Check the synchronization status of cluster nodes using the command:

    d8 k get ng
    

    The number of nodes in the UPTODATE state must match the total number of nodes in each group.

    Example output:

    NAME       TYPE     READY   NODES   UPTODATE   INSTANCES   DESIRED   MIN   MAX   STANDBY   STATUS   AGE    SYNCED
    frontend   Static   1       1       1                                                               118d   True
    master     Static   3       3       3                                                               118d   True
    system     Static   2       2       2                                                               118d   True
    worker     Static   2       2       2                                                               118d   True
    

How to migrate node groups to Cluster API (CAPI)?

Deckhouse Platform is migrating CloudEphemeral node management from Machine Controller Manager (MCM) to Cluster API (CAPI).

At the moment, migration from MCM to CAPI is supported for the following cloud providers:

After CAPI support is introduced for a cloud provider, existing CloudEphemeral node groups continue to use MCM (status.engine: MCM). New node groups use CAPI by default (status.engine: CAPI).

To check which management mechanism is used for a node group, run the following command:

d8 k get nodegroup -o custom-columns=NAME:.metadata.name,ENGINE:.status.engine

To migrate a node group from MCM to CAPI:

  1. Create a new NodeGroup of the CloudEphemeral type with the required configuration. Do not set the node.deckhouse.io/use-mcm annotation — otherwise the group will remain on MCM.

  2. Make sure the new group is managed by CAPI (status.engine: CAPI):

    d8 k get nodegroup <NODE_GROUP_NAME> -o custom-columns=NAME:.metadata.name,ENGINE:.status.engine
    
  3. Wait until the nodes in the new group become Ready:

    d8 k get nodes -l node.deckhouse.io/group=<NODE_GROUP_NAME>
    
  4. Move workloads from the old group to the new one.

    For example, update application nodeSelector and tolerations so that pods are scheduled onto nodes of the new group, or use another workload placement method adopted in your infrastructure. For details on allocating nodes for workloads, see Allocating nodes for specific workloads.

  5. Make sure the migrated workloads are running successfully on the new group nodes and that no required pods remain on the old group nodes.
  6. Delete the old NodeGroup.

Deleting a NodeGroup deletes all nodes in that group. Before deleting the old group, make sure the required workloads have been moved on the new group nodes and are running correctly.

If necessary, you can force a node group to be created using MCM. To do this, add the node.deckhouse.io/use-mcm annotation before creating the NodeGroup (or before recreating it):

d8 k annotate nodegroup <NODE_GROUP_NAME> node.deckhouse.io/use-mcm="true"

For example:

d8 k annotate nodegroup worker node.deckhouse.io/use-mcm="true"

The node.deckhouse.io/use-mcm annotation is a temporary workaround and its use is not recommended. Migrating node groups to CAPI is the preferred approach.

What to do if problems adding a node to the cluster via Cluster API Provider Static are present?

If, when adding a node to the cluster via Cluster API Provider Static (CAPS), it remains in Pending or Bootstrapping status, perform the following steps:

  1. Verify that the access keys specified in the SSHCredentials resource are correct. Ensure that the username and SSH key specified in SSHCredentials are correct.

  2. On the node where the problem occurred, check that the public key corresponding to the private key from SSHCredentials is present in authorized_keys. Example command for checking:

    cat ~/.ssh/authorized_keys
    
  3. Check the number of nodes specified in NodeGroup, which should include the node being added. Make sure that the maximum number of nodes is not exceeded.

  4. Check the status of the bashible.service on the node that caused the problem:

    systemctl status bashible.service
    

    It must have the status active (running). If the service has the status inactive or failed, the service has not started. This indicates a problem with the configuration process.

  5. If the steps above did not resolve the issue, remove the problematic node and its StaticInstance resource from the cluster so that the system will attempt to recreate them. To do this:

    • Get a list of nodes and locate the problematic one:

      d8 k get nodes
      
    • Find the corresponding StaticInstance resource:

      kubectl get staticinstances -n <namespace-name>
      
    • Remove the problematic node:

      kubectl delete node <node-name>
      
    • Remove the corresponding StaticInstance resource:

      kubectl delete staticinstances -n <namespace-name> <static-instance-name>
      

How do I replace the OS image in a cloud cluster?

Deckhouse Platform creates CloudEphemeral and CloudPermanent nodes, including master nodes, based on an operating system (OS) image specified in the node group configuration. To move the nodes to a new image, specify it in this configuration instead of the previous one. Deckhouse Platform recreates the CloudEphemeral nodes from the new image automatically. The CloudPermanent nodes and master nodes are recreated when you apply the configuration with the dhctl converge command.

For CloudStatic nodes, the administrator creates and maintains the virtual machines. To replace the image of such a node, prepare a virtual machine with the required OS and connect it using the instructions for adding CloudStatic nodes.

The resource where the image is specified depends on the node type and the cloud provider configuration method:

Node type Provider configuration method Where to change the image
CloudEphemeral Through ModuleConfig or <PROVIDER>ClusterConfiguration In the <PROVIDER>InstanceClass resource referenced by the node group
CloudPermanent and master nodes Through ModuleConfig In the <PROVIDER>InstanceClass resource referenced by the node group
CloudPermanent and master nodes Through <PROVIDER>ClusterConfiguration In the instanceClass section of the required node group in that resource

Migration from <PROVIDER>ClusterConfiguration to the new ModuleConfig-based configuration model is supported only for built-in virtualization (DVP), Yandex Cloud, and zVirt.

The image parameters depend on the resource whose configuration you edit. Select the appropriate tab.

  • InstanceClass
  • ClusterConfiguration

For CloudEphemeral nodes, use the instance class parameters with either provider configuration method. With a ModuleConfig-based configuration, these parameters also define the image for CloudPermanent nodes and master nodes.

Cloud provider Parameter in the spec section Instance class resource
Amazon Web Services ami AWSInstanceClass
Microsoft Azure urn AzureInstanceClass
Google Cloud Platform image GCPInstanceClass
Yandex Cloud imageID YandexInstanceClass
OpenStack imageName OpenStackInstanceClass
Huawei Cloud imageName HuaweiCloudInstanceClass
Basis Dynamix imageName DynamixInstanceClass
VMware Cloud Director template VCDInstanceClass
VMware vSphere template VsphereInstanceClass
zVirt template ZvirtInstanceClass
Built-in virtualization rootDisk.image DVPInstanceClass

With cloud providers other than VMware Cloud Director, VMware vSphere, zVirt, and built-in virtualization, the parameter value is a string with the OS image name or ID. In VMware Cloud Director, VMware vSphere, and zVirt, the template parameter sets a virtual machine template prepared based on the OS image. To move the nodes of these providers to a new image, prepare a new virtual machine template based on it.

In built-in virtualization, the image is specified as an object. In the rootDisk.image.kind parameter, specify the type of the resource with the image (ClusterVirtualImage, VirtualImage, or VirtualDisk), and in the rootDisk.image.name parameter, specify the name of this resource.

Example of the image parameter in built-in virtualization

apiVersion: deckhouse.io/v1alpha1
kind: DVPInstanceClass
metadata:
  name: worker
spec:
  rootDisk:
    # The other instance class parameters are not shown.
    image:
      kind: ClusterVirtualImage
      name: ubuntu-24-04-20260204

Requirements for the new image

The OS image or the virtual machine template prepared from it must meet the requirements of the selected cloud provider module.

Before the replacement, make sure that the new image or virtual machine template meets the following general requirements:

  • The image parameter value differs from the previous one. If the image or the template is set by name, include the version or the date in the name, for example ubuntu-24-04-20260204.
  • The image or the template prepared based on it exists in the cloud by the time the nodes are recreated.
  • The image contains cloud-init and an operating system supported by Deckhouse Platform. The list of supported systems is given in “Supported OS versions”.

For some providers, preparing the image or template involves additional steps. Follow the instructions in the corresponding module documentation:

Deckhouse Platform compares the parameter values in the configuration, not the image content in the cloud. If you replace the image or the template in the cloud keeping its previous name, the nodes are not recreated.

Changing the image in the configuration

Create a new image in the cloud that meets the requirements above. In VMware Cloud Director, VMware vSphere, and zVirt, prepare a new virtual machine template based on it. Then select the cloud provider configuration method.

In Yandex Cloud, CloudEphemeral groups whose instance classes do not specify imageID use the master node image by default. Changing the master node image does not recreate the existing nodes in these groups. New nodes may use the new image, so a group can contain nodes with different images.

To keep the previous group image, explicitly set its imageID in YandexInstanceClass. To recreate the entire group from the new image, explicitly set the new imageID.

For DVP, Yandex Cloud, and zVirt, check the master node group configuration before selecting a tab:

d8 k get nodegroup master -o yaml

If spec.cloudInstances.classReference.kind and spec.cloudInstances.classReference.name are set and migration is complete, select the “Through ModuleConfig” tab. Make sure the migration alert for your provider is not firing:

If the alert is firing, complete the migration first. If the master node group has no classReference, or the cluster uses another provider, select the “Through ClusterConfiguration” tab.

After completing the migration to ModuleConfig, change the image in the instance class referenced by the required node group.

The cluster may retain the d8-provider-cluster-configuration Secret with the old provider configuration. The d8 system edit provider-cluster-configuration command lets you edit this configuration, but the platform no longer applies image changes from it.

Select the appropriate tab depending on how the cloud provider is configured:

  • Through ModuleConfig
  • Through ClusterConfiguration

With a ModuleConfig-based provider configuration, the image for CloudEphemeral and CloudPermanent nodes, including master nodes, is specified in an instance class. To replace the image, edit the instance class referenced by the required node group:

  1. Open the node group configuration:

    d8 k get nodegroup <NODE_GROUP_NAME> -o yaml
    

    Replace <NODE_GROUP_NAME> with the node group name. For master nodes, use master.

    Check spec.nodeType for the node type and spec.cloudInstances.classReference.kind and spec.cloudInstances.classReference.name for the instance class kind and name. If multiple groups use the instance class, changing the image affects all of them.

  2. Specify the new image or virtual machine template in the instance class of the node group and save the file:

    d8 k edit <PROVIDER>instanceclass <INSTANCE_CLASS_NAME>
    

    Replace <PROVIDER> with the cloud provider name in lowercase, for example yandex, and <INSTANCE_CLASS_NAME> with the instance class name.

    An example for Yandex Cloud:

    apiVersion: deckhouse.io/v1
    kind: YandexInstanceClass
    metadata:
      name: worker
    spec:
      cores: 4
      memory: 8192
      imageID: fd83ergat2e815oohe7o
    

Example of the master node configuration in built-in virtualization

apiVersion: deckhouse.io/v1alpha1
kind: DVPInstanceClass
metadata:
  name: master
spec:
  rootDisk:
    # The other instance class parameters are not shown.
    image:
      kind: ClusterVirtualImage
      name: ubuntu-24-04-20260204
---
apiVersion: deckhouse.io/v1
kind: NodeGroup
metadata:
  name: master
spec:
  nodeType: CloudPermanent
  cloudInstances:
    classReference:
      kind: DVPInstanceClass
      name: master
    minPerZone: 1
    maxPerZone: 1

After changing the instance class, proceed to applying the changes.

With a <PROVIDER>ClusterConfiguration-based provider configuration, the resource to edit depends on the node type. Select the node type of the group.

  • CloudEphemeral
  • CloudPermanent
  • Master nodes

For CloudEphemeral nodes, the image is specified in the instance class referenced by the node group:

  1. Check spec.cloudInstances.classReference.kind and spec.cloudInstances.classReference.name in the NodeGroup resource for the instance class kind and name:

    d8 k get nodegroup <NODE_GROUP_NAME> -o yaml
    

    Replace <NODE_GROUP_NAME> with the node group name.

  2. Open the instance class and change the image parameter listed in the table above:

    d8 k edit <PROVIDER>instanceclass <INSTANCE_CLASS_NAME>
    

    Replace <PROVIDER> with the cloud provider name in lowercase, for example yandex, and <INSTANCE_CLASS_NAME> with the instance class name.

For CloudPermanent nodes, change the image in the provider configuration:

  1. Open the cloud provider configuration:

    d8 system edit provider-cluster-configuration
    
  2. Change the image parameter in the instanceClass section of the required group in the nodeGroups list and save the file. An example for Yandex Cloud:

    nodeGroups:
    - name: front
      replicas: 2
      instanceClass:
        cores: 4
        memory: 8192
        imageID: fd83ergat2e815oohe7o
    

For master nodes, change the image in the provider configuration:

  1. Open the cloud provider configuration:

    d8 system edit provider-cluster-configuration
    
  2. Change the image parameter in the masterNodeGroup.instanceClass section and save the file.

After saving the configuration, apply the changes according to the node type.

Applying the changes

The way to apply the changes depends on the node type and is the same for both provider configuration methods. Select the appropriate tab.

  • CloudEphemeral
  • CloudPermanent and master nodes

Deckhouse Platform automatically recreates CloudEphemeral nodes after the image in the instance class changes. To check that the replacement is complete, follow these steps:

  1. Wait until Deckhouse Platform recreates the nodes of the group:

    d8 k get nodes -l node.deckhouse.io/group=<NODE_GROUP_NAME>
    

    Replace <NODE_GROUP_NAME> with the node group name.

    During the replacement, the list may contain both the previous and the new nodes. The new nodes have a smaller value in the AGE column. An example of the output during the replacement:

    NAME                                     STATUS     ROLES    AGE     VERSION
    cloud-demo-worker-92f9049e-gm2lw-6hkm2   NotReady   worker   37s     v1.34.1
    cloud-demo-worker-92f9049e-xn29v-66rm9   Ready      worker   8m21s   v1.34.1
    

    The replacement is complete when only the new nodes in the Ready status remain in the list.

To apply the new image to CloudPermanent nodes and master nodes, run dhctl converge. The command reads the current configuration from the cluster, including ModuleConfig, NodeGroup, and instance classes when the provider uses a ModuleConfig-based configuration.

Before replacing master nodes, back up etcd. Also back up the /etc/kubernetes directory and copy both backups outside the cluster. Master nodes are recreated one at a time. In a single-master cluster, the command may offer a temporary transition to three master nodes and a return to one master node after the replacement.

  1. On the local machine, get the Deckhouse Platform edition and version in the cluster:

    DH_VERSION=$(d8 k -n d8-system get deployment deckhouse -o jsonpath='{.metadata.annotations.core\.deckhouse\.io\/version}')
    DH_EDITION=$(d8 k -n d8-system get deployment deckhouse -o jsonpath='{.metadata.annotations.core\.deckhouse\.io\/edition}' | tr '[:upper:]' '[:lower:]')
    
  2. On the local machine, run the Deckhouse Platform installer container of the same edition and version:

    docker run --pull=always -it -v "$HOME/.ssh/:/tmp/.ssh/" \
      registry.deckhouse.io/deckhouse/${DH_EDITION}/install:${DH_VERSION} bash
    

    If the cluster uses a different container registry, specify its address instead of registry.deckhouse.io.

  3. In the installer container, check which changes will be made in the cloud:

    dhctl terraform check --ssh-agent-private-keys=/tmp/.ssh/<SSH_KEY_FILENAME> \
      --ssh-user=<USERNAME> --ssh-host=<MASTER_IP>
    

    Replace <SSH_KEY_FILENAME> with the name of the private SSH key file, <USERNAME> with the user for connecting to the master node, and <MASTER_IP> with the master node address.

    When replacing master nodes, specify all master node addresses by repeating --ssh-host for each address.

    In the dhctl terraform check output, verify that the changes to nodes and other cloud resources match your expectations. In the final nodes list, nodes that require recreation have the destructively_changed status. If multiple groups use the same instance class, check the changes for each group.

  4. In the installer container, apply the changes using the same SSH connection parameters as in the check:

    dhctl converge --ssh-agent-private-keys=/tmp/.ssh/<SSH_KEY_FILENAME> \
      --ssh-user=<USERNAME> --ssh-host=<MASTER_IP>
    

    The command displays the plan of changes and asks for the Do you want to CHANGE objects state in the cloud? confirmation before replacing each node. To replace the node, enter y. The command also asks you to confirm that the master node name matches its address.

    Before confirming the replacement of the next master node, check etcd and the control plane using the master node verification procedure.

  5. Check that the nodes of the group are recreated:

    d8 k get nodes -l node.deckhouse.io/group=<NODE_GROUP_NAME>
    

    Replace <NODE_GROUP_NAME> with the node group name. For master nodes, use master. Wait until the new nodes reach the Ready status.

How to change the instance type for nodes with the CloudPermanent type?

To change the instance type for nodes with the CloudPermanent type, follow these steps:

  1. Make a backup of etcd and the /etc/kubernetes directory.
  2. Transfer the archive to a server outside the cluster (e.g., on a local machine).
  3. Ensure there are no alerts in the cluster that can prevent the update of the master nodes.
  4. Make sure that Deckhouse queue is empty. To view the status of all Deckhouse job queues, run the following command:

    d8 s queue list
    

    Example output (queues are empty):

    Summary:
    - 'main' queue: empty.
    - 88 other queues (0 active, 88 empty): 0 tasks.
    - no tasks to handle.
    
  5. Run the appropriate edition and version of the Deckhouse installer container on the local machine (change the container registry address if necessary):

    DH_VERSION=$(d8 k -n d8-system get deployment deckhouse -o jsonpath='{.metadata.annotations.core\.deckhouse\.io\/version}') 
    DH_EDITION=$(d8 k -n d8-system get deployment deckhouse -o jsonpath='{.metadata.annotations.core\.deckhouse\.io\/edition}' | tr '[:upper:]' '[:lower:]' ) 
    docker run --pull=always -it -v "$HOME/.ssh/:/tmp/.ssh/" \
      registry.deckhouse.ru/deckhouse/${DH_EDITION}/install:${DH_VERSION} bash
    
  6. In the installer container, run the following command to check the state before working:

    dhctl terraform check --ssh-agent-private-keys=/tmp/.ssh/<SSH_KEY_FILENAME> --ssh-user=<USERNAME> \
      --ssh-host <MASTER-NODE-0-HOST> --ssh-host <MASTER-NODE-1-HOST> --ssh-host <MASTER-NODE-2-HOST>
    

    The command output should indicate that Terraform found no inconsistencies and no changes are required.

  7. In the installer container, run the command to edit the cluster configuration (specify the addresses of all master nodes in the --ssh-host parameter):

    dhctl config edit provider-cluster-configuration --ssh-agent-private-keys=/tmp/.ssh/<SSH_KEY_FILENAME> --ssh-user=<USERNAME> \
      --ssh-host <MASTER-NODE-0-HOST> --ssh-host <MASTER-NODE-1-HOST> --ssh-host <MASTER-NODE-2-HOST>
    
  8. Edit the instanceClass parameter of the desired node group by changing the instance type and save the changes. Example settings for the masterNodeGroup of the Yandex Cloud provider:

    masterNodeGroup:
     replicas: 3  # required number of master nodes
     instanceClass:
       cores: 4      # change the number of CPUs
       memory: 8192  # change the memory size (in MB)
       # other instance parameters...
       externalIPAddresses:
       - "Auto"      # for each master node
       - "Auto"
       - "Auto"
    
  9. In the installer container, run the following command to perform nodes upgrade:

    You should read carefully what converge is going to do when it asks for approval.

    When the command is executed, the nodes will be replaced by new nodes with confirmation on each node. The replacement will be performed one by one in reverse order (2,1,0).

    dhctl converge --ssh-agent-private-keys=/tmp/.ssh/<SSH_KEY_FILENAME> --ssh-user=<USERNAME> \
      --ssh-host <MASTER-NODE-0-HOST> --ssh-host <MASTER-NODE-1-HOST> --ssh-host <MASTER-NODE-2-HOST>
    

    Repeat the steps below (Sec. 9-12) for each master node one by one, starting with the node with the highest number (suffix 2) and ending with the node with the lowest number (suffix 0).

  10. On the newly created node, check the systemd-unit log for the bashible.service. Wait until the node configuration is complete (you will see a message nothing to do in the log):

    journalctl -fu bashible.service
    
  11. Make sure the node is listed as an etcd cluster member:

    for pod in $(d8 k -n kube-system get pod -l component=etcd,tier=control-plane -o name); do
      d8 k -n kube-system exec "$pod" -- etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt \
      --cert /etc/kubernetes/pki/etcd/ca.crt --key /etc/kubernetes/pki/etcd/ca.key \
      --endpoints https://127.0.0.1:2379/ member list -w table
      if [ $? -eq 0 ]; then
        break
      fi
    done
    
  12. Make sure control-plane-manager is running on the node:

    d8 k -n kube-system wait pod --timeout=10m --for=condition=ContainersReady \
      -l app=d8-control-plane-manager --field-selector spec.nodeName=<MASTER-NODE-N-NAME>
    
  13. Proceed to update the next node (repeat the steps above).

How to migrate a cloud provider to ModuleConfig-based configuration?

Migration to the new ModuleConfig-based configuration model is supported only for built-in virtualization (DVP), Yandex Cloud, and zVirt. If the cluster uses DVPClusterConfiguration, YandexClusterConfiguration, or ZvirtClusterConfiguration, migrate its configuration using the procedure below.

Deckhouse Platform is transitioning from a single <PROVIDER>ClusterConfiguration resource to a model where the cloud provider configuration is split across four separate resources:

  • ModuleConfig of the cloud-provider-<PROVIDER> module: Provider and layout settings
  • Secret with credentials of the cloud-provider.deckhouse.io/credentials type: Access to the cloud API
  • Provider InstanceClass (for example, DVPInstanceClass): Virtual machine parameters
  • NodeGroup: Node group configuration

Migration is mandatory. This is not an optional step: support for <PROVIDER>ClusterConfiguration will be removed. Until the migration is completed, Deckhouse Platform updates may be blocked.

The migration is safe. Applying the resources automatically prepared by Deckhouse Platform does not cause nodes to be recreated. However, explicit administrator action is required, as the resources have to be reviewed and applied manually.

How to migrate

  1. Wait for the migration alert. Each provider has its own, for example, D8CloudProviderDVPMigrationPending for DVP, D8CloudProviderYandexCloudMigrationPending for Yandex Cloud, and D8CloudProviderZvirtMigrationPending for zVirt. The alert means the module has detected the legacy configuration and prepared manifests for the transition.

  2. Review the prepared resources in the d8-migration-resources Secret in the d8-cloud-provider-<PROVIDER> namespace:

    d8 k -n d8-cloud-provider-<PROVIDER> get secret d8-migration-resources -o jsonpath='{.data.resources\.yaml}' | base64 -d
    

    Example for built-in virtualization:

    d8 k -n d8-cloud-provider-dvp get secret d8-migration-resources -o jsonpath='{.data.resources\.yaml}' | base64 -d
    
  3. Verify that the manifests match the expected cluster configuration (ModuleConfig, credentials Secret, InstanceClass, and NodeGroup).

  4. Apply the resources:

    d8 k -n d8-cloud-provider-<PROVIDER> get secret d8-migration-resources -o jsonpath='{.data.resources\.yaml}' | base64 -d | d8 k apply -f -
    

    Example for built-in virtualization:

    d8 k -n d8-cloud-provider-dvp get secret d8-migration-resources -o jsonpath='{.data.resources\.yaml}' | base64 -d | d8 k apply -f -
    
  5. Wait until the alert disappears. The module clears it automatically once it detects that all required resources are in place.

How to delete the legacy secret

After the migration of a cluster deployed in built-in virtualization, Yandex Cloud, or zVirt, the provider settings are defined in ModuleConfig, and you can manually delete the legacy d8-provider-cluster-configuration Secret in the kube-system namespace.

To delete the Secret, follow these steps:

  1. Make sure the migration is complete and its alert no longer fires. For built-in virtualization, it is the D8CloudProviderDVPMigrationPending alert, for Yandex Cloud, the D8CloudProviderYandexCloudMigrationPending alert, and for zVirt, the D8CloudProviderZvirtMigrationPending alert. If you delete the Secret before the migration is complete, the cloud provider module stops working correctly.

  2. Check the number of nodes in node groups of the CloudPermanent type:

    d8 k get nodegroups \
      -o custom-columns='NAME:.metadata.name,TYPE:.spec.nodeType,MIN_PER_ZONE:.spec.cloudInstances.minPerZone'
    

    After the Secret is deleted, dhctl converge takes the number of nodes in these groups from the spec.cloudInstances.minPerZone parameter. If the parameter is not set for a node group of the CloudPermanent type, dhctl converge fails.

  3. Make sure the cloud infrastructure data, such as network IDs and zones, is stored in a separate Secret. The command must print a non-empty value:

    d8 k -n d8-cloud-provider-<PROVIDER> get secret d8-candi-cloud-provider-discovery-data \
      -o jsonpath='{.data.cloud-provider-discovery-data\.json}'
    

    Replace <PROVIDER> with dvp for built-in virtualization, yandex for Yandex Cloud, or zvirt for zVirt.

    If the command prints nothing or the Secret is not found, the data is stored only in the legacy Secret. If you delete it without a copy of the data, the terraform-manager module is disabled. The cloud-provider-yandex module also loses the IDs of the network, subnets, and route table that the platform created in Yandex Cloud. Copy the data to a separate Secret and delete the temporary file:

    d8 k -n kube-system get secret d8-provider-cluster-configuration \
      -o jsonpath='{.data.cloud-provider-discovery-data\.json}' \
      | base64 -d > /tmp/discovery-data.json
    
    d8 k -n d8-cloud-provider-<PROVIDER> create secret generic d8-candi-cloud-provider-discovery-data \
      --from-file=cloud-provider-discovery-data.json=/tmp/discovery-data.json \
      --dry-run=client -o yaml \
      | d8 k apply -f -
    
    rm /tmp/discovery-data.json
    

    Then repeat the check and make sure the command prints a non-empty value.

  4. Back up the Secret. The Secret contains cloud credentials, so store the file securely:

    d8 k -n kube-system get secret d8-provider-cluster-configuration \
      -o yaml > d8-provider-cluster-configuration-backup.yaml
    
  5. Allow deletion of the Secret. The Secret is protected by a webhook that rejects deletion without the deckhouse.io/allow-delete annotation:

    d8 k -n kube-system annotate secret d8-provider-cluster-configuration \
      deckhouse.io/allow-delete=true
    
  6. Delete the Secret:

    d8 k -n kube-system delete secret d8-provider-cluster-configuration
    

What to do if it takes a long time to switch to custom nodes in lower-priority groups?

When using multiple node groups with different priorities in a cloud cluster (the spec.cloudInstances.priority parameter), switching to nodes from groups with lower priority can take a long time. An example of this scenario is when groups of preemptible nodes (spot, etc.) are set to the highest priority, and if such instances are unavailable, ordering nodes from other groups takes a very long time.

When provisioning nodes, the cluster autoscaler sequentially selects the group with the highest priority that is not in a backoff state. A backoff is a temporary lock on a group following a failed attempt to provision a node (for example, due to a lack of instances of the required type in the cloud).

How it works when using multiple node groups:

  1. The cluster autoscaler attempts to provision a node in the group with the highest priority.
  2. If the node is not provisioned within the time specified by the max-node-provision-time parameter, the attempt is considered a failure.
  3. The group is marked as failed and blocked for the duration specified in the initial-node-group-backoff-duration parameter.
  4. If the same group fails again, the lockout time is doubled, but does not exceed the time specified in the max-node-group-backoff-duration parameter.
  5. The cluster autoscaler selects the next highest-priority group that is not locked out.

Here:

Parameter Default value Description
initial-node-group-backoff-duration 5 minutes Initial duration of the group lockout after the first failure. Doubles after each failed node ordering attempt
max-node-group-backoff-duration 30 minutes Maximum duration of the group lockout
max-node-provision-time 15 minutes The time after which the cluster autoscaler considers a node provisioning attempt to have failed

If these parameters are set to their default values, switching to node ordering in lower-priority groups may take a considerable amount of time (for more details, see this example).

These settings can be adjusted to speed up the switching of nodes in lower-priority groups. To set the desired values for the initial-node-group-backoff-duration, max-node-group-backoff-duration, and max-node-provision-time parameters, edit the cluster-autoscaler Deployment object in the d8-cloud-instance-manager namespace. The parameters are specified in the args field of the cluster-autoscaler container. Example:

...
     containers:
      - name: cluster-autoscaler
        args:
        - --initial-node-group-backoff-duration=1m # The initial duration of the group lockout after the first failure has been reduced.
        - --max-node-provision-time=5m # The time after which the cluster autoscaler considers a node request to have failed has been reduced.
...

If you encounter issues with node ordering in a cluster with a single node group

If cloud cluster uses only one node group (for example, exclusively for ordering spot or preemptible nodes) and you encounter issues with ordering such nodes during automatic scaling, follow these steps:

  1. Create one or more InstanceClass objects to be used when creating instances in the cluster. In the InstanceClass, specify node types that differ from those listed in the group mentioned above.

    In DP, InstanceClass objects vary depending on the provider. Examples: AWSInstanceClass, AzureInstanceClass, YandexInstanceClass.

  2. Create one or more new node groups (NodeGroup). In the spec.cloudInstances.classReference parameter, specify the InstanceClass objects created in the previous step. In the spec.cloudInstances.priority parameter, set the priority of the node group.
  3. If necessary, modify the max-node-provision-time, max-node-group-backoff-duration, and initial-node-group-backoff-duration parameters to speed up switching to lower-priority groups.

Example of the node ordering process with default settings

Suppose there are 3 groups of nodes with different priorities:

  • Group A (priority 50)
  • Group B (priority 30)
  • Group C (priority 0)

In this case, the node ordering process might be as follows:

Time Event
10:00 The cluster autoscaler attempts to order a node in A
10:15 The attempt to provision a node in A within 15 minutes (max-node-provision-time) fails → A blocked for 5 minutes (initial-node-group-backoff-duration)
10:15:xx The cluster autoscaler attempts to order a node in B (next in priority, not blocked)
10:30 The attempt to provision a node in B within 15 minutes (max-node-provision-time) fails → B blocked for 5 minutes (initial-node-group-backoff-duration)
10:30:xx The cluster autoscaler attempts to order a node in A again, as the lockout period (the default value for initial-node-group-backoff-duration is 5 minutes) has ended
10:45 The attempt to order a node in A fails again → A is blocked for 10 minutes (initial-node-group-backoff-duration for A is doubled)
… And so on, until the lockout time for A and B exceeds 15 minutes (max-node-provision-time). In this case, the queue may not reach group C for about 1.5 hours

What to do if the resources generated by an object are not created after adding the object?

If the required resources do not appear after creating an object in the system (for example, dexAuthenticator), follow these steps:

  1. Check if there are any critical alerts in the cluster that may block the creation of the required objects. To do this, use the command:

    d8 k get clusteralerts.deckhouse.io
    

    Example output:

    NAME               ALERT                                           SEVERITY   AGE   LAST RECEIVED   STATUS
    012f602592aa7a91   K8SSchedulerTargetDown                          3          16h   54s             firing
    0836dc893d5ecc65   KubernetesDeploymentReplicasUnavailable         5          15h   62s             firing
    08742f87d62d0063   NTPDaemonOnNodeDoesNotSynchronizeTime           5          16h   46s             firing
    172cfd38d2f7fd19   D8DeckhouseQueueIsHung                          7          12h   66s             firing
    1c5705daf731f5cf   D8StrongholdNoActiveNodes                       3          16h   55s             firing
    1d2c2f7d69f69f4b   D8DeckhouseIsNotOnReleaseChannel                9          12h   53s             firing
    205a551243d795f3   D8LogShipperAgentNotScheduledInCluster          7          15h   63s             firing
    2e34039aa7a3018e   D8NodeIsNotUpdating                             9          12h   47s             firing
    31baf9a70d657275   D8StrongholdClusterNotHealthy                   7          16h   55s             firing
    

    For more information about alerts, see the The list of alerts section.

  2. Check the Deckhouse task queue:

    d8 s queue list
    

    Example output (queues are empty):

    Summary:
    - 'main' queue: empty.
    - 88 other queues (0 active, 88 empty): 0 tasks.
    - no tasks to handle.
    

    If there are many unprocessed or long-running tasks in the queue, this may indicate problems.

  3. Analyze DP logs and events:

    • To view logs in real time, use the command:

      d8 k -n d8-system logs -f -l app=deckhouse
      

      Example output:

      {"level":"info","logger":"addon-operator","msg":"ConvergeModules task for OperatorStartup in phase '', trigger is Operator-Startup","binding":"ConvergeModules","event.type":"OperatorStartup","queue":"main","task.flow":"start","task.id":"fde0eb3b-5c3e-4da6-a0d8-a52f8ae03428","time":"2025-11-26T08:29:33Z"}
      {"level":"warn","logger":"addon-operator.converge-modules","msg":"ConvergeModules: functional scheduler not finished","binding":"ConvergeModules","event.type":"OperatorStartup","queue":"main","task.id":"fde0eb3b-5c3e-4da6-a0d8-a52f8ae03428","time":"2025-11-26T08:29:33Z"}
      

      When analyzing logs, pay special attention to warnings (WARNING) and error messages (ERROR).

    • To view events, use the command:

      d8 k -n d8-system get events
      

      Example output:

      LAST SEEN   TYPE      REASON              OBJECT                                          MESSAGE
      11m         Warning   Unhealthy           pod/deckhouse-5886c9bd77-vgdbw                  Readiness probe failed: HTTP probe failed with statuscode: 500
      7m22s       Normal    SuccessfulDelete    replicaset/deckhouse-5886c9bd77                 Deleted pod: deckhouse-5886c9bd77-vgdbw
      7m20s       Normal    Scheduled           pod/deckhouse-6bc5c4494-fwx6z                   Successfully assigned d8-system/deckhouse-6bc5c4494-fwx6z to sandbox1-master-0
      7m20s       Normal    Pulling             pod/deckhouse-6bc5c4494-fwx6z                   Pulling image "dev-registry.deckhouse.io/sys/deckhouse-oss@sha256:17ac07634e17422df52720264cddec3916ed6985a77782dc8a24fe5352290e6e"
      

    When analyzing events, pay special attention to those with the Warning type.

Deckhouse

How can I check DP versions available in different release channels?

For information about the current DP versions across all release channels, visit releases.deckhouse.io.

How to check the job queue in Deckhouse?

How to check the status of all Deckhouse task queues?

To view the status of all Deckhouse job queues, run the following command:

d8 s queue list

Example output (queues are empty):

Summary:
- 'main' queue: empty.
- 88 other queues (0 active, 88 empty): 0 tasks.
- no tasks to handle.

How to view the status of the main task queue?

To view the status of the Deckhouse main task queue, run the following command:

d8 s queue main

Example output (38 tasks in the main queue):

Queue 'main': length 38, status: 'run first task'

Example output (the main queue is empty):

Queue 'main': length 0, status: 'waiting for task 0s'

What should I do if DP is not receiving updates from the configured channel?

  • Ensure the correct release channel is configured.
  • Check that DNS resolution for the Deckhouse image registry is working correctly.

    Get and compare the IP addresses of registry.deckhouse.io from both a node and the deckhouse Pod. They must match.

    Example of obtaining an IP of registry.deckhouse.io from a node:

    getent ahosts registry.deckhouse.io
    

    Example output:

    185.193.90.38    STREAM registry.deckhouse.io
    185.193.90.38    DGRAM
    185.193.90.38    RAW
    

    Example of obtaining an IP of registry.deckhouse.io from the deckhouse Pod:

    d8 k -n d8-system exec -ti svc/deckhouse-leader -c deckhouse -- getent ahosts registry.deckhouse.io
    

    Example output:

    185.193.90.38    STREAM registry.deckhouse.io
    185.193.90.38    DGRAM  registry.deckhouse.io
    

    If the resulted IPs do not match, check DNS settings on the node. Pay attention to the search domain list in /etc/resolv.conf, which affects name resolution in the deckhouse Pod. If the search parameter in /etc/resolv.conf file specifies a domain with wildcard DNS resolution configured, this may lead to incorrect IP address resolution for the Deckhouse image registry (see example below).

Example DNS settings that may cause issues resolving the DP image registry IP address

Below is an example of how DNS settings may result in different resolution behavior on the node and in a Kubernetes Pod:

  • Example of /etc/resolv.conf on the node:

    nameserver 10.0.0.10
    search company.my
    

    On nodes, the default ndot setting is 1 (options ndots:1), while in Kubernetes Pods, it’s 5. This causes different resolution logic for DNS names with 5 or fewer dots on a node and on the Pod.

  • The DNS zone company.my has a wildcard entry *.company.my that resolves to 10.0.0.100. This means any undefined DNS name in the company.my zone resolves to 10.0.0.100.

Taking into account the search parameter in /etc/resolv.conf, when accessing registry.deckhouse.io from a node, the system will attempt to resolve the IP address for registry.deckhouse.io (because it considers it fully qualified due to the default options ndots:1 setting).

However, when accessing registry.deckhouse.io from a Kubernetes Pod, considering the options ndots:5 setting used by default in Kubernetes and the search parameter, the system will first attempt to resolve the name registry.deckhouse.io.company.my. This name will resolve to the IP address 10.0.0.100 because, according to the company.my DNS zone’s wildcard configuration, *.company.my is resolved to 10.0.0.100. As a result, the Pod will fail to connect to the registry.deckhouse.io host and will be unable to download information about available Deckhouse updates.

Does Deckhouse support realtime (rt) and low-latency Linux kernels?

In general, realtime (rt) and lowlatency kernels are supported with no extra configuration needed for Deckhouse Platform. Such kernels have been tested with DP EE version v1.75.7 and above on the following OSes and kernels:

  • CentOS 9 Stream, linux-5.14.0-706.el9.x86_64+rt
  • RedOS 8, linux-6.12.85-1.red80.x86_64-rt
  • Astra Linux 1.7.5, linux-5.15-lowlatency

How can I know when a new DP version is available for the cluster?

As soon as a new version appears on the configured release channel:

  • The DeckhouseReleaseIsWaitingManualApproval alert will appear if the cluster is in manual update mode.
  • A new DeckhouseRelease custom resource will be created. To see the list of releases, run d8 k get deckhousereleases. If the new version is in Pending state, it means it hasn’t been installed yet. Possible reasons:
    • Manual update mode is enabled.
    • Automatic update mode is enabled and update windows are scheduled, but the window hasn’t started yet.
    • Automatic update mode is enabled and update windows are not scheduled, but the update is delayed by a random period to reduce load on the container image registry. The status.message field of the DeckhouseRelease resource will show a corresponding message.
    • The update.notification.minimalNotificationTime parameter is set, and the delay period hasn’t elapsed.

How can I receive information about upcoming updates in advance?

You can get information about upcoming minor DP version updates on the release channel in one of the following ways:

How can I tell that the update was successful?

If the DeckhouseUpdating alert is gone, the update has finished.

You can also check the status of DP releases in the cluster with the following command:

d8 k get deckhouserelease

Example output:

NAME       PHASE        TRANSITIONTIME   MESSAGE
v1.46.8    Superseded   13d
v1.46.9    Superseded   11d
v1.47.0    Superseded   4h12m
v1.47.1    Deployed     4h12m

The Deployed status means the cluster has switched to the corresponding version, but it doesn’t guarantee that the update has been successful.

To ensure the update completed successfully, check the state of the deckhouse Pod with the following command:

d8 k -n d8-system get pods -l app=deckhouse

Example output:

NAME                   READY  STATUS   RESTARTS  AGE
deckhouse-7844b47bcd-qtbx9  1/1   Running  0       1d
  • If the Pod is Running and shows 1/1 under READY, it means the update completed successfully.
  • If the Pod is Running but shows 0/1 under READY, it means the update is still in progress. If it stays like this for more than 20–30 minutes, it may indicate a problem with DP that requires investigation.
  • If the Pod is not Running, it may indicate a problem with DP that requires investigation.

If something goes wrong

  • Check the logs using the following command:

    d8 k -n d8-system logs -f -l app=deckhouse | jq -Rr 'fromjson? | .msg'
    
  • Collect debug information and contact DP technical support.

How can I tell that an update is in progress?

During an update:

  • The DeckhouseUpdating alert is active.
  • The deckhouse Pod is not in the Ready state. If the Pod stays in a non-Ready state for a long time, it may indicate an issue with DP that requires investigation.

What to do if there are problems updating DP?

Deckhouse Platform update fails, one or more Deckhouse pods are in an unworkable state

If the Deckhouse Platform update fails, one or more Deckhouse pods in the d8-system namespace are in an unworkable state. Perform the following steps:

  1. Check the Deckhouse logs using the command:

    d8 k -n d8-system logs -f -l app=deckhouse | jq -Rr 'fromjson? | .msg'
    

    If there are any problems, information about them will be included in the output. When analyzing logs, pay special attention to warnings (WARNING) and error messages (ERROR).

  2. Check Deckhouse events using the command:

    d8 k -n d8-system describe po -l app=deckhouse | awk '
    /^Name:/ { 
        pod = $2; 
        print "=== " pod " ==="; 
        in_events = 0 
    }
    /Events:/ { 
        in_events = 1; 
        next 
    }
    in_events && /^$/ { 
        in_events = 0; 
        print "---" 
    }
    in_events && !/^Events:/ { 
        print $0 
    }
    ' | sed '/^---$/N;/^\n$/D'
    

    Pod events contain key information about problems (e.g., planning errors, image loading errors, etc.). When analyzing events, pay special attention to those with the Warning type.

    Example output:

    Type     Reason     Age                      From     Message
    ----     ------     ----                     ----     -------
    Warning  Unhealthy  4m44s (x1918 over 154m)  kubelet  Readiness probe failed: HTTP probe failed with statuscode: 500
    

DP update is stuck in the Release is suspended status

The status Release is suspended indicates that it has been postponed and is currently unavailable (not recommended) for installation. In this case, it is recommended to remain on the latest available release or on the one currently installed (it will have the status Deployed).

To view the list of releases, use the command:

d8 k get deckhousereleases.deckhouse.io

Example output:

NAME       PHASE        TRANSITIONTIME   MESSAGE
v1.69.13   Skipped      3h46m
v1.69.14   Skipped      3h46m
v1.69.15   Skipped      3h46m
v1.69.16   Superseded   160m
v1.70.12   Suspended    49d              Release is suspended
v1.70.13   Skipped      36d
v1.70.14   Skipped      34d
v1.70.15   Skipped      28d
v1.70.16   Skipped      19d
v1.70.17   Deployed     160m
v1.71.3    Suspended    14d              Release is suspended

IAM

What to do if there are problems applying DexProvider settings?

If you have changed the DexProvider settings in the user-authn module and one of the following problems occurs:

  • No changes are visible (settings are not applied).
  • When attempting to log in to the platform’s web interface with any type of authorization, a 500 Internal Server Error occurs without a detailed description.

Perform the following steps:

  1. Check the status of the dex deployment:

    d8 k -n d8-user-authn get pod
    

    Example output:

    NAME                                    READY   STATUS    RESTARTS   AGE
    dex-5ddb779b7d-6pbhs                    2/2     Running   0          20h
    kubeconfig-generator-7c46977b9f-5kdmc   1/1     Running   0          20h
    

    If the module is functioning properly and the correct configuration is specified in DexProvider, all pods will have the status Running.

  2. Check the logs for the problematic pod:

    d8 k -n d8-user-authn logs dex-<pod-name>
    

    Based on the information from the logs, correct the configuration in the DexProvider resource and wait for the dex pods to restart. Within a few minutes, the pods will restart automatically, and the platform’s web interface (located at console.<CLUSTER_NAME_TEMPLATE>) will become available and will reflect the changes made to the DexProvider resource.

Kubernetes and Scheduling

What to do when the API server is overloaded?

The following signs may indicate problems with API server load and memory consumption:

  • kubectl (d8) responds slowly or does not respond at all (commands are executed slowly or not at all).
  • Pods are recreated in the cluster for no apparent reason.

If these signs are present, perform the following actions:

  1. Check the resource consumption of API server pods. To do this, use the command:

    d8 k -n kube-system top po -l component=kube-apiserver
    

    Pay attention to MEMORY consumption and CPU.

    Example output:

    NAME                               CPU(cores)   MEMORY(bytes)
    kube-apiserver-sandbox1-master-0   251m         1476Mi
    
  2. Check the metrics in Grafana.

    To view the metrics, open the dashboard “Home” → “Dashboards” → “Kubernetes Cluster” → “Control Plane Status”. Review the graphs related to the API server (“Kube-apiserver CPU Usage”, “Kube-apiserver Memory Usage”, “Kube-apiserver latency”, etc.).

  3. Review the API server audit logs to identify the source of high memory consumption. One common cause of high memory consumption is a large number of requests.

How to check the Kubernetes version in use?

To check the Kubernetes version in use, run the command:

d8 k get nodes

Example output:

NAME                   STATUS   ROLES                  AGE    VERSION
frontend-0             Ready    frontend               118d   v1.31.9
master-0               Ready    control-plane,master   118d   v1.31.9
master-1               Ready    control-plane,master   118d   v1.31.9
master-2               Ready    control-plane,master   118d   v1.31.9
system-0               Ready    system                 118d   v1.31.9
system-1               Ready    system                 118d   v1.31.9
worker-0               Ready    worker                 37d    v1.31.9
worker-1               Ready    worker                 19d    v1.31.9

Network

Is the ingress-nginx module supported? Will it affect passing a PCI DSS audit?

The ingress-nginx module remains supported by Deckhouse Platform for the entire platform support lifecycle and does not depend on the upstream project’s status. The Deckhouse team tracks vulnerabilities in the controller and related components (NGINX, Lua modules, base images), delivers fixes in DP releases, and provides a migration path toward Gateway API.

Flant is the vendor responsible for support. Together with ongoing module maintenance and vulnerability management, this allows passing a PCI DSS audit and meets expectations for vendor accountability for a platform component.

Virtualization

How to use Ansible to provision virtual machines?

Ansible is an automation tool for running tasks on remote servers over SSH. This example shows how to use Ansible with virtual machines (VMs) in the demo-app project.

The example assumes that:

  • demo-app namespace contains a VM named frontend.
  • VM has a cloud user with SSH access.
  • Private SSH key on the machine where Ansible runs is stored in /home/user/.ssh/id_rsa.
  1. Create an inventory.yaml file:

    ---
    all:
      vars:
        ansible_ssh_common_args: '-o ProxyCommand="d8 v port-forward --stdio=true %h %p"'
        # Default user for SSH access.
        ansible_user: cloud
        # Path to private key.
        ansible_ssh_private_key_file: /home/user/.ssh/id_rsa
      hosts:
        # Host name in the format <VM_NAME>.<NAMESPACE>.
        frontend.demo-app:
    
    
  2. Check the virtual machine uptime:

    ansible -m shell -a "uptime" -i inventory.yaml all
    
    # frontend.demo-app | CHANGED | rc=0 >>
    # 12:01:20 up 2 days,  4:59,  0 users,  load average: 0.00, 0.00, 0.00
    

If you do not want to use an inventory file, pass all parameters on the command line:

ansible -m shell -a "uptime" \
  -i "frontend.demo-app," \
  -e "ansible_ssh_common_args='-o ProxyCommand=\"d8 v port-forward --stdio=true %h %p\"'" \
  -e "ansible_user=cloud" \
  -e "ansible_ssh_private_key_file=/home/user/.ssh/id_rsa" \
  all

How to automatically generate inventory for Ansible?

Instead of manually creating an inventory file, you can use the d8 v ansible-inventory command, which automatically generates an Ansible inventory from virtual machines in the specified namespace. The command is compatible with the ansible inventory script interface.

The d8 v ansible-inventory command requires d8 v0.27.0 or higher. The command works only for virtual machines that have the main cluster network (Main) connected.

Only machines in the Running phase that have an assigned IP address get into the inventory. Host names are formatted as <VM_NAME>.<NAMESPACE> (for example, frontend.demo-app).

  1. Optionally set host variables via annotations (for example, the SSH user):

    d8 k -n demo-app annotate vm frontend vars.ansible.deckhouse.io/ansible_user="cloud"
    
  2. Run Ansible with a dynamically generated inventory:

    ANSIBLE_INVENTORY_ENABLED=yaml ansible -m shell -a "uptime" all -i <(d8 v ansible-inventory -n demo-app -o yaml)
    

The <(...) construct is necessary because Ansible expects a file or script as the source of the host list. Simply specifying the command in quotes won’t work, because Ansible tries to execute the string as a script. The <(...) construct passes the command output as a file that Ansible can read.

  1. Or save the inventory to a file and run the check:

    d8 v ansible-inventory --list -o yaml -n demo-app > inventory.yaml
    ansible -m shell -a "uptime" -i inventory.yaml all
    

How to use cloud-init to configure virtual machines?

Cloud-init is used for initial guest OS configuration on first boot. The configuration is written in YAML and starts with the #cloud-config directive.

When using cloud images (for example, official distribution images), you must provide a cloud-init configuration. Without it, some distributions do not configure network connectivity, and the virtual machine becomes unreachable on the network, even if the main network (Main) is attached.

In addition, cloud images do not allow login by default — you must either add SSH keys for the default user or create a new user with SSH access. Otherwise, you will not be able to access the virtual machine.

Updating and installing packages

Example cloud-config for updating the system and installing packages from a list:

#cloud-config
# Update package lists.
package_update: true
# Upgrade installed packages to latest versions.
package_upgrade: true
# List of packages to install.
packages:
  - nginx
  - curl
  - htop
# Commands to run after package installation.
runcmd:
  - systemctl enable --now nginx.service

Creating a user

Example cloud-config for creating a local user with a password and SSH key:

#cloud-config
# List of users to create.
users:
    # Username.
  - name: cloud
    # Password hash.
    passwd: "<PASSWORD_HASH>"
    # Do not lock the account.
    lock_passwd: false
    # Sudo privileges without a password prompt.
    sudo: ALL=(ALL) NOPASSWD:ALL
    # Default shell.
    shell: /bin/bash
    # SSH keys for access.
    ssh-authorized-keys:
      - <SSH_PUBLIC_KEY>
# Allow password authentication via SSH.
ssh_pwauth: true

To generate a password hash for the passwd field, run:

mkpasswd --method=SHA-512 --rounds=4096

Creating a file with required permissions

Example cloud-config for creating a file with specified access permissions:

#cloud-config
# List of files to create.
write_files:
    # File path.
  - path: /opt/scripts/start.sh
    # File content.
    content: |
      #!/bin/bash
      echo "Starting application"
    # File owner, user and group.
    owner: cloud:cloud
    # Access permissions in octal format.
    permissions: '0755'

Configuring disk and filesystem

Example cloud-config for disk partitioning, filesystem creation, and mounting:

#cloud-config
# Disk partitioning setup.
disk_setup:
  # Disk device.
  /dev/sdb:
    # Partition table type, gpt or mbr.
    table_type: gpt
    # Automatically create partitions.
    layout: true
    # Do not overwrite existing partitions.
    overwrite: false

# Filesystem setup.
fs_setup:
    # Filesystem label.
  - label: data
    # Filesystem type.
    filesystem: ext4
    # Partition device.
    device: /dev/sdb1
    # Automatically detect partition.
    partition: auto

# Filesystem mounting.
mounts:
  # [device, mount_point, fs_type, options, dump, pass]
  - ["/dev/sdb1", "/mnt/data", "ext4", "defaults", "0", "2"]

Configuring network interfaces for additional networks

The settings described in this section apply only to additional networks. The main network (Main) is configured automatically via cloud-init and does not require manual configuration.

Additional networks are configured manually via cloud-init. The write_files block creates the configuration files, and the runcmd block applies the settings.

Connecting additional networks to a virtual machine is covered in Additional network interfaces.

The following examples cover common ways to configure networking in the guest OS:

  • systemd-networkd
  • Netplan (Ubuntu)
  • ifcfg (RHEL/CentOS)
  • Alpine Linux

Example cloud-config for distributions that use systemd-networkd (Debian, CoreOS, and others):

#cloud-config
write_files:
  - path: /etc/systemd/network/10-eth1.network
    content: |
      [Match]
      Name=eth1

      [Network]
      Address=192.168.1.10/24
      Gateway=192.168.1.1
      DNS=8.8.8.8

runcmd:
  - systemctl restart systemd-networkd

Example cloud-config for Ubuntu and other systems that use Netplan:

#cloud-config
write_files:
  - path: /etc/netplan/99-custom.yaml
    content: |
      network:
        version: 2
        ethernets:
          eth1:
            addresses:
              - 10.0.0.5/24
            gateway4: 10.0.0.1
            nameservers:
              addresses: [8.8.8.8]
          eth2:
            dhcp4: true

runcmd:
  - netplan apply

Example cloud-config for RHEL-compatible distributions that use the ifcfg scheme and NetworkManager:

#cloud-config
write_files:
  - path: /etc/sysconfig/network-scripts/ifcfg-eth1
    content: |
      DEVICE=eth1
      BOOTPROTO=none
      ONBOOT=yes
      IPADDR=192.168.1.10
      PREFIX=24
      GATEWAY=192.168.1.1
      DNS1=8.8.8.8

runcmd:
  - nmcli connection reload
  - nmcli connection up eth1

Example cloud-config for distributions that use the traditional /etc/network/interfaces format (Alpine and similar):

#cloud-config
write_files:
  - path: /etc/network/interfaces
    append: true
    content: |
      auto eth1
      iface eth1 inet static
          address 192.168.1.10
          netmask 255.255.255.0
          gateway 192.168.1.1

runcmd:
  - /etc/init.d/networking restart

How to increase the DVCR size?

The DVCR volume size is set in the virtualization module ModuleConfig (spec.settings.dvcr.storage.persistentVolumeClaim.size). The new value must be greater than the current one.

  1. Check the current DVCR size:

    d8 k get mc virtualization -o jsonpath='{.spec.settings.dvcr.storage.persistentVolumeClaim}'
    

    Example output:

    {"size":"58G","storageClass":"linstor-thick-data-r1"}
    
  2. Increase size using patch (set the value you need):

    d8 k patch mc virtualization \
      --type merge -p '{"spec": {"settings": {"dvcr": {"storage": {"persistentVolumeClaim": {"size":"59G"}}}}}}'
    

    Example output:

    moduleconfig.deckhouse.io/virtualization patched
    
  3. Verify that ModuleConfig shows the new size:

    d8 k get mc virtualization -o jsonpath='{.spec.settings.dvcr.storage.persistentVolumeClaim}'
    

    Example output:

    {"size":"59G","storageClass":"linstor-thick-data-r1"}
    
  4. Check the current DVCR status:

    d8 k get pvc dvcr -n d8-virtualization
    

    Example output:

    NAME STATUS VOLUME                                    CAPACITY    ACCESS MODES   STORAGECLASS           AGE
    dvcr Bound  pvc-6a6cedb8-1292-4440-b789-5cc9d15bbc6b  57617188Ki  RWO            linstor-thick-data-r1  7d
    

How to change the DVCR StorageClass when a PVC already exists?

You can change the DVCR storage StorageClass only by recreating the PVC. All images previously loaded into DVCR are lost, along with data for existing ClusterVirtualImage and VirtualImage resources.

The spec.settings.dvcr.storage.persistentVolumeClaim.storageClassName field in the virtualization module ModuleConfig sets the StorageClass for the virtual machine image storage volume (DVCR). While a PVC for that volume exists in the d8-virtualization namespace, you cannot change the field via the API.

You cannot change storageClassName on an existing PVC in place, and DVCR data is not migrated between storage classes.

To change the DVCR StorageClass, perform the following steps:

  1. Stop DVCR:

    d8 k -n d8-virtualization scale deployment dvcr --replicas=0
    
  2. List PVCs in the d8-virtualization namespace and find the PVC for the DVCR volume:

    d8 k get pvc -n d8-virtualization
    
  3. Delete the PVC you found. Replace <PVC_NAME> with the resource name. If the command fails because of insufficient permissions, run it as system:sudouser:

    d8 k --as system:sudouser -n d8-virtualization delete pvc/<PVC_NAME>
    
  4. Set the new StorageClass in ModuleConfig, substituting the class you need for <STORAGE_CLASS_NAME>:

    d8 k patch mc virtualization --type merge -p '{"spec":{"settings":{"dvcr":{"storage":{"persistentVolumeClaim":{"storageClassName":"<STORAGE_CLASS_NAME>"}}}}}}'
    

    Example output:

    moduleconfig.deckhouse.io/virtualization patched
    
  5. Start DVCR:

    d8 k -n d8-virtualization scale deployment dvcr --replicas=1
    
  6. Verify the PVC:

    d8 k get pvc -n d8-virtualization
    

    Example output:

    NAME   STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS          VOLUMEATTRIBUTESCLASS   AGE
    dvcr   Bound    pvc-b43f2e33-32cc-435a-aa1d-b53df35b030a   100Gi      RWO            linstor-thin-r1-hdd   <unset>                 34s
    

The storage for the chosen StorageClass must be reachable from the nodes where DVCR runs: system nodes, or worker nodes if the cluster has no system nodes.

How to install an operating system in a virtual machine from an ISO image?

Below is a typical Windows guest OS installation scenario from an ISO image. Before you begin, host the ISO on an HTTP endpoint reachable from the cluster.

  1. Create an empty VirtualDisk for OS installation:

    apiVersion: virtualization.deckhouse.io/v1alpha2
    kind: VirtualDisk
    metadata:
      name: win-disk
      namespace: default
    spec:
      persistentVolumeClaim:
        size: 100Gi
        storageClassName: local-path
    
  2. Create ClusterVirtualImage resources for the Windows OS ISO and the VirtIO driver ISO:

    apiVersion: virtualization.deckhouse.io/v1alpha2
    kind: ClusterVirtualImage
    metadata:
      name: win-11-iso
    spec:
      dataSource:
        type: HTTP
        http:
          url: "http://example.com/win11.iso"
    
    apiVersion: virtualization.deckhouse.io/v1alpha2
    kind: ClusterVirtualImage
    metadata:
      name: win-virtio-iso
    spec:
      dataSource:
        type: HTTP
        http:
          url: "https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso"
    
  3. Create a virtual machine (VM):

    apiVersion: virtualization.deckhouse.io/v1alpha2
    kind: VirtualMachine
    metadata:
      name: win-vm
      namespace: default
      labels:
        vm: win
    spec:
      virtualMachineClassName: generic
      runPolicy: Manual
      osType: Windows
      bootloader: EFI
      cpu:
        cores: 6
        coreFraction: 50%
      memory:
        size: 8Gi
      enableParavirtualization: true
      blockDeviceRefs:
        - kind: VirtualDisk
          name: win-disk
        - kind: ClusterVirtualImage
          name: win-11-iso
        - kind: ClusterVirtualImage
          name: win-virtio-iso
    
  4. Start the virtual machine:

    d8 v start win-vm
    
  5. Connect to the VM console and complete the OS installation and VirtIO drivers using the graphical installer.

    VNC connection:

    d8 v vnc -n default win-vm
    
  6. After the installation is complete, restart the virtual machine.

  7. For further work, connect via VNC again:

    d8 v vnc -n default win-vm
    

How to create a golden image for Linux?

A golden image is a pre-configured virtual machine (VM) image that can be used to quickly create new VMs with pre-installed software and settings.

  1. Create a virtual machine, install the required software on it, and perform all necessary configurations.

  2. Install and configure qemu-guest-agent (recommended):

    • For RHEL/CentOS:

      yum install -y qemu-guest-agent
      
    • For Debian/Ubuntu:

      apt-get update
      apt-get install -y qemu-guest-agent
      
  3. Enable and start the service:

    systemctl enable qemu-guest-agent
    systemctl start qemu-guest-agent
    
  4. Set the machine run policy to AlwaysOnUnlessStoppedManually, otherwise you will not be able to shut it down.

  5. Prepare the image. Clean unused filesystem blocks:

    fstrim -v /
    fstrim -v /boot
    
  6. Clean network settings:

    • For RHEL:

      nmcli con delete $(nmcli -t -f NAME,DEVICE con show | grep -v ^lo: | cut -d: -f1)
      rm -f /etc/sysconfig/network-scripts/ifcfg-eth*
      
    • For Debian/Ubuntu:

      rm -f /etc/network/interfaces.d/*
      
  7. Clean system identifiers:

    echo -n > /etc/machine-id
    rm -f /var/lib/dbus/machine-id
    ln -s /etc/machine-id /var/lib/dbus/machine-id
    
  8. Remove the SSH host keys:

    rm -f /etc/ssh/ssh_host_*
    
  9. Clean the systemd journal:

    journalctl --vacuum-size=100M --vacuum-time=7d
    
  10. Clean package manager cache:

    • For RHEL:

      yum clean all
      
    • For Debian/Ubuntu:

      apt-get clean
      
  11. Clean temporary files:

    rm -rf /tmp/*
    rm -rf /var/tmp/*
    
  12. Clean logs:

    find /var/log -name "*.log" -type f -exec truncate -s 0 {} \;
    
  13. Clean command history:

    history -c
    
  14. On RHEL, reset and restore the SELinux contexts in one of two ways.

    Restore the contexts right away:

    restorecon -R /
    

    Or schedule a relabel for the next boot:

    touch /.autorelabel
    
  15. Verify that /etc/fstab references UUID or LABEL rather than names like /dev/sdX:

    blkid
    cat /etc/fstab
    
  16. Reset cloud-init state (logs and seed):

    cloud-init clean --logs --seed
    
  17. Perform final synchronization and buffer cleanup:

    sync
    echo 3 > /proc/sys/vm/drop_caches
    
  18. Shut down the virtual machine:

    poweroff
    
  19. Create a VirtualImage resource that references the prepared VM’s VirtualDisk:

    d8 k apply -f -<<EOF
    apiVersion: virtualization.deckhouse.io/v1alpha2
    kind: VirtualImage
    metadata:
      name: <IMAGE_NAME>
      namespace: <NAMESPACE>
    spec:
      dataSource:
        type: ObjectRef
        objectRef:
          kind: VirtualDisk
          name: <SOURCE_DISK_NAME>
    EOF
    

    Or create a ClusterVirtualImage resource so the image is available cluster-wide for all projects:

    d8 k apply -f -<<EOF
    apiVersion: virtualization.deckhouse.io/v1alpha2
    kind: ClusterVirtualImage
    metadata:
      name: <IMAGE_NAME>
    spec:
      dataSource:
        type: ObjectRef
        objectRef:
          kind: VirtualDisk
          name: <SOURCE_DISK_NAME>
          namespace: <NAMESPACE>
    EOF
    

    Here, <IMAGE_NAME> is the name of the image being created, <NAMESPACE> is the namespace of the prepared machine, and <SOURCE_DISK_NAME> is the name of its disk.

  20. Create a new VirtualDisk from the resulting image:

    d8 k apply -f -<<EOF
    apiVersion: virtualization.deckhouse.io/v1alpha2
    kind: VirtualDisk
    metadata:
      name: <VM_DISK_NAME>
      namespace: <NAMESPACE>
    spec:
      dataSource:
        type: ObjectRef
        objectRef:
          kind: VirtualImage
          name: <IMAGE_NAME>
    EOF
    

    Here, <VM_DISK_NAME> is the name of the disk of the new machine.

After completing these steps, you will have a golden image that can be used to quickly create new virtual machines with pre-installed software and configurations.

How to restore the cluster if images from registry.deckhouse.io cannot be pulled after a license change?

After a license change on a cluster with containerd v1 and removal of the outdated license, images from registry.deckhouse.io may stop being pulled. Nodes then retain the outdated configuration file /etc/containerd/conf.d/dvcr.toml, which is not removed automatically. Because of it, the registry module does not start, and without it DVCR does not work.

Applying a NodeGroupConfiguration (NGC) manifest removes the file on the nodes. After the registry module starts, delete the manifest, since this is a one-time fix.

  1. Save the manifest to a file (for example, containerd-dvcr-remove-old-config.yaml):

    apiVersion: deckhouse.io/v1alpha1
    kind: NodeGroupConfiguration
    metadata:
      name: containerd-dvcr-remove-old-config.sh
    spec:
      weight: 32 # Must be in range 32–90.
      nodeGroups: ["*"]
      bundles: ["*"]
      content: |
        # Copyright 2023 Flant JSC
        # Licensed under the Apache License, Version 2.0 (the "License");
        # you may not use this file except in compliance with the License.
        # You may obtain a copy of the License at
        #      http://www.apache.org/licenses/LICENSE-2.0
        # Unless required by applicable law or agreed to in writing, software
        # distributed under the License is distributed on an "AS IS" BASIS,
        # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
        # See the License for the specific language governing permissions and
        # limitations under the License.
    
        rm -f /etc/containerd/conf.d/dvcr.toml
    
  2. Apply the saved manifest:

    d8 k apply -f containerd-dvcr-remove-old-config.yaml
    
  3. Verify that the registry module is running:

    d8 k -n d8-system -o yaml get secret registry-state | yq -C -P '.data | del .state | map_values(@base64d) | .conditions = (.conditions | from_yaml)'
    

    Example output when the registry module has started successfully:

    conditions:
    # ...
      - lastTransitionTime: "..."
        message: ""
        reason: ""
        status: "True"
        type: Ready
    
  4. Delete the one-time NodeGroupConfiguration manifest:

    d8 k delete -f containerd-dvcr-remove-old-config.yaml
    

The migration procedure is covered in Migrating container runtime to containerd v2.

How to redirect traffic to a virtual machine?

A virtual machine runs in a Kubernetes cluster, so traffic reaches it the same way it reaches any other workload. Routing is handled by the standard Kubernetes Service resource, which selects targets by labels.

  1. Create a service with the required settings.

    For example, consider a virtual machine with the label vm: frontend-0, an HTTP service exposed on ports 80 and 443, and SSH access on port 22:

    apiVersion: virtualization.deckhouse.io/v1alpha2
    kind: VirtualMachine
    metadata:
      name: frontend-0
      namespace: dev
      labels:
        vm: frontend-0
    spec: ...
    
  2. To route network traffic to the virtual machine’s ports, create a service:

    This service listens on ports 80 and 443 and forwards traffic to the corresponding ports of the target virtual machine. SSH access from outside is provided on port 2211:

    apiVersion: v1
    kind: Service
    metadata:
      name: frontend-0-svc
      namespace: dev
    spec:
      type: LoadBalancer
      ports:
      - name: ssh
        port: 2211
        protocol: TCP
        targetPort: 22
      - name: http
        port: 80
        protocol: TCP
        targetPort: 80
      - name: https
        port: 443
        protocol: TCP
        targetPort: 443
      selector:
        vm: frontend-0
    

Why is only one connection method available for a virtual machine?

You can connect to a virtual machine (VM) via the serial console (d8 v console) or VNC (d8 v vnc). These methods use different communication channels with the guest OS and depend on its configuration. Both methods are covered in Connecting to a virtual machine.

The sections below describe common situations where only one connection method works.

Why does VNC not work when the serial console is available?

VNC displays the guest OS screen and requires virtual terminal support in the kernel. The serial console works independently of the graphics subsystem.

Check in the guest OS whether virtual terminal support is enabled in the kernel configuration:

cat /boot/config-$(uname -r) | grep CONFIG_VT

The output should show CONFIG_VT=y:

CONFIG_VT=y

If the output shows CONFIG_VT is not set, rebuild the kernel with the option enabled or use an OS image with a suitable kernel configuration.

Why does the serial console not work when VNC is available?

The serial console connects to the ttyS0 port in the guest OS. If the getty service for this port is not running, d8 v console will not show a login prompt even though VNC continues to work.

In the guest OS, enable and start the serial-getty service for ttyS0:

sudo systemctl enable --now serial-getty@ttyS0.service

Then connect to the serial console again.

How to provide a Windows answer file (Sysprep)?

Unattended Windows installation uses an answer file (unattend.xml or autounattend.xml).

The example answer file below:

  • Sets the English UI language and keyboard layout.
  • Connects the VirtIO drivers for the setup stage (the order of devices in blockDeviceRefs on the VirtualMachine resource must match the paths in the file).
  • Creates disk layout for installation with EFI.
  • Creates the cloud administrator and the regular user account.
<p><b>Example of the contents of the autounattend.xml file…</b>&lt;/summary&gt;</p>

<div class="language-xml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">&lt;?xml version="1.0" encoding="utf-8"?&gt;</span> <span class="nt">&lt;unattend</span> <span class="na">xmlns=</span><span class="s">"urn:schemas-microsoft-com:unattend"</span> <span class="na">xmlns:wcm=</span><span class="s">"http://schemas.microsoft.com/WMIConfig/2002/State"</span><span class="nt">&gt;</span>   <span class="nt">&lt;settings</span> <span class="na">pass=</span><span class="s">"offlineServicing"</span><span class="nt">&gt;&lt;/settings&gt;</span>   <span class="nt">&lt;settings</span> <span class="na">pass=</span><span class="s">"windowsPE"</span><span class="nt">&gt;</span>
<span class="nt">&lt;component</span> <span class="na">name=</span><span class="s">"Microsoft-Windows-International-Core-WinPE"</span> <span class="na">processorArchitecture=</span><span class="s">"amd64"</span> <span class="na">publicKeyToken=</span><span class="s">"31bf3856ad364e35"</span> <span class="na">language=</span><span class="s">"neutral"</span> <span class="na">versionScope=</span><span class="s">"nonSxS"</span><span class="nt">&gt;</span>
  <span class="nt">&lt;SetupUILanguage&gt;</span>
    <span class="nt">&lt;UILanguage&gt;</span>en-US<span class="nt">&lt;/UILanguage&gt;</span>
  <span class="nt">&lt;/SetupUILanguage&gt;</span>
  <span class="nt">&lt;InputLocale&gt;</span>0409:00000409<span class="nt">&lt;/InputLocale&gt;</span>
  <span class="nt">&lt;SystemLocale&gt;</span>en-US<span class="nt">&lt;/SystemLocale&gt;</span>
  <span class="nt">&lt;UILanguage&gt;</span>en-US<span class="nt">&lt;/UILanguage&gt;</span>
  <span class="nt">&lt;UserLocale&gt;</span>en-US<span class="nt">&lt;/UserLocale&gt;</span>
<span class="nt">&lt;/component&gt;</span>
<span class="nt">&lt;component</span> <span class="na">name=</span><span class="s">"Microsoft-Windows-PnpCustomizationsWinPE"</span> <span class="na">processorArchitecture=</span><span class="s">"amd64"</span> <span class="na">publicKeyToken=</span><span class="s">"31bf3856ad364e35"</span> <span class="na">language=</span><span class="s">"neutral"</span> <span class="na">versionScope=</span><span class="s">"nonSxS"</span><span class="nt">&gt;</span>
  <span class="nt">&lt;DriverPaths&gt;</span>
    <span class="nt">&lt;PathAndCredentials</span> <span class="na">wcm:keyValue=</span><span class="s">"4b29ba63"</span> <span class="na">wcm:action=</span><span class="s">"add"</span><span class="nt">&gt;</span>
      <span class="nt">&lt;Path&gt;</span>E:\amd64\w11<span class="nt">&lt;/Path&gt;</span>
    <span class="nt">&lt;/PathAndCredentials&gt;</span>
    <span class="nt">&lt;PathAndCredentials</span> <span class="na">wcm:keyValue=</span><span class="s">"25fe51ea"</span> <span class="na">wcm:action=</span><span class="s">"add"</span><span class="nt">&gt;</span>
      <span class="nt">&lt;Path&gt;</span>E:\NetKVM\w11\amd64<span class="nt">&lt;/Path&gt;</span>
    <span class="nt">&lt;/PathAndCredentials&gt;</span>
  <span class="nt">&lt;/DriverPaths&gt;</span>
<span class="nt">&lt;/component&gt;</span>
<span class="nt">&lt;component</span> <span class="na">name=</span><span class="s">"Microsoft-Windows-Setup"</span> <span class="na">processorArchitecture=</span><span class="s">"amd64"</span> <span class="na">publicKeyToken=</span><span class="s">"31bf3856ad364e35"</span> <span class="na">language=</span><span class="s">"neutral"</span> <span class="na">versionScope=</span><span class="s">"nonSxS"</span><span class="nt">&gt;</span>
  <span class="nt">&lt;DiskConfiguration&gt;</span>
    <span class="nt">&lt;Disk</span> <span class="na">wcm:action=</span><span class="s">"add"</span><span class="nt">&gt;</span>
      <span class="nt">&lt;DiskID&gt;</span>0<span class="nt">&lt;/DiskID&gt;</span>
      <span class="nt">&lt;WillWipeDisk&gt;</span>true<span class="nt">&lt;/WillWipeDisk&gt;</span>
      <span class="nt">&lt;CreatePartitions&gt;</span>
        <span class="c">&lt;!-- Recovery partition --&gt;</span>
        <span class="nt">&lt;CreatePartition</span> <span class="na">wcm:action=</span><span class="s">"add"</span><span class="nt">&gt;</span>
          <span class="nt">&lt;Order&gt;</span>1<span class="nt">&lt;/Order&gt;</span>
          <span class="nt">&lt;Type&gt;</span>Primary<span class="nt">&lt;/Type&gt;</span>
          <span class="nt">&lt;Size&gt;</span>250<span class="nt">&lt;/Size&gt;</span>
        <span class="nt">&lt;/CreatePartition&gt;</span>
        <span class="c">&lt;!-- EFI system partition (ESP) --&gt;</span>
        <span class="nt">&lt;CreatePartition</span> <span class="na">wcm:action=</span><span class="s">"add"</span><span class="nt">&gt;</span>
          <span class="nt">&lt;Order&gt;</span>2<span class="nt">&lt;/Order&gt;</span>
          <span class="nt">&lt;Type&gt;</span>EFI<span class="nt">&lt;/Type&gt;</span>
          <span class="nt">&lt;Size&gt;</span>100<span class="nt">&lt;/Size&gt;</span>
        <span class="nt">&lt;/CreatePartition&gt;</span>
        <span class="c">&lt;!-- Microsoft reserved partition (MSR) --&gt;</span>
        <span class="nt">&lt;CreatePartition</span> <span class="na">wcm:action=</span><span class="s">"add"</span><span class="nt">&gt;</span>
          <span class="nt">&lt;Order&gt;</span>3<span class="nt">&lt;/Order&gt;</span>
          <span class="nt">&lt;Type&gt;</span>MSR<span class="nt">&lt;/Type&gt;</span>
          <span class="nt">&lt;Size&gt;</span>128<span class="nt">&lt;/Size&gt;</span>
        <span class="nt">&lt;/CreatePartition&gt;</span>
        <span class="c">&lt;!-- Windows partition --&gt;</span>
        <span class="nt">&lt;CreatePartition</span> <span class="na">wcm:action=</span><span class="s">"add"</span><span class="nt">&gt;</span>
          <span class="nt">&lt;Order&gt;</span>4<span class="nt">&lt;/Order&gt;</span>
          <span class="nt">&lt;Type&gt;</span>Primary<span class="nt">&lt;/Type&gt;</span>
          <span class="nt">&lt;Extend&gt;</span>true<span class="nt">&lt;/Extend&gt;</span>
        <span class="nt">&lt;/CreatePartition&gt;</span>
      <span class="nt">&lt;/CreatePartitions&gt;</span>
      <span class="nt">&lt;ModifyPartitions&gt;</span>
        <span class="c">&lt;!-- Recovery partition --&gt;</span>
        <span class="nt">&lt;ModifyPartition</span> <span class="na">wcm:action=</span><span class="s">"add"</span><span class="nt">&gt;</span>
          <span class="nt">&lt;Order&gt;</span>1<span class="nt">&lt;/Order&gt;</span>
          <span class="nt">&lt;PartitionID&gt;</span>1<span class="nt">&lt;/PartitionID&gt;</span>
          <span class="nt">&lt;Label&gt;</span>Recovery<span class="nt">&lt;/Label&gt;</span>
          <span class="nt">&lt;Format&gt;</span>NTFS<span class="nt">&lt;/Format&gt;</span>
          <span class="nt">&lt;TypeID&gt;</span>de94bba4-06d1-4d40-a16a-bfd50179d6ac<span class="nt">&lt;/TypeID&gt;</span>
        <span class="nt">&lt;/ModifyPartition&gt;</span>
        <span class="c">&lt;!-- EFI system partition (ESP) --&gt;</span>
        <span class="nt">&lt;ModifyPartition</span> <span class="na">wcm:action=</span><span class="s">"add"</span><span class="nt">&gt;</span>
          <span class="nt">&lt;Order&gt;</span>2<span class="nt">&lt;/Order&gt;</span>
          <span class="nt">&lt;PartitionID&gt;</span>2<span class="nt">&lt;/PartitionID&gt;</span>
          <span class="nt">&lt;Label&gt;</span>System<span class="nt">&lt;/Label&gt;</span>
          <span class="nt">&lt;Format&gt;</span>FAT32<span class="nt">&lt;/Format&gt;</span>
        <span class="nt">&lt;/ModifyPartition&gt;</span>
        <span class="c">&lt;!-- MSR partition does not need to be modified --&gt;</span>
        <span class="c">&lt;!-- Windows partition --&gt;</span>
        <span class="nt">&lt;ModifyPartition</span> <span class="na">wcm:action=</span><span class="s">"add"</span><span class="nt">&gt;</span>
          <span class="nt">&lt;Order&gt;</span>3<span class="nt">&lt;/Order&gt;</span>
          <span class="nt">&lt;PartitionID&gt;</span>4<span class="nt">&lt;/PartitionID&gt;</span>
          <span class="nt">&lt;Label&gt;</span>Windows<span class="nt">&lt;/Label&gt;</span>
          <span class="nt">&lt;Letter&gt;</span>C<span class="nt">&lt;/Letter&gt;</span>
          <span class="nt">&lt;Format&gt;</span>NTFS<span class="nt">&lt;/Format&gt;</span>
        <span class="nt">&lt;/ModifyPartition&gt;</span>
      <span class="nt">&lt;/ModifyPartitions&gt;</span>
    <span class="nt">&lt;/Disk&gt;</span>
    <span class="nt">&lt;WillShowUI&gt;</span>OnError<span class="nt">&lt;/WillShowUI&gt;</span>
  <span class="nt">&lt;/DiskConfiguration&gt;</span>
  <span class="nt">&lt;ImageInstall&gt;</span>
    <span class="nt">&lt;OSImage&gt;</span>
      <span class="nt">&lt;InstallTo&gt;</span>
        <span class="nt">&lt;DiskID&gt;</span>0<span class="nt">&lt;/DiskID&gt;</span>
        <span class="nt">&lt;PartitionID&gt;</span>4<span class="nt">&lt;/PartitionID&gt;</span>
      <span class="nt">&lt;/InstallTo&gt;</span>
    <span class="nt">&lt;/OSImage&gt;</span>
  <span class="nt">&lt;/ImageInstall&gt;</span>
  <span class="nt">&lt;UserData&gt;</span>
    <span class="nt">&lt;ProductKey&gt;</span>
      <span class="nt">&lt;Key&gt;&lt;PRODUCT_KEY&gt;&lt;/Key&gt;</span>
      <span class="nt">&lt;WillShowUI&gt;</span>OnError<span class="nt">&lt;/WillShowUI&gt;</span>
    <span class="nt">&lt;/ProductKey&gt;</span>
    <span class="nt">&lt;AcceptEula&gt;</span>true<span class="nt">&lt;/AcceptEula&gt;</span>
  <span class="nt">&lt;/UserData&gt;</span>
  <span class="nt">&lt;UseConfigurationSet&gt;</span>false<span class="nt">&lt;/UseConfigurationSet&gt;</span>
<span class="nt">&lt;/component&gt;</span>   <span class="nt">&lt;/settings&gt;</span>   <span class="nt">&lt;settings</span> <span class="na">pass=</span><span class="s">"generalize"</span><span class="nt">&gt;&lt;/settings&gt;</span>   <span class="nt">&lt;settings</span> <span class="na">pass=</span><span class="s">"specialize"</span><span class="nt">&gt;</span>
<span class="nt">&lt;component</span> <span class="na">name=</span><span class="s">"Microsoft-Windows-Deployment"</span> <span class="na">processorArchitecture=</span><span class="s">"amd64"</span> <span class="na">publicKeyToken=</span><span class="s">"31bf3856ad364e35"</span> <span class="na">language=</span><span class="s">"neutral"</span> <span class="na">versionScope=</span><span class="s">"nonSxS"</span><span class="nt">&gt;</span>
  <span class="nt">&lt;RunSynchronous&gt;</span>
    <span class="nt">&lt;RunSynchronousCommand</span> <span class="na">wcm:action=</span><span class="s">"add"</span><span class="nt">&gt;</span>
      <span class="nt">&lt;Order&gt;</span>1<span class="nt">&lt;/Order&gt;</span>
      <span class="nt">&lt;Path&gt;</span>powershell.exe -NoProfile -Command "$xml = [xml]::new(); $xml.Load('C:\Windows\Panther\unattend.xml'); $sb = [scriptblock]::Create( $xml.unattend.Extensions.ExtractScript ); Invoke-Command -ScriptBlock $sb -ArgumentList $xml;"<span class="nt">&lt;/Path&gt;</span>
    <span class="nt">&lt;/RunSynchronousCommand&gt;</span>
    <span class="nt">&lt;RunSynchronousCommand</span> <span class="na">wcm:action=</span><span class="s">"add"</span><span class="nt">&gt;</span>
      <span class="nt">&lt;Order&gt;</span>2<span class="nt">&lt;/Order&gt;</span>
      <span class="nt">&lt;Path&gt;</span>powershell.exe -NoProfile -Command "Get-Content -LiteralPath 'C:\Windows\Setup\Scripts\Specialize.ps1' -Raw | Invoke-Expression;"<span class="nt">&lt;/Path&gt;</span>
    <span class="nt">&lt;/RunSynchronousCommand&gt;</span>
    <span class="nt">&lt;RunSynchronousCommand</span> <span class="na">wcm:action=</span><span class="s">"add"</span><span class="nt">&gt;</span>
      <span class="nt">&lt;Order&gt;</span>3<span class="nt">&lt;/Order&gt;</span>
      <span class="nt">&lt;Path&gt;</span>reg.exe load "HKU\DefaultUser" "C:\Users\Default\NTUSER.DAT"<span class="nt">&lt;/Path&gt;</span>
    <span class="nt">&lt;/RunSynchronousCommand&gt;</span>
    <span class="nt">&lt;RunSynchronousCommand</span> <span class="na">wcm:action=</span><span class="s">"add"</span><span class="nt">&gt;</span>
      <span class="nt">&lt;Order&gt;</span>4<span class="nt">&lt;/Order&gt;</span>
      <span class="nt">&lt;Path&gt;</span>powershell.exe -NoProfile -Command "Get-Content -LiteralPath 'C:\Windows\Setup\Scripts\DefaultUser.ps1' -Raw | Invoke-Expression;"<span class="nt">&lt;/Path&gt;</span>
    <span class="nt">&lt;/RunSynchronousCommand&gt;</span>
    <span class="nt">&lt;RunSynchronousCommand</span> <span class="na">wcm:action=</span><span class="s">"add"</span><span class="nt">&gt;</span>
      <span class="nt">&lt;Order&gt;</span>5<span class="nt">&lt;/Order&gt;</span>
      <span class="nt">&lt;Path&gt;</span>reg.exe unload "HKU\DefaultUser"<span class="nt">&lt;/Path&gt;</span>
    <span class="nt">&lt;/RunSynchronousCommand&gt;</span>
  <span class="nt">&lt;/RunSynchronous&gt;</span>
<span class="nt">&lt;/component&gt;</span>   <span class="nt">&lt;/settings&gt;</span>   <span class="nt">&lt;settings</span> <span class="na">pass=</span><span class="s">"auditSystem"</span><span class="nt">&gt;&lt;/settings&gt;</span>   <span class="nt">&lt;settings</span> <span class="na">pass=</span><span class="s">"auditUser"</span><span class="nt">&gt;&lt;/settings&gt;</span>   <span class="nt">&lt;settings</span> <span class="na">pass=</span><span class="s">"oobeSystem"</span><span class="nt">&gt;</span>
<span class="nt">&lt;component</span> <span class="na">name=</span><span class="s">"Microsoft-Windows-International-Core"</span> <span class="na">processorArchitecture=</span><span class="s">"amd64"</span> <span class="na">publicKeyToken=</span><span class="s">"31bf3856ad364e35"</span> <span class="na">language=</span><span class="s">"neutral"</span> <span class="na">versionScope=</span><span class="s">"nonSxS"</span><span class="nt">&gt;</span>
  <span class="nt">&lt;InputLocale&gt;</span>0409:00000409<span class="nt">&lt;/InputLocale&gt;</span>
  <span class="nt">&lt;SystemLocale&gt;</span>en-US<span class="nt">&lt;/SystemLocale&gt;</span>
  <span class="nt">&lt;UILanguage&gt;</span>en-US<span class="nt">&lt;/UILanguage&gt;</span>
  <span class="nt">&lt;UserLocale&gt;</span>en-US<span class="nt">&lt;/UserLocale&gt;</span>
<span class="nt">&lt;/component&gt;</span>
<span class="nt">&lt;component</span> <span class="na">name=</span><span class="s">"Microsoft-Windows-Shell-Setup"</span> <span class="na">processorArchitecture=</span><span class="s">"amd64"</span> <span class="na">publicKeyToken=</span><span class="s">"31bf3856ad364e35"</span> <span class="na">language=</span><span class="s">"neutral"</span> <span class="na">versionScope=</span><span class="s">"nonSxS"</span><span class="nt">&gt;</span>
  <span class="nt">&lt;UserAccounts&gt;</span>
    <span class="nt">&lt;LocalAccounts&gt;</span>
      <span class="nt">&lt;LocalAccount</span> <span class="na">wcm:action=</span><span class="s">"add"</span><span class="nt">&gt;</span>
        <span class="nt">&lt;Name&gt;</span>cloud<span class="nt">&lt;/Name&gt;</span>
        <span class="nt">&lt;DisplayName&gt;</span>cloud<span class="nt">&lt;/DisplayName&gt;</span>
        <span class="nt">&lt;Group&gt;</span>Administrators<span class="nt">&lt;/Group&gt;</span>
        <span class="nt">&lt;Password&gt;</span>
          <span class="nt">&lt;Value&gt;&lt;ADMIN_PASSWORD&gt;&lt;/Value&gt;</span>
          <span class="nt">&lt;PlainText&gt;</span>true<span class="nt">&lt;/PlainText&gt;</span>
        <span class="nt">&lt;/Password&gt;</span>
      <span class="nt">&lt;/LocalAccount&gt;</span>
      <span class="nt">&lt;LocalAccount</span> <span class="na">wcm:action=</span><span class="s">"add"</span><span class="nt">&gt;</span>
        <span class="nt">&lt;Name&gt;</span>User<span class="nt">&lt;/Name&gt;</span>
        <span class="nt">&lt;DisplayName&gt;</span>user<span class="nt">&lt;/DisplayName&gt;</span>
        <span class="nt">&lt;Group&gt;</span>Users<span class="nt">&lt;/Group&gt;</span>
        <span class="nt">&lt;Password&gt;</span>
          <span class="nt">&lt;Value&gt;&lt;USER_PASSWORD&gt;&lt;/Value&gt;</span>
          <span class="nt">&lt;PlainText&gt;</span>true<span class="nt">&lt;/PlainText&gt;</span>
        <span class="nt">&lt;/Password&gt;</span>
      <span class="nt">&lt;/LocalAccount&gt;</span>
    <span class="nt">&lt;/LocalAccounts&gt;</span>
  <span class="nt">&lt;/UserAccounts&gt;</span>
  <span class="nt">&lt;AutoLogon&gt;</span>
    <span class="nt">&lt;Username&gt;</span>cloud<span class="nt">&lt;/Username&gt;</span>
    <span class="nt">&lt;Enabled&gt;</span>true<span class="nt">&lt;/Enabled&gt;</span>
    <span class="nt">&lt;LogonCount&gt;</span>1<span class="nt">&lt;/LogonCount&gt;</span>
    <span class="nt">&lt;Password&gt;</span>
      <span class="nt">&lt;Value&gt;&lt;ADMIN_PASSWORD&gt;&lt;/Value&gt;</span>
      <span class="nt">&lt;PlainText&gt;</span>true<span class="nt">&lt;/PlainText&gt;</span>
    <span class="nt">&lt;/Password&gt;</span>
  <span class="nt">&lt;/AutoLogon&gt;</span>
  <span class="nt">&lt;OOBE&gt;</span>
    <span class="nt">&lt;ProtectYourPC&gt;</span>3<span class="nt">&lt;/ProtectYourPC&gt;</span>
    <span class="nt">&lt;HideEULAPage&gt;</span>true<span class="nt">&lt;/HideEULAPage&gt;</span>
    <span class="nt">&lt;HideWirelessSetupInOOBE&gt;</span>true<span class="nt">&lt;/HideWirelessSetupInOOBE&gt;</span>
    <span class="nt">&lt;HideOnlineAccountScreens&gt;</span>false<span class="nt">&lt;/HideOnlineAccountScreens&gt;</span>
  <span class="nt">&lt;/OOBE&gt;</span>
  <span class="nt">&lt;FirstLogonCommands&gt;</span>
    <span class="nt">&lt;SynchronousCommand</span> <span class="na">wcm:action=</span><span class="s">"add"</span><span class="nt">&gt;</span>
      <span class="nt">&lt;Order&gt;</span>1<span class="nt">&lt;/Order&gt;</span>
      <span class="nt">&lt;CommandLine&gt;</span>powershell.exe -NoProfile -Command "Get-Content -LiteralPath 'C:\Windows\Setup\Scripts\FirstLogon.ps1' -Raw | Invoke-Expression;"<span class="nt">&lt;/CommandLine&gt;</span>
    <span class="nt">&lt;/SynchronousCommand&gt;</span>
  <span class="nt">&lt;/FirstLogonCommands&gt;</span>
<span class="nt">&lt;/component&gt;</span>   <span class="nt">&lt;/settings&gt;</span> <span class="nt">&lt;/unattend&gt;</span> </code></pre></div>    </div>

<p>&lt;/details&gt;</p>

<p>Replace <code class="language-plaintext highlighter-rouge">&lt;PRODUCT_KEY&gt;</code> with your Windows product key, and <code class="language-plaintext highlighter-rouge">&lt;ADMIN_PASSWORD&gt;</code> and <code class="language-plaintext highlighter-rouge">&lt;USER_PASSWORD&gt;</code> with the passwords of the accounts being created. Windows reads these passwords from the file in plain text, so don’t leave the finished answer file on a shared resource.</p>

<ol>
  <li>
    <p>Save the answer file as <code class="language-plaintext highlighter-rouge">autounattend.xml</code> (use the example above or adjust it to your needs).</p>
  </li>
  <li>
    <p>Create a secret with the type <code class="language-plaintext highlighter-rouge">provisioning.virtualization.deckhouse.io/sysprep</code>:</p>

    <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>d8 k create secret generic sysprep-config <span class="nt">--type</span><span class="o">=</span><span class="s2">"provisioning.virtualization.deckhouse.io/sysprep"</span> <span class="nt">--from-file</span><span class="o">=</span>./autounattend.xml </code></pre></div>        </div>
  </li>
  <li>
    <p>Create a virtual machine that will use the answer file during installation. Specify <code class="language-plaintext highlighter-rouge">provisioning</code> with type <code class="language-plaintext highlighter-rouge">SysprepRef</code> in the specification. If necessary, add other Base64-encoded files to the specification required for the answer file scripts to run successfully.</p>

    <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">virtualization.deckhouse.io/v1alpha2</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">VirtualMachine</span> <span class="na">metadata</span><span class="pi">:</span>   <span class="na">name</span><span class="pi">:</span> <span class="s">win-vm</span>   <span class="na">namespace</span><span class="pi">:</span> <span class="s">default</span>   <span class="na">labels</span><span class="pi">:</span>
<span class="na">vm</span><span class="pi">:</span> <span class="s">win</span> <span class="na">spec</span><span class="pi">:</span>   <span class="na">virtualMachineClassName</span><span class="pi">:</span> <span class="s">generic</span>   <span class="na">provisioning</span><span class="pi">:</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">SysprepRef</span>
<span class="na">sysprepRef</span><span class="pi">:</span>
  <span class="na">kind</span><span class="pi">:</span> <span class="s">Secret</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">sysprep-config</span>   <span class="na">runPolicy</span><span class="pi">:</span> <span class="s">AlwaysOn</span>   <span class="na">osType</span><span class="pi">:</span> <span class="s">Windows</span>   <span class="na">bootloader</span><span class="pi">:</span> <span class="s">EFI</span>   <span class="na">cpu</span><span class="pi">:</span>
<span class="na">cores</span><span class="pi">:</span> <span class="m">6</span>
<span class="na">coreFraction</span><span class="pi">:</span> <span class="s">50%</span>   <span class="na">memory</span><span class="pi">:</span>
<span class="na">size</span><span class="pi">:</span> <span class="s">8Gi</span>   <span class="na">enableParavirtualization</span><span class="pi">:</span> <span class="kc">true</span>   <span class="na">blockDeviceRefs</span><span class="pi">:</span>
<span class="pi">-</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">VirtualDisk</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">win-disk</span>
<span class="pi">-</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ClusterVirtualImage</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">win-11-iso</span>
<span class="pi">-</span> <span class="na">kind</span><span class="pi">:</span> <span class="s">ClusterVirtualImage</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">win-virtio-iso</span> </code></pre></div>        </div>
  </li>
</ol>