Scaling and changing master nodes

Scaling and single/multi-master transition

Control plane operation modes

Deckhouse Kubernetes Platform (DKP) supports two operation modes for the control plane:

1. Single-master:
   • kube-apiserver uses only the local etcd instance.
   • A proxy server runs on the node to handle requests on localhost.
   • The kube-apiserver listens only on the master node’s IP address.
2. Multi-master:
   • kube-apiserver interacts with all etcd instances in the cluster.
   • A proxy is configured on all nodes:
     • If the local kube-apiserver is unavailable, requests are redirected to other nodes.
   • This ensures high availability and supports scaling.

Automatic scaling of master nodes

DKP allows automatic addition and removal of master nodes using the label node-role.kubernetes.io/control-plane="".

Automatic control of master nodes includes:

• Adding the label node-role.kubernetes.io/control-plane="" to a node:
  • All control plane components are deployed.
  • The node is added to the etcd cluster.
  • Certificates and configuration files are regenerated automatically.
• Removing the label:
  • Control plane components are removed.
  • The node is properly removed from the etcd cluster.
  • Related configuration files are updated.

Common scaling scenarios

DKP supports both automatic and manual scaling of master nodes in cloud and bare-metal clusters:

1. Single-master → Multi-master:

   • Add one or more master nodes.
   • Apply the label node-role.kubernetes.io/control-plane="" to them.
   • DKP will automatically:
     • Deploy all control plane components.
     • Configure the nodes to work with the etcd cluster.
     • Synchronize certificates and configuration files.

2. Multi-master → Single-master:

   • Remove the labels node-role.kubernetes.io/control-plane="" and node-role.kubernetes.io/master="" from the extra master nodes.
   • For bare-metal clusters:
     • To correctly remove the nodes from etcd:
       • Run d8 k delete node <node-name>;
       • Power off the corresponding VMs or servers.

1. Changing the number of master nodes in a cloud cluster:

   • Similar to node addition/removal, typically done using the dhctl converge command or cloud tools.

Removing the master role from a node without deleting the node itself

If you need to remove a node from the set of master nodes but keep it in the cluster for other purposes, follow these steps:

1. Remove the labels so the node is no longer treated as a master:

   d8 k label node <node-name> node-role.kubernetes.io/control-plane-
   d8 k label node <node-name> node-role.kubernetes.io/master-
   d8 k label node <node-name> node.deckhouse.io/group-
   

2. Make sure that the master node to be deleted is no longer listed as a member of the etcd cluster:

   for pod in $(d8 k -n kube-system get pod -l component=etcd,tier=control-plane -o name); do
     d8 k -n kube-system exec "$pod" -- etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt \
     --cert /etc/kubernetes/pki/etcd/ca.crt --key /etc/kubernetes/pki/etcd/ca.key \
     --endpoints https://127.0.0.1:2379/ member list -w table
     if [ $? -eq 0 ]; then
       break
     fi
   done
   

3. Delete the static manifests of the control plane components so they no longer start on the node, and remove unnecessary PKI files. Exec to the node and run the following commands:

   rm -f /etc/kubernetes/manifests/{etcd,kube-apiserver,kube-scheduler,kube-controller-manager}.yaml
   rm -f /etc/kubernetes/{scheduler,controller-manager}.conf
   rm -f /etc/kubernetes/authorization-webhook-config.yaml
   rm -f /etc/kubernetes/admin.conf /root/.kube/config
   rm -rf /etc/kubernetes/deckhouse
   rm -rf /etc/kubernetes/pki/{ca.key,apiserver*,etcd/,front-proxy*,sa.*}
   rm -rf /var/lib/etcd/member/
   

After completing these steps, the node will no longer be considered a master node, but it will remain part of the cluster and can be used for other tasks.

Changing the OS image of master nodes in a multi-master cluster

1. Create a backup of etcd and the /etc/kubernetes directory.
2. Copy the resulting archive outside the cluster (e.g., to a local machine).
3. Make sure there are no alerts in the cluster that could interfere with updating master nodes.
4. Ensure the DKP queue is empty.

5. On your local machine, run the Deckhouse installer container for the corresponding edition and version (adjust the container registry address if necessary):

   DH_VERSION=$(d8 k -n d8-system get deployment deckhouse -o jsonpath='{.metadata.annotations.core\.deckhouse\.io\/version}') 
   DH_EDITION=$(d8 k -n d8-system get deployment deckhouse -o jsonpath='{.metadata.annotations.core\.deckhouse\.io\/edition}' | tr '[:upper:]' '[:lower:]' ) 
   docker run --pull=always -it -v "$HOME/.ssh/:/tmp/.ssh/" \
     registry.deckhouse.io/deckhouse/${DH_EDITION}/install:${DH_VERSION} bash
   

6. In the installer container, run the following command to check the state before starting the operation:

   dhctl terraform check --ssh-agent-private-keys=/tmp/.ssh/<SSH_KEY_FILENAME> --ssh-user=<USERNAME> \
     --ssh-host <MASTER-NODE-0-HOST> --ssh-host <MASTER-NODE-1-HOST> --ssh-host <MASTER-NODE-2-HOST>
   

   The output should indicate that Terraform has found no discrepancies and no changes are required.

7. In the installer container, run the following command and specify the desired OS image in the masterNodeGroup.instanceClass parameter
   (provide all master node addresses using the --ssh-host parameter):

   dhctl config edit provider-cluster-configuration --ssh-agent-private-keys=/tmp/.ssh/<SSH_KEY_FILENAME> --ssh-user=<USERNAME> \
     --ssh-host <MASTER-NODE-0-HOST> --ssh-host <MASTER-NODE-1-HOST> --ssh-host <MASTER-NODE-2-HOST>
   

8. In the installer container, run the following command to update the nodes:

   Carefully review the actions that converge plans to perform when it prompts for confirmation.

   During execution, nodes will be replaced with new ones, one by one, starting from the highest numbered node (2) down to the lowest (0):

   dhctl converge --ssh-agent-private-keys=/tmp/.ssh/<SSH_KEY_FILENAME> --ssh-user=<USERNAME> \
     --ssh-host <MASTER-NODE-0-HOST> --ssh-host <MASTER-NODE-1-HOST> --ssh-host <MASTER-NODE-2-HOST>
   

   The following steps (9–12) should be performed sequentially on each master node, starting with the highest numbered node (with suffix 2) and ending with the lowest (with suffix 0).

9. On the newly created node, open the systemd journal for the bashible.service.
   Wait until the setup process is complete — the log should contain the message nothing to do:

   journalctl -fu bashible.service
   

10. Verify that the etcd node appears in the cluster node list:

    for pod in $(d8 k -n kube-system get pod -l component=etcd,tier=control-plane -o name); do
      d8 k -n kube-system exec "$pod" -- etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt \
      --cert /etc/kubernetes/pki/etcd/ca.crt --key /etc/kubernetes/pki/etcd/ca.key \
      --endpoints https://127.0.0.1:2379/ member list -w table
      if [ $? -eq 0 ]; then
        break
      fi
    done
    

11. Make sure that control-plane-manager is running on the node:

    d8 k -n kube-system wait pod --timeout=10m --for=condition=ContainersReady \
      -l app=d8-control-plane-manager --field-selector spec.nodeName=<MASTER-NODE-N-NAME>
    

12. Proceed to updating the next master node.

Changing the OS image in a single-master cluster

1. Convert the single-master cluster into a multi-master one according to the instructions.
2. Update the master nodes as described in the instructions.
3. Convert the multi-master cluster back to a single-master one following the instructions.

Adding master nodes to a static or hybrid cluster

It is important to have an odd number of masters to ensure a quorum.

When installing Deckhouse Kubernetes Platform with default settings, the NodeGroup master lacks the section spec.staticInstances.labelSelector with label filter settings for staticInstances resources. Because of this, after changing the number of staticInstances nodes in the NodeGroup master (parameter spec.staticInstances.count), when adding a regular node using Cluster API Provider Static (CAPS), it can be “intercepted” and added to the NodeGroup master, even if the corresponding StaticInstance (in metadata) specifies a label with a role different from master. To avoid this “interception”, after installing DKP, edit the NodeGroup master — add the section spec.staticInstances.labelSelector with label filter settings for staticInstances resources. Example of NodeGroup master with spec.staticInstances.labelSelector:

apiVersion: deckhouse.io/v1
kind: NodeGroup
metadata:
  name: master
spec:
  nodeType: Static
  staticInstances:
    count: 2
    labelSelector:
      matchLabels:
        role: master

Next, when adding master nodes to the cluster using CAPS, specify the label specified in spec.staticInstances.labelSelector NodeGroup master in the corresponding StaticInstance. Example:

apiVersion: deckhouse.io/v1alpha1
kind: StaticInstance
metadata:
  name: static-master-1
  labels:
    # The label specified in spec.staticInstances.labelSelector NodeGroup master.
    role: master
spec:
  # Specify the IP address of the static node server.
  address: "<SERVER-IP>"
  credentialsRef:
    kind: SSHCredentials
    name: credentials

Otherwise, adding a master node to a static or hybrid cluster is similar to adding a regular node. To do this, use the corresponding examples. All the necessary actions to configure a cluster control plane components on the new master nodes are performed automatically. Wait until the master nodes appear in Ready status.

Adding master nodes in a cloud cluster

This section describes how to convert a single-master cluster into a multi-master cluster.

1. Create a backup of etcd and the /etc/kubernetes directory.
2. Copy the resulting archive outside the cluster (e.g., to a local machine).
3. Ensure there are no active alerts in the cluster that may interfere with adding new master nodes.

4. Make sure the Deckhouse queue is empty:

   d8 platform queue list
   

5. On the local machine, run the Deckhouse installer container for the appropriate edition and version (adjust the container registry address if necessary):

   DH_VERSION=$(d8 k -n d8-system get deployment deckhouse -o jsonpath='{.metadata.annotations.core\.deckhouse\.io\/version}') 
   DH_EDITION=$(d8 k -n d8-system get deployment deckhouse -o jsonpath='{.metadata.annotations.core\.deckhouse\.io\/edition}' | tr '[:upper:]' '[:lower:]' ) 
   docker run --pull=always -it -v "$HOME/.ssh/:/tmp/.ssh/" \
     registry.deckhouse.io/deckhouse/${DH_EDITION}/install:${DH_VERSION} bash
   

6. In the installer container, run the following command to verify the state before proceeding:

   dhctl terraform check --ssh-agent-private-keys=/tmp/.ssh/<SSH_KEY_FILENAME> --ssh-user=<USERNAME> --ssh-host <MASTER-NODE-0-HOST>
   

   The output should confirm that Terraform found no differences and no changes are needed.

7. In the installer container, run the following command and set the target number of master nodes in the masterNodeGroup.replicas parameter:

   dhctl config edit provider-cluster-configuration --ssh-agent-private-keys=/tmp/.ssh/<SSH_KEY_FILENAME> --ssh-user=<USERNAME> \
     --ssh-host <MASTER-NODE-0-HOST>
   

   For Yandex Cloud, if public IPs are assigned to master nodes, the number of elements in the masterNodeGroup.instanceClass.externalIPAddresses array must match the number of master nodes. Even when using the Auto value (automatic public IP assignment), the number of items in the array must still match.

   For example, with three master nodes (masterNodeGroup.replicas: 3) and automatic IP assignment, the masterNodeGroup.instanceClass.externalIPAddresses section would look like:

   externalIPAddresses:
   - "Auto"
   - "Auto"
   - "Auto"
   

8. In the installer container, run the following command to trigger scaling:

   dhctl converge --ssh-agent-private-keys=/tmp/.ssh/<SSH_KEY_FILENAME> --ssh-user=<USERNAME> --ssh-host <MASTER-NODE-0-HOST>
   

9. Wait until the required number of master nodes reaches the Ready status and all control-plane-manager pods become ready:

   d8 k -n kube-system wait pod --timeout=10m --for=condition=ContainersReady -l app=d8-control-plane-manager
   

Reducing the number of master nodes in a cloud cluster

This section describes the process of converting a multi-master cluster into a single-master cluster.

1. Create a backup of etcd and the /etc/kubernetes directory.
2. Copy the resulting archive outside the cluster (e.g., to a local machine).
3. Ensure there are no alerts in the cluster that may interfere with the master node update process.

4. Make sure the DKP queue is empty:

   d8 platform queue list
   

5. On the local machine, run the DKP installer container for the corresponding edition and version (change the container registry address if needed):

   DH_VERSION=$(d8 k -n d8-system get deployment deckhouse -o jsonpath='{.metadata.annotations.core\.deckhouse\.io\/version}') 
   DH_EDITION=$(d8 k -n d8-system get deployment deckhouse -o jsonpath='{.metadata.annotations.core\.deckhouse\.io\/edition}' | tr '[:upper:]' '[:lower:]' ) 
   docker run --pull=always -it -v "$HOME/.ssh/:/tmp/.ssh/" \
     registry.deckhouse.io/deckhouse/${DH_EDITION}/install:${DH_VERSION} bash
   

6. In the installer container, run the following command and set masterNodeGroup.replicas to 1:

   dhctl config edit provider-cluster-configuration --ssh-agent-private-keys=/tmp/.ssh/<SSH_KEY_FILENAME> \
     --ssh-user=<USERNAME> --ssh-host <MASTER-NODE-0-HOST>
   

   For Yandex Cloud, if external IPs are used for master nodes, the number of items in the masterNodeGroup.instanceClass.externalIPAddresses array must match the number of master nodes. Even when using Auto (automatic public IP allocation), the number of entries must still match.

   For example, for a single master node (masterNodeGroup.replicas: 1) and automatic IP assignment, the masterNodeGroup.instanceClass.externalIPAddresses section would look like:

   externalIPAddresses:
   - "Auto"
   

7. Remove the following labels from the master nodes you plan to delete:
   • node-role.kubernetes.io/control-plane
   • node-role.kubernetes.io/master
   • node.deckhouse.io/group

   Command to remove the labels:

   d8 k label node <MASTER-NODE-N-NAME> node-role.kubernetes.io/control-plane- node-role.kubernetes.io/master- node.deckhouse.io/group-
   

8. Make sure the nodes to be removed are no longer part of the etcd cluster:

   for pod in $(d8 k -n kube-system get pod -l component=etcd,tier=control-plane -o name); do
     d8 k -n kube-system exec "$pod" -- etcdctl --cacert /etc/kubernetes/pki/etcd/ca.crt \
     --cert /etc/kubernetes/pki/etcd/ca.crt --key /etc/kubernetes/pki/etcd/ca.key \
     --endpoints https://127.0.0.1:2379/ member list -w table
     if [ $? -eq 0 ]; then
       break
     fi
   done
   

9. Drain the nodes to be removed:

   d8 k drain <MASTER-NODE-N-NAME> --ignore-daemonsets --delete-emptydir-data
   

10. Power off the corresponding VMs, delete their instances from the cloud, and detach any associated disks (e.g., kubernetes-data-master-<N>).

11. Delete any remaining pods on the removed nodes:

    d8 k delete pods --all-namespaces --field-selector spec.nodeName=<MASTER-NODE-N-NAME> --force
    

12. Delete the Node objects for the removed nodes:

    d8 k delete node <MASTER-NODE-N-NAME>
    

13. In the installer container, run the following command to trigger the scaling operation:

    dhctl converge --ssh-agent-private-keys=/tmp/.ssh/<SSH_KEY_FILENAME> --ssh-user=<USERNAME> --ssh-host <MASTER-NODE-0-HOST>
    

Accessing the DKP controller in a multi-master cluster

In clusters with multiple master nodes, DKP runs in high-availability mode (with multiple replicas). To access the active DKP controller, you can use the following command (example shown for the deckhouse-controller queue list command):

d8 platform queue list