Hybrid cluster with DVP

This section describes how to add nodes from Deckhouse Virtualization Platform (DVP) to an existing static DKP cluster.

Integration with DVP uses the cloud-provider-dvp module. It enables DKP to interact with the DVP cluster API, create virtual machines, connect the created VMs to an existing Kubernetes cluster, and manage the lifecycle of nodes through Cluster API mechanisms.

This section describes two ways to add nodes:

• Automatically creating nodes in DVP. DKP creates virtual machines through the DVP API. VM parameters are specified using the DVPInstanceClass resource, while the required number of nodes and placement zones are specified using the NodeGroup resource with the CloudEphemeral type.
• Connecting manually created nodes using a bootstrap script. A virtual machine is created in advance by the user and connected to the cluster using a DKP bootstrap script. This scenario uses a NodeGroup with the CloudStatic type.

Prerequisites

Before you begin, make sure that the following requirements are met:

• The cluster is created with the clusterType: Static parameter.
• Network connectivity is configured between the network of the static DKP cluster nodes and the network of virtual machines in DVP. For details, see Network requirements. Nodes created in DVP have access to the Kubernetes API of the target DKP cluster, DNS, and the required addresses according to the Network interaction and Configuring network policies sections. If Cilium is used with pod traffic tunneling, the tunnelMode mode is selected according to the network connectivity between sites. The Kubernetes API of the DVP cluster is accessible from the DKP cluster.
• The requirements from the Preparing the environment section are met:
  • a ServiceAccount has been created to access the DVP API;
  • a kubeconfig has been generated to connect to the DVP API;
  • a namespace has been prepared where virtual machines and disks will be created.
• A Linux OS image with cloud-init support is available in DVP, for example ubuntu-24-04-lts.
• A suitable VirtualMachineClass is available in DVP, for example amd-epyc-gen-3.
• A StorageClass for root disks of virtual machines is available in DVP, for example replicated.
• If a virtual machine template is used, make sure that it contains only one disk.

Adding automatically created nodes

1. On the administrator’s machine where access to the DVP cluster is configured, prepare a kubeconfig for the cloud-provider-dvp module to access the DVP API.

   Follow the steps in the “Preparing the environment” section and encode the generated kubeconfig in Base64:

   export DVP_PROVIDER_KUBECONFIG="./kubeconfig"
   export DVP_KUBECONFIG_B64="$(base64 -w0 ${DVP_PROVIDER_KUBECONFIG})"
   

2. Specify the DVP namespace where virtual machines and disks will be created:

   export DVP_NAMESPACE="<DVP_NAMESPACE>"
   

3. Specify the DVP zone where nodes will be created.

   Currently, zoning in DVP is under development, so use the default value for the zones parameters in ModuleConfig and NodeGroup:

   export DVP_ZONE="default"
   

   If necessary, you can check the topology labels of nodes in the DVP cluster:

   d8 k get nodes -L topology.kubernetes.io/region,topology.kubernetes.io/zone
   

   The zone value in ModuleConfig and NodeGroup must match. Currently, only the default value is available in DVP.

4. Create a file with the cloud-provider-dvp module configuration. For example, cloud-provider-dvp-mc.yaml:

   cat > cloud-provider-dvp-mc.yaml <<EOF
   apiVersion: deckhouse.io/v1alpha1
   kind: ModuleConfig
   metadata:
     name: cloud-provider-dvp
   spec:
     enabled: true
     version: 1
     settings:
       provider:
         kubeconfigDataBase64: ${DVP_KUBECONFIG_B64}
         namespace: ${DVP_NAMESPACE}
       zones:
         - ${DVP_ZONE}
   EOF
   

   The manifest automatically uses the values of the environment variables set in the previous steps: DVP_KUBECONFIG_B64, DVP_NAMESPACE, and DVP_ZONE.

5. Apply ModuleConfig:

   d8 k apply -f cloud-provider-dvp-mc.yaml
   

6. Wait until the cloud-provider-dvp module is enabled:

   d8 k get module cloud-provider-dvp -o wide
   

   The module must switch to the Ready state, and the pods in the d8-cloud-provider-dvp namespace must be in the Running state.

7. Make sure that the node-manager module is in the Ready state:

   d8 k get module node-manager -o wide
   

   If the module is in the Error state, check that available DVP zones are specified in ModuleConfig and NodeGroup.

8. Make sure that the DVPInstanceClass resource is available in the cluster:

   d8 k get crd dvpinstanceclasses.deckhouse.io
   

9. On the administrator’s machine where access to the DVP cluster is configured, check the available virtual machine classes, images, and StorageClasses:

   d8 k --kubeconfig ${DVP_PROVIDER_KUBECONFIG} get virtualmachineclasses
   d8 k --kubeconfig ${DVP_PROVIDER_KUBECONFIG} get clustervirtualimages
   d8 k --kubeconfig ${DVP_PROVIDER_KUBECONFIG} get storageclasses
   

   Use the obtained values when creating DVPInstanceClass.

10. Create a file with DVPInstanceClass and NodeGroup resources. For example, dvp-instanceclass-nodegroup.yaml:

    apiVersion: deckhouse.io/v1alpha1
    kind: DVPInstanceClass
    metadata:
      name: dvp-worker
    spec:
      virtualMachine:
        cpu:
          cores: 3
          coreFraction: 20%
        memory:
          size: 6Gi
        virtualMachineClassName: <VIRTUAL_MACHINE_CLASS_NAME>
        bootloader: EFI
      rootDisk:
        size: 15Gi
        storageClass: <STORAGE_CLASS_NAME>
        image:
          kind: ClusterVirtualImage
          name: <CLUSTER_VIRTUAL_IMAGE_NAME>
    ---
    apiVersion: deckhouse.io/v1
    kind: NodeGroup
    metadata:
      name: dvp-worker
    spec:
      nodeType: CloudEphemeral
      cloudInstances:
        classReference:
          kind: DVPInstanceClass
          name: dvp-worker
        minPerZone: 1
        maxPerZone: 1
        zones:
          - default
    

    Where:

    • virtualMachineClassName — name of the VirtualMachineClass in DVP, for example amd-epyc-gen-3;
    • rootDisk.storageClass — name of the StorageClass in DVP, for example replicated;
    • rootDisk.image.kind — image source type. For a cluster image, use ClusterVirtualImage;
    • rootDisk.image.name — name of the OS image in DVP, for example ubuntu-24-04-lts;
    • cloudInstances.zones — DVP zone where the node will be created. The value must match the zones value in ModuleConfig.

11. Apply the manifest:

    d8 k apply -f dvp-instanceclass-nodegroup.yaml
    

    After the manifest is applied, DKP will start creating a virtual machine in DVP and connect it to the cluster as a node.

12. Check the NodeGroup status:

    d8 k get nodegroup dvp-worker -o wide
    d8 k describe nodegroup dvp-worker
    

13. Check that a new node appears in the DKP cluster:

    d8 k get nodes -o wide
    

    Example expected output:

    NAME                              STATUS   ROLES                  AGE   VERSION    INTERNAL-IP
    dvp-hybrid-master-0               Ready    control-plane,master   1h    v1.33.10   10.12.0.69
    dvp-worker-c75a75c1-twqp4-bjpvl   Ready    dvp-worker             10m   v1.33.10   10.12.3.15
    

Adding manually created nodes using a bootstrap script

Before you begin, make sure that the following requirements are met:

• The cloud-provider-dvp module is enabled:

  d8 k get module cloud-provider-dvp -o wide
  

• The cloud-provider-dvp module components are in the Running state:

  d8 k -n d8-cloud-provider-dvp get pods -o wide
  

• A virtual machine that will be connected to the cluster has been created in DVP.
• The virtual machine is connected to the DVP network used for hybrid integration with the cluster.
• The virtual machine IP address belongs to the range specified in internalNetworkCIDRs.
• The virtual machine name in DVP matches the hostname inside the operating system.
• SSH access is available on the virtual machine for copying and running the bootstrap script.
• The SSH user can run commands using sudo without entering a password.
• One of the package managers (apt/apt-get, yum, or rpm) for a supported OS is installed on the virtual machine.

1. Create a NodeGroup with the CloudStatic node type. In this example and the following steps, the cloud-static name is used:

   d8 k apply -f - <<EOF
   apiVersion: deckhouse.io/v1
   kind: NodeGroup
   metadata:
     name: cloud-static
   spec:
     nodeType: CloudStatic
   EOF
   

2. Make sure that NodeGroup has been created and synchronized:

   d8 k get nodegroup cloud-static
   

   Example expected output:

   NAME           TYPE          READY   NODES   UPTODATE   INSTANCES   DESIRED   MIN   MAX   STANDBY   STATUS   AGE   SYNCED
   cloud-static   CloudStatic   0       0       0                                                               1m    True
   

3. Get the bootstrap script for the created NodeGroup:

   NODE_GROUP=cloud-static
   
   d8 k -n d8-cloud-instance-manager get secret manual-bootstrap-for-${NODE_GROUP} \
     -o jsonpath='{.data.bootstrap\.sh}' > ${NODE_GROUP}-bootstrap.b64
   

4. On the master node, verify that the file contains valid Base64 data of the bootstrap script:

   base64 -d ${NODE_GROUP}-bootstrap.b64 > /dev/null
   

   Check the beginning of the decoded content:

   base64 -d ${NODE_GROUP}-bootstrap.b64 | head -n 5
   

   The decoded content must start with a bash script:

   #!/bin/bash
   ...
   

5. Copy the bootstrap script to the virtual machine being connected:

   scp ${NODE_GROUP}-bootstrap.b64 <USER>@<NODE_IP>:/tmp/bootstrap.b64
   

6. Connect to the virtual machine over SSH:

   ssh <USER>@<NODE_IP>
   

7. On the virtual machine, decode the bootstrap script, set permissions, and run it:

   base64 -d /tmp/bootstrap.b64 > /tmp/bootstrap.sh
   chmod +x /tmp/bootstrap.sh
   
   sudo bash /tmp/bootstrap.sh
   

   After the bootstrap script starts, it will install the required components, configure the container runtime and kubelet, and connect the node to the cluster.

8. On the master node, check that a new node appears:

   d8 k get nodes -o wide
   

   Example expected output:

   NAME                   STATUS   ROLES                  AGE   VERSION    INTERNAL-IP
   dvp-hybrid-master-0    Ready    control-plane,master   1h    v1.33.12   10.12.0.69
   cloud-static-worker-0  Ready    cloud-static           5m    v1.33.12   10.12.3.88
   

9. If connection fails, check the NodeGroup status, events, and bootstrap logs on the virtual machine being connected:

   d8 k get nodegroup cloud-static
   d8 k describe nodegroup cloud-static
   d8 k get events -A --sort-by=.lastTimestamp | tail -n 100
   

   On the virtual machine being connected:

   sudo tail -n 120 /var/log/d8/bashible/bootstrap.log
   

   If the logs contain the Failed to discover node_ip that matches internalNetworkCIDRs error, check that the virtual machine IP address belongs to internalNetworkCIDRs.

Additional resources

• Global custom resources
• Module cloud-provider-dvp documentation
• Module cni-cilium parameters
• Module node-manager custom resources
• Module virtualization documentation