Hybrid integrations

Deckhouse Kubernetes Platform (DKP) can use cloud provider resources to expand the capacity of static clusters. Currently, integration is supported with OpenStack and vSphere-based clouds.

A hybrid cluster is a Kubernetes cluster that combines bare-metal nodes with nodes running on vSphere or OpenStack. To create such a cluster, an L2 network must be available between all nodes.

Hybrid cluster with vSphere

Follow these steps:

1. Remove flannel from the kube-system namespace:

   kubectl -n kube-system delete ds flannel-ds
   

2. Configure the integration and set the required parameters.

Hybrid cluster with OpenStack

Follow these steps:

1. Remove flannel from the kube-system namespace:

   kubectl -n kube-system delete ds flannel-ds
   

2. Configure the integration and set the required parameters.
3. Create one or more OpenStackInstanceClass custom resources.
4. Create one or more NodeGroup resources to manage the number and provisioning of cloud-based VMs.

Storage integration

If you require PersistentVolumes on nodes connected to the cluster from OpenStack, you must create a StorageClass with the appropriate OpenStack volume type. You can get a list of available types using the following command:

openstack volume type list

Example for ceph-ssd volume type:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ceph-ssd
provisioner: csi-cinderplugin # Leave this as shown here.
parameters:
  type: ceph-ssd
volumeBindingMode: WaitForFirstConsumer

Hybrid cluster with Yandex Cloud

To create a hybrid cluster combining static nodes and nodes in Yandex Cloud, follow these steps.

Prerequisites

• A working cluster with the parameter clusterType: Static.
• The CNI controller switched to VXLAN mode. For details, refer to the tunnelMode parameter.
• Configured network connectivity between the static cluster node network and VCD (either at L2 level, or at L3 level with port access according to the required network policies for DKP operation).

Setup steps

1. Create a Service Account in the required Yandex Cloud folder:

   • Assign the editor role.
   • Provide access to the used VPC with the vpc.admin role.

2. Create the d8-provider-cluster-configuration secret with the required data. Example cloud-provider-cluster-configuration.yaml:

   apiVersion: deckhouse.io/v1
   kind: YandexClusterConfiguration
   layout: WithoutNAT
   masterNodeGroup:
     replicas: 1
     instanceClass:
       cores: 4
       memory: 8192
       imageID: fd80bm0rh4rkepi5ksdi
       diskSizeGB: 100
       platform: standard-v3
       externalIPAddresses:
       - "Auto"
   nodeNetworkCIDR: 10.160.0.0/16
   existingNetworkID: empty
   provider:
     cloudID: CLOUD_ID
     folderID: FOLDER_ID
     serviceAccountJSON: '{"id":"ajevk1dp8f9...--END PRIVATE KEY-----\n"}'
   sshPublicKey: <ssh-rsa SSHKEY>
   

   Parameter descriptions:

   • nodeNetworkCIDR — CIDR of the network covering all node subnets in Yandex Cloud.
   • cloudID — Your cloud ID.
   • folderID — Your folder ID.
   • serviceAccountJSON — The Service Account in JSON format.

   • sshPublicKey — Public SSH key for deployed machines.

     Values in masterNodeGroup are irrelevant since master nodes are not created.

3. Fill in data.cloud-provider-discovery-data.json in the same secret. Example:

   {
     "apiVersion": "deckhouse.io/v1",
     "defaultLbTargetGroupNetworkId": "empty",
     "internalNetworkIDs": [
       "<NETWORK-ID>"
     ],
     "kind": "YandexCloudDiscoveryData",
     "monitoringAPIKey": "",
     "region": "ru-central1",
     "routeTableID": "empty",
     "shouldAssignPublicIPAddress": false,
     "zoneToSubnetIdMap": {
       "ru-central1-a": "<A-SUBNET-ID>",
       "ru-central1-b": "<B-SUBNET-ID>", 
       "ru-central1-d": "<D-SUBNET-ID>"
     },
     "zones": [
       "ru-central1-a",
       "ru-central1-b",
      "ru-central1-d"
     ]
   }
   

   Parameter descriptions:

   • internalNetworkIDs — List of network IDs in Yandex Cloud providing internal node connectivity.
   • zoneToSubnetIdMap — Zone-to-subnet mapping (one subnet per zone).
   • shouldAssignPublicIPAddress: true — Assigns public IPs to created nodes if required. For zones without subnets, can be set to empty.

4. Encode the above files (YandexClusterConfiguration and YandexCloudDiscoveryData) in Base64, then insert them into the cloud-provider-cluster-configuration.yaml and cloud-provider-discovery-data.json fields in the secret.

   apiVersion: v1
   data:
     cloud-provider-cluster-configuration.yaml: <YANDEXCLUSTERCONFIGURATION_BASE64_ENCODED>
     cloud-provider-discovery-data.json: <YANDEXCLOUDDISCOVERYDATA-BASE64-ENCODED>
   kind: Secret
   metadata:
     labels:
       heritage: deckhouse
       name: d8-provider-cluster-configuration
     name: d8-provider-cluster-configuration
     namespace: kube-system
   type: Opaque
   ---
   apiVersion: deckhouse.io/v1alpha1
   kind: ModuleConfig
   metadata:
     name: cloud-provider-yandex
   spec:
     version: 1
     enabled: true
     settings:
       storageClass:
         default: network-ssd
   

5. Remove the ValidatingAdmissionPolicyBinding object to avoid conflicts:

   d8 k delete validatingadmissionpolicybindings.admissionregistration.k8s.io heritage-label-objects.deckhouse.io
   

6. Apply the manifests in the cluster.

7. Wait for the cloud-provider-yandex module activation and CRD creation:

   d8 k get mc cloud-provider-yandex
   d8 k get crd yandexinstanceclasses
   

8. Create and apply NodeGroup and YandexInstanceClass manifests:

   ---
   apiVersion: deckhouse.io/v1alpha1
   kind: NodeGroup
   metadata:
     name: worker
   spec:
     nodeType: Cloud
     cloudInstances:
       classReference:
         kind: YandexInstanceClass
         name: worker
       minPerZone: 1
       maxPerZone: 3
       zones:
         - ru-central1-d
   ---
   apiVersion: deckhouse.io/v1alpha1
   kind: YandexInstanceClass
   metadata:
     name: worker
   spec:
     cores: 4
     memory: 8192
     diskSizeGB: 50
     diskType: network-ssd
     mainSubnet: <YOUR-SUBNET-ID>
   

   The mainSubnet parameter must contain the subnet ID from Yandex Cloud that is used for interconnection with your infrastructure (L2 connectivity with static node groups).

   After applying the manifests, the provisioning of virtual machines in Yandex Cloud managed by the node-manager module will begin.

9. Check the machine-controller-manager logs for troubleshooting:

   d8 k -n d8-cloud-provider-yandex get machine
   d8 k -n d8-cloud-provider-yandex get machineset
   d8 k -n d8-cloud-instance-manager logs deploy/machine-controller-manager
   

Hybrid cluster with VCD

This section describes the process of creating a hybrid cluster that combines static (bare-metal) nodes and cloud nodes in VMware vCloud Director (VCD) using Deckhouse Kubernetes Platform (DKP).

Before you begin, ensure the following conditions are met:

• Infrastructure:
  • A bare-metal DKP cluster is installed.
  • A tenant is configured in VCD with allocated resources.
  • Configured network connectivity between the static cluster node network and VCD (either at L2 level, or at L3 level with port access according to the required network policies for DKP operation).
  • A working network is configured in VCD with DHCP enabled.
  • A user with a static password and VCD administrator privileges has been created.
• Software settings:
  • The CNI controller is switched to VXLAN mode. More details — tunnelMode configuration.
  • A list of required VCD resources is prepared (VDC, VAPP, templates, policies, etc.).

Setup

1. Create a configuration file cloud-provider-vcd-token.yml with the following content:

   apiVersion: deckhouse.io/v1
   kind: VCDClusterConfiguration
   layout: Standard
   mainNetwork: <NETWORK_NAME>
   internalNetworkCIDR: <NETWORK_CIDR>
   organization: <ORGANIZATION>
   virtualApplicationName: <VAPP_NAME>
   virtualDataCenter: <VDC_NAME>
   provider:
     server: <API_URL>
     apiToken: <PASSWORD>
     username: <USER_NAME>
     insecure: false
   masterNodeGroup:
     instanceClass:
       etcdDiskSizeGb: 10
       mainNetworkIPAddresses:
       - 192.168.199.2
       rootDiskSizeGb: 50
       sizingPolicy: <SIZING_POLICY>
       storageProfile: <STORAGE_PROFILE>
       template: <VAPP_TEMPLATE>
     replicas: 1
   sshPublicKey: <SSH_PUBLIC_KEY>
   

   Where:

   • mainNetwork — the name of the network where cloud nodes will be deployed in your VCD cluster.
   • internalNetworkCIDR — the CIDR of the specified network.
   • organization — the name of your VCD organization.
   • virtualApplicationName — the name of the vApp where nodes will be created (e.g., dkp-vcd-app).
   • virtualDataCenter — the name of the virtual data center.
   • template — the VM template used to create nodes.
   • sizingPolicy and storageProfile — corresponding policies configured in VCD.
   • provider.server — the API URL of your VCD instance.
   • provider.apiToken — the access token (password) of a user with administrator privileges in VCD.
   • provider.username — the name of the static user that will be used to interact with VCD.
   • mainNetworkIPAddresses — a list of IP addresses from the specified network that will be assigned to master nodes.
   • storageProfile — the name of the storage profile defining where the VM disks will be placed.

2. Encode the cloud-provider-vcd-token.yml file in Base64:

   base64 -i $PWD/cloud-provider-vcd-token.yml
   

3. Create a secret with the following content:

   apiVersion: v1
   data:
     cloud-provider-cluster-configuration.yaml: <BASE64_STRING_OBTAINED_IN_THE_PREVIOUS_STEP>
     cloud-provider-discovery-data.json: eyJhcGlWZXJzaW9uIjoiZGVja2hvdXNlLmlvL3YxIiwia2luZCI6IlZDRENsb3VkUHJvdmlkZXJEaXNjb3ZlcnlEYXRhIiwiem9uZXMiOlsiZGVmYXVsdCJdfQo=
   kind: Secret
     metadata:
       labels:
         heritage: deckhouse
         name: d8-provider-cluster-configuration
       name: d8-provider-cluster-configuration
       namespace: kube-system
   type: Opaque
   

4. Enable the cloud-provider-vcd module:

   d8 platform module enable cloud-provider-vcd
   

5. Edit the d8-cni-configuration secret so that the mode parameter is determined from mc cni-cilium (change .data.cilium to .data.necilium if necessary).

6. Verify that all pods in the d8-cloud-provider-vcd namespace are in the Running state:

   d8 k get pods -n d8-cloud-provider-vcd
   

7. Reboot the master node and wait for initialization to complete.

8. Create instance classes in VCD:

   apiVersion: deckhouse.io/v1
   kind: VCDInstanceClass
   metadata:
     name: worker
   spec:
     rootDiskSizeGb: 50
     sizingPolicy: <SIZING_POLICY>
     storageProfile: <STORAGE_PROFILE>
     template: <VAPP_TEMPLATE>
   

9. Create a NodeGroup resource:

   apiVersion: deckhouse.io/v1
   kind: NodeGroup
   metadata:
     name: worker
   spec:
     cloudInstances:
       classReference:
         kind: VCDInstanceClass
         name: worker
       maxPerZone: 2
       minPerZone: 1
     nodeTemplate:
       labels:
         node-role/worker: ""
     nodeType: CloudEphemeral
   

10. Verify that the required number of nodes has appeared in the cluster:

    d8 k get nodes -o wide