The following describes the process of adding nodes from Yandex Cloud to an existing static Deckhouse Platform cluster.

Integration with Yandex Cloud uses the cloud-provider-yandex module. It provides interaction between DP and the Yandex Cloud API, retrieval of information about cloud infrastructure, creation of virtual machines, work with network parameters, and connection of nodes to an existing cluster.

This section describes two ways to add nodes:

  • Automatic node creation in Yandex Cloud. DP creates virtual machines through the Yandex Cloud API. VM parameters are defined by the YandexInstanceClass resource, and the required number of nodes and placement zones are defined by the NodeGroup resource with the CloudEphemeral type.
  • Connecting manually created nodes through a bootstrap script. A virtual machine is created by the user in advance and connected to the cluster using the DP bootstrap script. This scenario uses NodeGroup with the CloudStatic type.

Prerequisites

Before you begin, make sure that the following conditions are met:

Adding automatically created nodes

To run the preparation commands, you need the Yandex Cloud CLI (yc). You can use it on the administrator’s workstation. The yc CLI is not required on the cluster master node: only the prepared manifests need to be applied in the cluster.

  1. Get the identifiers of the cloud and folder where nodes will be created:

    yc resource-manager cloud list
    yc resource-manager folder list
    
  2. Specify the obtained identifiers in variables:

    export CLOUD_ID="<CLOUD_ID>"
    export FOLDER_ID="<FOLDER_ID>"
    

    Where:

    • CLOUD_ID — Yandex Cloud cloud ID;
    • FOLDER_ID — ID of the folder where resources will be created.
  3. Get the identifiers of the network, subnet, and zone where nodes will be created:

    yc vpc network list --folder-id "$FOLDER_ID"
    yc vpc subnet list --folder-id "$FOLDER_ID"
    yc compute zone list
    
  4. Specify the obtained values in variables:

    export NETWORK_ID="<NETWORK_ID>"
    export SUBNET_ID="<SUBNET_ID>"
    export ZONE="<ZONE>"
    export ROUTE_TABLE_ID="<ROUTE_TABLE_ID>"
    

    Where:

    • NETWORK_ID — VPC network ID;
    • SUBNET_ID — ID of the subnet where nodes will be created;
    • ZONE — availability zone that corresponds to the selected subnet, for example ru-central1-a;
    • ROUTE_TABLE_ID — ID of the route table attached to the subnet. It is shown in the ROUTE TABLE ID column of the yc vpc subnet list output.

    For details, see “Connecting and authorizing in Yandex Cloud”.

  5. Create a service account in the required Yandex Cloud folder and assign permissions to it:

    yc iam service-account create \
      --name dkp-hybrid \
      --folder-id "$FOLDER_ID"
    
    export SA_ID="$(yc iam service-account get \
      --name dkp-hybrid \
      --folder-id "$FOLDER_ID" \
      --format json | jq -r .id)"
    
    yc resource-manager folder add-access-binding "$FOLDER_ID" \
      --role editor \
      --subject "serviceAccount:${SA_ID}"
    
    yc resource-manager folder add-access-binding "$FOLDER_ID" \
      --role vpc.admin \
      --subject "serviceAccount:${SA_ID}"
    

    The editor role is required to create and manage cloud resources, and the vpc.admin role grants access to VPC network resources.

  6. Create a service account key and save it to a JSON file:

    yc iam key create \
      --service-account-id "$SA_ID" \
      --output dkp-hybrid-sa-key.json
    

    Save the service account JSON key to the SERVICE_ACCOUNT_JSON environment variable in single-line format:

    export SERVICE_ACCOUNT_JSON="$(jq -c . dkp-hybrid-sa-key.json)"
    
  7. Save the administrator’s public SSH key to the SSH_PUBLIC_KEY environment variable:

    export SSH_PUBLIC_KEY="$(cat ~/.ssh/id_rsa.pub)"
    

    If you use another key, specify its path instead of ~/.ssh/id_rsa.pub.

    The SSH_PUBLIC_KEY variable must contain the administrator’s public SSH key that will be used to access the nodes being created. Do not use the public key from the service account JSON file.

  8. Get the ID of the operating system image that will be used to create virtual machines and save it to the IMAGE_ID environment variable:

    export IMAGE_ID="$(yc compute image get-latest-from-family ubuntu-2404-lts \
      --folder-id standard-images \
      --format json | jq -r .id)"
    

    The IMAGE_ID variable must contain the OS image ID in Yandex Cloud. Do not use an existing virtual machine ID or a service account key ID.

  9. Specify the CIDR of the network where Yandex Cloud nodes will be placed:

    export NODE_NETWORK_CIDR="<NODE_NETWORK_CIDR>"
    

    NODE_NETWORK_CIDR is the CIDR that includes the internal IP addresses of Yandex Cloud nodes. For a single zone, it usually matches the CIDR of the selected subnet. For example, if nodes are created in the 10.128.0.0/24 subnet, specify 10.128.0.0/24. You can get the subnet CIDR with the following command:

    yc vpc subnet list --folder-id "$FOLDER_ID"
    
  10. Create the ModuleConfig manifest for the cloud-provider-yandex module. For example, yandex-module-config.yaml:

    cat > yandex-module-config.yaml <<EOF
    apiVersion: deckhouse.io/v1alpha1
    kind: ModuleConfig
    metadata:
      name: cloud-provider-yandex
    spec:
      version: 2
      enabled: true
      settings:
        provider:
          parameters:
            cloudID: ${CLOUD_ID}
            folderID: ${FOLDER_ID}
        nodes:
          parameters:
            layout: WithoutNAT
            nodeNetworkCIDR: ${NODE_NETWORK_CIDR}
            sshPublicKey: '${SSH_PUBLIC_KEY}'
            existingNetworkID: ${NETWORK_ID}
            existingRouteTableID: ${ROUTE_TABLE_ID}
            existingZoneToSubnetIDMap:
              ${ZONE}: ${SUBNET_ID}
            zones:
              - ${ZONE}
    EOF
    

    The manifest automatically uses the values of the environment variables set in the previous steps: CLOUD_ID, FOLDER_ID, NODE_NETWORK_CIDR, SSH_PUBLIC_KEY, NETWORK_ID, ZONE, and SUBNET_ID.

    The existingNetworkID, existingZoneToSubnetIDMap, and existingRouteTableID parameters describe the Yandex Cloud resources that already exist, because in a hybrid cluster DP does not create the network, the subnets, or the route table. The route table is required because the cloud controller manager creates the pod subnet routes in it and does not start with an empty value.

    The layout is set by the layout parameter. The available layouts are described in “Configuration and layouts”.

  11. Copy the yandex-module-config.yaml file to the cluster master node. Apply the manifest:

    d8 k apply -f yandex-module-config.yaml
    

    While DP is enabling the module, the validating webhooks of DP and the provider may be unavailable, and the command fails with failed calling webhook. This state is temporary, repeat the command in half a minute. The same applies to the next step with the Secret.

  12. Wait until the module creates the d8-cloud-provider-yandex namespace:

    d8 k get namespace d8-cloud-provider-yandex
    
  13. Create the Secret with the service account key:

    d8 k create -f - <<EOF
    apiVersion: v1
    kind: Secret
    metadata:
      name: d8-credentials
      namespace: d8-cloud-provider-yandex
    type: cloud-provider.deckhouse.io/credentials
    stringData:
      authScheme: serviceAccount
      secret: '${SERVICE_ACCOUNT_JSON}'
    EOF
    

    The Secret format is described in “Credentials Secret”.

  14. Wait until the cloud-provider-yandex module is enabled and the YandexInstanceClass resource appears:

    d8 k get module cloud-provider-yandex -o wide
    d8 k get crd yandexinstanceclasses.deckhouse.io
    
  15. Create a file with YandexInstanceClass and NodeGroup manifests. For example, yandex-instanceclass-nodegroup.yaml:

    apiVersion: deckhouse.io/v1
    kind: YandexInstanceClass
    metadata:
      name: yc-worker
    spec:
      cores: 4
      memory: 8192
      diskSizeGB: 50
      diskType: network-ssd
      mainSubnet: <SUBNET_ID>
    ---
    apiVersion: deckhouse.io/v1
    kind: NodeGroup
    metadata:
      name: yc-worker
    spec:
      nodeType: CloudEphemeral
      cloudInstances:
        classReference:
          kind: YandexInstanceClass
          name: yc-worker
        minPerZone: 1
        maxPerZone: 1
        zones:
          - ru-central1-a
    

    Where:

    • YandexInstanceClass describes the parameters of the virtual machine that will be created in Yandex Cloud;
    • mainSubnet — ID of the subnet from which the created nodes must have access to the static cluster nodes;
    • NodeGroup describes the node group that DP must maintain in the cluster;
    • nodeType: CloudEphemeral means that nodes will be created automatically through the cloud provider;
    • cloudInstances.zones must contain zones from the zones list in the module settings.

    In the WithoutNAT layout, the created nodes get a public IP address. To keep the nodes of the group on internal addresses only, set the assignPublicIPAddress: false parameter in the YandexInstanceClass and provide the nodes with another way out to the internet, for example through a subnet NAT gateway.

  16. Apply the manifest:

    d8 k apply -f yandex-instanceclass-nodegroup.yaml
    

    After the manifest is applied, DP will start creating a virtual machine in Yandex Cloud through machine-controller-manager.

  17. Check that the node appears in the cluster:

    d8 k get nodes -o wide
    

    Example expected output:

    NAME                                 STATUS   ROLES                  AGE   VERSION    INTERNAL-IP
    static-master-0                      Ready    control-plane,master   1h    v1.33.10   10.128.0.15
    yc-worker-f3564dca-7fc59-s2w5d       Ready    yc-worker              10m   v1.33.10   10.128.0.21
    
  18. To diagnose the state and troubleshoot possible issues, check the machine-controller-manager logs:

    d8 k -n d8-cloud-instance-manager get machinedeployments.machine.sapcloud.io -o wide
    d8 k -n d8-cloud-instance-manager get machinesets.machine.sapcloud.io -o wide
    d8 k -n d8-cloud-instance-manager get machines.machine.sapcloud.io -o wide
    d8 k -n d8-cloud-instance-manager logs deploy/machine-controller-manager --tail=200
    

Adding manually created nodes through a bootstrap script

Before you begin, make sure that the following conditions are met:

  • The cloud-provider-yandex module is enabled:

    d8 k get module cloud-provider-yandex -o wide
    
  • The cloud-provider-yandex module components are in the Running state:

    d8 k -n d8-cloud-provider-yandex get pods -o wide
    
  • A virtual machine that will be connected to the cluster has been created in Yandex Cloud.
  • The virtual machine is connected to the Yandex Cloud network and subnet used for hybrid integration with the cluster.
  • The virtual machine has a network interface in the Yandex Cloud VPC network and subnet used for hybrid integration with the cluster. The IP address of this interface must belong to the CIDR specified in nodeNetworkCIDR and be reachable from the static cluster nodes.
  • The virtual machine name in Yandex Cloud matches the hostname inside the operating system.
  • One of the package managers (apt/apt-get, yum, or rpm) for a supported OS is installed on the virtual machine.
  1. Check the virtual machine metadata in Yandex Cloud.

    The VM metadata must have cloud-init configured with the user that will be used for SSH connection.

    Example metadata:

    #cloud-config
    datasource:
      Ec2:
        strict_id: false
    ssh_pwauth: no
    users:
      - name: <USER>
        sudo: ALL=(ALL) NOPASSWD:ALL
        shell: /bin/bash
        ssh_authorized_keys:
          - <SSH_PUBLIC_KEY>
    

    Where:

    • <USER>: Username for SSH access to the virtual machine.
    • <SSH_PUBLIC_KEY>: Administrator’s public SSH key.
  2. On the master node, create a file with the NodeGroup manifest and specify the node group name. In this example and the following steps, the yc-manual name is used. For example, yandex-manual-nodegroup.yaml:

    d8 k apply -f - <<EOF
    apiVersion: deckhouse.io/v1
    kind: NodeGroup
    metadata:
      name: yc-manual
    spec:
      nodeType: CloudStatic
    EOF
    
  3. Make sure that the NodeGroup has been created and synchronized:

    d8 k get nodegroup yc-manual
    d8 k describe nodegroup yc-manual
    

    Example expected output:

    NAME        TYPE          READY   NODES   UPTODATE   INSTANCES   DESIRED   MIN   MAX   STANDBY   STATUS   AGE   SYNCED
    yc-manual   CloudStatic   0       0       0                                                               1m    True
    
  4. On the master node, get the bootstrap script for the created NodeGroup:

    NODE_GROUP=yc-manual
    
    d8 k -n d8-cloud-instance-manager get secret manual-bootstrap-for-${NODE_GROUP} \
      -o jsonpath='{.data.bootstrap\.sh}' > ${NODE_GROUP}-bootstrap.b64
    
  5. Copy the bootstrap script to the VM being connected. If SSH access to the VM is available from the master node, run the following on the master node:

    scp ${NODE_GROUP}-bootstrap.b64 <USER>@<NODE_PUBLIC_OR_INTERNAL_IP>:/tmp/bootstrap.b64
    

    To copy and run the bootstrap script, use the user specified in the VM metadata.

    If SSH access to the VM is available only from the administrator’s workstation, first copy the file from the master node to the workstation, and then from the workstation to the VM:

    scp <MASTER_USER>@<MASTER_IP>:/root/${NODE_GROUP}-bootstrap.b64 ./bootstrap.b64
    scp ./bootstrap.b64 <USER>@<NODE_PUBLIC_OR_INTERNAL_IP>:/tmp/bootstrap.b64
    

    Where:

    • <MASTER_USER>: User for SSH access to the master node.
    • <MASTER_IP>: IP address of the master node.
    • <USER>: User on the VM being connected.
    • <NODE_PUBLIC_OR_INTERNAL_IP>: Public or internal IP address of the VM being connected.
  6. On the VM being connected, decode the bootstrap script, set permissions, and run it:

    base64 -d /tmp/bootstrap.b64 > /tmp/bootstrap.sh
    chmod +x /tmp/bootstrap.sh
    
    sudo bash /tmp/bootstrap.sh
    

    After the bootstrap script is started, it will install the required components, configure the container runtime and kubelet, and connect the node to the cluster.

  7. On the master node, check that the new node has appeared:

    d8 k get nodes -o wide
    

    Example expected output:

    NAME                    STATUS   ROLES       AGE   VERSION    INTERNAL-IP   EXTERNAL-IP
    static-master-0         Ready    master      1h    v1.33.10   10.128.0.15   <none>
    yandex-worker-hybrid    Ready    yc-manual   5m    v1.33.10   10.128.0.17   <PUBLIC_IP>
    
  8. If connection fails, check the NodeGroup status, events, and component logs:

    d8 k get nodegroup yc-manual
    d8 k describe nodegroup yc-manual
    d8 k describe node yandex-worker-hybrid
    d8 k -n d8-cloud-instance-manager logs deploy/machine-controller-manager --tail=200
    

Additional resources