Cloud provider — Yandex Cloud: FAQ

Setting up the INTERNAL LoadBalancer

Attach the following annotation to the service:

yandex.cpi.flant.com/listener-subnet-id: SubnetID

The annotation links the LoadBalancer with the appropriate Subnet.

Using a separate target group for a NodeGroup

By default, Yandex Cloud Controller Manager adds all suitable cluster nodes to the default target group. To assign nodes to a separate target group, use the yandex.cpi.flant.com/target-group-name-prefix annotation. The annotation is described in the Service annotations section.

1. In the NodeGroup whose nodes should be included in a separate target group, specify the yandex.cpi.flant.com/target-group-name-prefix annotation in the spec.nodeTemplate.annotations parameter. For example:

   spec:
     nodeTemplate:
       annotations:
         yandex.cpi.flant.com/target-group-name-prefix: frontend-
   

   The annotation is propagated to the Node objects. Yandex Cloud Controller Manager adds all suitable nodes carrying the same annotation value to the target group, regardless of their NodeGroup.

2. In the Service object of the LoadBalancer type, specify the same annotation with the same value. For example:

   metadata:
     annotations:
       yandex.cpi.flant.com/target-group-name-prefix: frontend-
   

   The annotation on the Service identifies the target group used by the LoadBalancer but does not select nodes.

3. Make sure that the values of yandex.cpi.flant.com/target-group-name-prefix in the NodeGroup and Service match. In the example above, both resources use the frontend- value.

4. The target group will be created with a name generated according to the following pattern:

   <ANNOTATION_VALUE><YANDEX_CLOUD_CLUSTER_NAME><NETWORK_ID>
   

   For example, if the annotation value is frontend-, the cluster name is my-cluster-, and the network ID is enp123456789, the target group name will be:

   frontend-my-cluster-enp123456789
   

Configuration example:

apiVersion: deckhouse.io/v1
kind: NodeGroup
metadata:
  name: frontend
spec:
  nodeType: CloudEphemeral
  nodeTemplate:
    annotations:
      yandex.cpi.flant.com/target-group-name-prefix: frontend-
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-frontend
  annotations:
    yandex.cpi.flant.com/target-group-name-prefix: frontend-
spec:
  type: LoadBalancer

Nodes with a dedicated target group prefix are excluded from the default target group. As a result, load balancers that use the default target group stop routing traffic to those nodes.

This automatic transition is performed only between target groups managed by Yandex Cloud Controller Manager. If a target already belongs to another target group that is not managed by Yandex Cloud Controller Manager, remove it from that group before using the annotation.

After the prefix is changed or removed, the previous target group may remain empty. If it is no longer used by any load balancers, delete it manually.

Reserving a public IP address

This on is used in externalIPAddresses and natInstanceExternalAddress. It also can be used for a bastion host.

$ yc vpc address create --external-ipv4 zone=ru-central1-a
id: e9b4cfmmnc1mhgij75n7
folder_id: b1gog0h9k05lhqe5d88l
created_at: "2020-09-01T09:29:33Z"
external_ipv4_address:
  address: 178.154.226.159
  zone_id: ru-central1-a
  requirements: {}
reserved: true

dhcpOptions-related problems and ways to address them

Using DNS servers that differ from those provided by Yandex Cloud in the DHCP settings is a temporary solution. It will be abandoned after Yandex Cloud will introduce the Managed DNS service. To get around the restrictions described below, we recommend using stubZones from the kube-dns module.

Editing parameters

Pay attention to the following nuances:

1. When changing parameters, you need to invoke netplan apply or a similar command that forces the update of the DHCP lease.
2. You will need to restart all hostNetwork Pods (especially kube-dns) for the new resolv.conf settings to take effect.

Aspects of the use

If the dhcpOptions parameter is set, all DNS are routed to the DNS servers specified. These DNS servers must serve DNS requests to the Internet and (if needed) resolve intranet resources.

Do not use this option if the recursive DNSs specified cannot resolve the same list of zones that the recursive DNSs in the Yandex Cloud subnet can resolve.

Changing networkType does not recreate CloudEphemeral nodes

In clusters that use Machine Controller Manager (MCM), changing the networkType parameter in a YandexInstanceClass resource does not trigger automatic recreation of existing CloudEphemeral nodes. Although the MachineClass is updated, the network acceleration type of existing virtual machines in Yandex Cloud remains unchanged.

DKP intentionally does not take the networkType parameter into account when determining whether CloudEphemeral nodes should be recreated in MCM. If it did, updating DKP would recreate CloudEphemeral nodes in all clusters where networkType is specified, even if its value had not changed.

In Cluster API, the networkType parameter is taken into account when determining whether nodes should be recreated, so changing it automatically triggers node recreation. New CloudEphemeral NodeGroups in Yandex Cloud use Cluster API by default.

If you changed networkType in an MCM-based cluster and want the new value to apply to existing CloudEphemeral nodes, recreate them manually — instructions in the documentation.

Setting a custom StorageClass as default

Specify the StorageClass name in the defaultClusterStorageClass parameter in the global module settings. Note that after doing so, the storageclass.kubernetes.io/is-default-class='true' annotation will be removed from the StorageClass that was previously set as the default one.

d8 k edit mc global

Resizing a PVC

You can increase the size of an existing PVC by changing the value of the spec.resources.requests.storage parameter, without stopping or recreating the pod that uses it.

After the value of spec.resources.requests.storage is changed, the CSI driver performs the following operations:

• increases the size of the disk in Yandex Cloud;
• updates the size of the associated PersistentVolume;
• expands the file system on the node to which the volume is attached.

During the operation, the pod continues running, and the mounted volume remains available to the application. After the resize operation is complete, the new file system size becomes available inside the container without restarting the pod.

To increase the size of a PVC, follow these steps:

1. Get the name of the StorageClass used by the PVC:

   d8 k -n <NAMESPACE> get pvc <PVC_NAME> \
     -o jsonpath='{.spec.storageClassName}{"\n"}'
   

   Where:

   • <NAMESPACE> — the namespace containing the PVC;
   • <PVC_NAME> — the name of the PVC to resize.

   For example:

   d8 k -n production get pvc application-data \
     -o jsonpath='{.spec.storageClassName}{"\n"}'
   

   Example output:

   network-ssd
   

   In this example, the application-data PVC uses the network-ssd StorageClass.

2. Make sure that the StorageClass allows volume expansion:

   d8 k get storageclass <STORAGECLASS_NAME> \
     -o jsonpath='{.allowVolumeExpansion}{"\n"}'
   

   Where <STORAGECLASS_NAME> is the name of the StorageClass obtained in the previous step.

   For example:

   d8 k get storageclass network-ssd \
     -o jsonpath='{.allowVolumeExpansion}{"\n"}'
   

   Example output:

   true
   

3. Check the current status and size of the PVC:

   d8 k -n <NAMESPACE> get pvc <PVC_NAME>
   

   For example:

   d8 k -n production get pvc application-data
   

   Example output:

   NAME               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
   application-data   Bound    pvc-65e92674-077c-4b4f-b65d-19e92f04e103   20Gi       RWO            network-ssd
   

   Make sure that:

   • the PVC is in the Bound state;
   • the CAPACITY field shows the current PVC size;
   • the STORAGECLASS field shows the StorageClass checked in the previous step.

4. Increase the PVC size:

   d8 k -n <NAMESPACE> edit pvc <PVC_NAME>
   

   For example:

   d8 k -n production edit pvc application-data
   

   Specify the new PVC size in the spec.resources.requests.storage field:

   spec:
     resources:
       requests:
         storage: 30Gi
   

   In this example, the PVC size is increased to 30Gi.

   Save the changes and close the editor.

   For the network-ssd-nonreplicated and network-ssd-io-m3 StorageClasses, the size must be a multiple of 93Gi.

5. Wait for the PVC resize operation to complete:

   d8 k -n <NAMESPACE> get pvc <PVC_NAME> --watch
   

   Where:

   • <NAMESPACE> — the namespace containing the PVC;
   • <PVC_NAME> — the name of the PVC being resized.

   For example:

   d8 k -n production get pvc application-data --watch
   

   During the resize operation, the CAPACITY field may continue to show the previous size:

   NAME               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
   application-data   Bound    pvc-65e92674-077c-4b4f-b65d-19e92f04e103   20Gi       RWO            network-ssd
   

   The operation is complete when the CAPACITY field shows the new PVC size:

   NAME               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
   application-data   Bound    pvc-65e92674-077c-4b4f-b65d-19e92f04e103   30Gi       RWO            network-ssd
   

6. Check the PVC events:

   d8 k -n <NAMESPACE> describe pvc <PVC_NAME>
   

   For example:

   d8 k -n production describe pvc application-data
   

   The following events may appear during the resize operation:

   ExternalExpanding
   Resizing
   FileSystemResizeRequired
   

   The following event indicates that the file system was successfully expanded:

   FileSystemResizeSuccessful
   

   For example:

   Normal  FileSystemResizeSuccessful  kubelet  MountVolume.NodeExpandVolume succeeded for volume "pvc-65e92674-077c-4b4f-b65d-19e92f04e103"
   

7. Check the file system size inside the pod:

   d8 k -n <NAMESPACE> exec <POD_NAME> -- \
     df -hT <MOUNT_PATH>
   

   Where:

   • <NAMESPACE> — the namespace containing the pod;
   • <POD_NAME> — the name of the pod using the PVC;
   • <MOUNT_PATH> — the path inside the container where the PVC is mounted.

   For example:

   d8 k -n production exec application-0 -- \
     df -hT /data
   

   Example output:

   Filesystem   Type   Size    Used   Avail   Use%   Mounted on
   /dev/vde     ext4   29.4G   22M    29.4G    1%    /data
   

   The actual file system size may be slightly smaller than the PVC size due to file system metadata.

Adding CloudStatic nodes to a cluster

For VMs that you want to add to the cluster as nodes, add the node-network-cidr key to the metadata (Edit VM -> Metadata) with a value equal to the cluster’s nodeNetworkCIDR.

You can find out the nodeNetworkCIDR of the cluster using the command below:

d8 k -n kube-system get secret d8-provider-cluster-configuration -o json | jq --raw-output '.data."cloud-provider-cluster-configuration.yaml"' | base64 -d | grep '^nodeNetworkCIDR'

Creating a cluster in a new VPC and setting up a bastion host to access the nodes

1. Bootstrap the base-infrastructure of the cluster:

   dhctl bootstrap-phase base-infra --config config.yml
   

2. Create a bastion host:

   yc compute instance create \
   --name bastion \
   --hostname bastion \
   --create-boot-disk image-family=ubuntu-2404-lts,image-folder-id=standard-images,size=20,type=network-hdd \
   --memory 2 \
   --cores 2 \
   --core-fraction 100 \
   --ssh-key ~/.ssh/<SSH_PUBLIC_KEY_FILE> \
   --zone ru-central1-a \
   --public-address 178.154.226.159
   

   Replace <SSH_PUBLIC_KEY_FILE> here with the name of your public key. For example, for a key with RSA encryption, it will be id_rsa.pub, and for a key with ED25519 encryption, it will be with id_ed25519.pub.

3. Continue installing the cluster by specifying the bastion host data. Answer y to the question about the Terraform cache:

   dhctl bootstrap --ssh-bastion-host=178.154.226.159 --ssh-bastion-user=yc-user \
   --ssh-user=ubuntu --ssh-agent-private-keys=/tmp/.ssh/<SSH_PRIVATE_KEY_FILE> --config=/config.yml
   

   Replace <SSH_PRIVATE_KEY_FILE> here with the name of your private key. For example, for a key with RSA encryption it can be id_rsa, and for a key with ED25519 encryption it can be id_ed25519.

Long switching to nodes in lower-priority groups

If switching to nodes in lower-priority groups takes a long time (for example, when node groups with preemptible instances are set to the highest priority and, if such instances are unavailable, switching to nodes from other groups takes a very long time), follow the instructions.