The module lifecycle stageGeneral Availability

Setting up the INTERNAL LoadBalancer

Attach the following annotation to the service:

yandex.cpi.flant.com/listener-subnet-id: SubnetID

The annotation links the LoadBalancer with the appropriate Subnet.

Reserving a public IP address

This on is used in externalIPAddresses and natInstanceExternalAddress. It also can be used for a bastion host.

$ yc vpc address create --external-ipv4 zone=ru-central1-a
id: e9b4cfmmnc1mhgij75n7
folder_id: b1gog0h9k05lhqe5d88l
created_at: "2020-09-01T09:29:33Z"
external_ipv4_address:
  address: 178.154.226.159
  zone_id: ru-central1-a
  requirements: {}
reserved: true

Using DNS servers that differ from those provided by Yandex Cloud in the DHCP settings is a temporary solution. It will be abandoned after Yandex Cloud will introduce the Managed DNS service. To get around the restrictions described below, we recommend using stubZones from the kube-dns module.

Editing parameters

Pay attention to the following nuances:

  1. When changing parameters, you need to invoke netplan apply or a similar command that forces the update of the DHCP lease.
  2. You will need to restart all hostNetwork Pods (especially kube-dns) for the new resolv.conf settings to take effect.

Aspects of the use

If the dhcpOptions parameter is set, all DNS are routed to the DNS servers specified. These DNS servers must serve DNS requests to the Internet and (if needed) resolve intranet resources.

Do not use this option if the recursive DNSs specified cannot resolve the same list of zones that the recursive DNSs in the Yandex Cloud subnet can resolve.

Setting a custom StorageClass as default

Specify the StorageClass name in the defaultClusterStorageClass parameter in the global module settings. Note that after doing so, the storageclass.kubernetes.io/is-default-class='true' annotation will be removed from the StorageClass that was previously set as the default one.

d8 k edit mc global

Resizing a PVC

You can increase the size of an existing PVC by changing the value of the spec.resources.requests.storage parameter, without stopping or recreating the pod that uses it.

After the value of spec.resources.requests.storage is changed, the CSI driver performs the following operations:

  • increases the size of the disk in Yandex Cloud;
  • updates the size of the associated PersistentVolume;
  • expands the file system on the node to which the volume is attached.

During the operation, the pod continues running, and the mounted volume remains available to the application. After the resize operation is complete, the new file system size becomes available inside the container without restarting the pod.

Decreasing the size of a PVC is not supported.

To increase the size of a PVC, follow these steps:

  1. Get the name of the StorageClass used by the PVC:

    d8 k -n <NAMESPACE> get pvc <PVC_NAME> \
      -o jsonpath='{.spec.storageClassName}{"\n"}'
    

    Where:

    • <NAMESPACE> — the namespace containing the PVC;
    • <PVC_NAME> — the name of the PVC to resize.

    For example:

    d8 k -n production get pvc application-data \
      -o jsonpath='{.spec.storageClassName}{"\n"}'
    

    Example output:

    network-ssd
    

    In this example, the application-data PVC uses the network-ssd StorageClass.

  2. Make sure that the StorageClass allows volume expansion:

    d8 k get storageclass <STORAGECLASS_NAME> \
      -o jsonpath='{.allowVolumeExpansion}{"\n"}'
    

    Where <STORAGECLASS_NAME> is the name of the StorageClass obtained in the previous step.

    For example:

    d8 k get storageclass network-ssd \
      -o jsonpath='{.allowVolumeExpansion}{"\n"}'
    

    Example output:

    true
    
  3. Check the current status and size of the PVC:

    d8 k -n <NAMESPACE> get pvc <PVC_NAME>
    

    For example:

    d8 k -n production get pvc application-data
    

    Example output:

    NAME               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
    application-data   Bound    pvc-65e92674-077c-4b4f-b65d-19e92f04e103   20Gi       RWO            network-ssd
    

    Make sure that:

    • the PVC is in the Bound state;
    • the CAPACITY field shows the current PVC size;
    • the STORAGECLASS field shows the StorageClass checked in the previous step.
  4. Increase the PVC size:

    d8 k -n <NAMESPACE> edit pvc <PVC_NAME>
    

    For example:

    d8 k -n production edit pvc application-data
    

    Specify the new PVC size in the spec.resources.requests.storage field:

    spec:
      resources:
        requests:
          storage: 30Gi
    

    In this example, the PVC size is increased to 30Gi.

    Save the changes and close the editor.

    For the network-ssd-nonreplicated and network-ssd-io-m3 StorageClasses, the size must be a multiple of 93Gi.

  5. Wait for the PVC resize operation to complete:

    d8 k -n <NAMESPACE> get pvc <PVC_NAME> --watch
    

    Where:

    • <NAMESPACE> — the namespace containing the PVC;
    • <PVC_NAME> — the name of the PVC being resized.

    For example:

    d8 k -n production get pvc application-data --watch
    

    During the resize operation, the CAPACITY field may continue to show the previous size:

    NAME               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
    application-data   Bound    pvc-65e92674-077c-4b4f-b65d-19e92f04e103   20Gi       RWO            network-ssd
    

    The operation is complete when the CAPACITY field shows the new PVC size:

    NAME               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS
    application-data   Bound    pvc-65e92674-077c-4b4f-b65d-19e92f04e103   30Gi       RWO            network-ssd
    
  6. Check the PVC events:

    d8 k -n <NAMESPACE> describe pvc <PVC_NAME>
    

    For example:

    d8 k -n production describe pvc application-data
    

    The following events may appear during the resize operation:

    ExternalExpanding
    Resizing
    FileSystemResizeRequired
    

    The following event indicates that the file system was successfully expanded:

    FileSystemResizeSuccessful
    

    For example:

    Normal  FileSystemResizeSuccessful  kubelet  MountVolume.NodeExpandVolume succeeded for volume "pvc-65e92674-077c-4b4f-b65d-19e92f04e103"
    
  7. Check the file system size inside the pod:

    d8 k -n <NAMESPACE> exec <POD_NAME> -- \
      df -hT <MOUNT_PATH>
    

    Where:

    • <NAMESPACE> — the namespace containing the pod;
    • <POD_NAME> — the name of the pod using the PVC;
    • <MOUNT_PATH> — the path inside the container where the PVC is mounted.

    For example:

    d8 k -n production exec application-0 -- \
      df -hT /data
    

    Example output:

    Filesystem   Type   Size    Used   Avail   Use%   Mounted on
    /dev/vde     ext4   29.4G   22M    29.4G    1%    /data
    

    The actual file system size may be slightly smaller than the PVC size due to file system metadata.

Adding CloudStatic nodes to a cluster

For VMs that you want to add to the cluster as nodes, add the node-network-cidr key to the metadata (Edit VM -> Metadata) with a value equal to the cluster’s nodeNetworkCIDR.

You can find out the nodeNetworkCIDR of the cluster using the command below:

d8 k -n kube-system get secret d8-provider-cluster-configuration -o json | jq --raw-output '.data."cloud-provider-cluster-configuration.yaml"' | base64 -d | grep '^nodeNetworkCIDR'

Creating a cluster in a new VPC and setting up a bastion host to access the nodes

  1. Bootstrap the base-infrastructure of the cluster:

    dhctl bootstrap-phase base-infra --config config.yml
    
  2. Create a bastion host:

    yc compute instance create \
    --name bastion \
    --hostname bastion \
    --create-boot-disk image-family=ubuntu-2404-lts,image-folder-id=standard-images,size=20,type=network-hdd \
    --memory 2 \
    --cores 2 \
    --core-fraction 100 \
    --ssh-key ~/.ssh/<SSH_PUBLIC_KEY_FILE> \
    --zone ru-central1-a \
    --public-address 178.154.226.159
    

    Replace <SSH_PUBLIC_KEY_FILE> here with the name of your public key. For example, for a key with RSA encryption, it will be id_rsa.pub, and for a key with ED25519 encryption, it will be with id_ed25519.pub.

  3. Continue installing the cluster by specifying the bastion host data. Answer y to the question about the Terraform cache:

    dhctl bootstrap --ssh-bastion-host=178.154.226.159 --ssh-bastion-user=yc-user \
    --ssh-user=ubuntu --ssh-agent-private-keys=/tmp/.ssh/<SSH_PRIVATE_KEY_FILE> --config=/config.yml
    

    Replace <SSH_PRIVATE_KEY_FILE> here with the name of your private key. For example, for a key with RSA encryption it can be id_rsa, and for a key with ED25519 encryption it can be id_ed25519.

Long switching to nodes in lower-priority groups

If switching to nodes in lower-priority groups takes a long time (for example, when node groups with preemptible instances are set to the highest priority and, if such instances are unavailable, switching to nodes from other groups takes a very long time), follow the instructions.