The module lifecycle stage: General Availability
Setting up the INTERNAL LoadBalancer
Attach the following annotation to the service:
yandex.cpi.flant.com/listener-subnet-id: SubnetID
The annotation links the LoadBalancer with the appropriate Subnet.
Reserving a public IP address
This on is used in externalIPAddresses and natInstanceExternalAddress. It also can be used for a bastion host.
$ yc vpc address create --external-ipv4 zone=ru-central1-a
id: e9b4cfmmnc1mhgij75n7
folder_id: b1gog0h9k05lhqe5d88l
created_at: "2020-09-01T09:29:33Z"
external_ipv4_address:
address: 178.154.226.159
zone_id: ru-central1-a
requirements: {}
reserved: true
dhcpOptions-related problems and ways to address them
Using DNS servers that differ from those provided by Yandex Cloud in the DHCP settings is a temporary solution. It will be abandoned after Yandex Cloud will introduce the Managed DNS service. To get around the restrictions described below, we recommend using stubZones from the kube-dns module.
Editing parameters
Pay attention to the following nuances:
- When changing parameters, you need to invoke
netplan applyor a similar command that forces the update of the DHCP lease. - You will need to restart all hostNetwork Pods (especially
kube-dns) for the newresolv.confsettings to take effect.
Aspects of the use
If the dhcpOptions parameter is set, all DNS are routed to the DNS servers specified. These DNS servers must serve DNS requests to the Internet and (if needed) resolve intranet resources.
Do not use this option if the recursive DNSs specified cannot resolve the same list of zones that the recursive DNSs in the Yandex Cloud subnet can resolve.
Setting a custom StorageClass as default
Specify the StorageClass name in the defaultClusterStorageClass parameter in the global module settings.
Note that after doing so, the storageclass.kubernetes.io/is-default-class='true' annotation will be removed from the StorageClass that was previously set as the default one.
d8 k edit mc global
Resizing a PVC
You can increase the size of an existing PVC by changing the value of the spec.resources.requests.storage parameter, without stopping or recreating the pod that uses it.
After the value of spec.resources.requests.storage is changed, the CSI driver performs the following operations:
- increases the size of the disk in Yandex Cloud;
- updates the size of the associated PersistentVolume;
- expands the file system on the node to which the volume is attached.
During the operation, the pod continues running, and the mounted volume remains available to the application. After the resize operation is complete, the new file system size becomes available inside the container without restarting the pod.
Decreasing the size of a PVC is not supported.
To increase the size of a PVC, follow these steps:
-
Get the name of the StorageClass used by the PVC:
d8 k -n <NAMESPACE> get pvc <PVC_NAME> \ -o jsonpath='{.spec.storageClassName}{"\n"}'Where:
<NAMESPACE>— the namespace containing the PVC;<PVC_NAME>— the name of the PVC to resize.
For example:
d8 k -n production get pvc application-data \ -o jsonpath='{.spec.storageClassName}{"\n"}'Example output:
network-ssdIn this example, the
application-dataPVC uses thenetwork-ssdStorageClass. -
Make sure that the StorageClass allows volume expansion:
d8 k get storageclass <STORAGECLASS_NAME> \ -o jsonpath='{.allowVolumeExpansion}{"\n"}'Where
<STORAGECLASS_NAME>is the name of the StorageClass obtained in the previous step.For example:
d8 k get storageclass network-ssd \ -o jsonpath='{.allowVolumeExpansion}{"\n"}'Example output:
true -
Check the current status and size of the PVC:
d8 k -n <NAMESPACE> get pvc <PVC_NAME>For example:
d8 k -n production get pvc application-dataExample output:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS application-data Bound pvc-65e92674-077c-4b4f-b65d-19e92f04e103 20Gi RWO network-ssdMake sure that:
- the PVC is in the
Boundstate; - the
CAPACITYfield shows the current PVC size; - the
STORAGECLASSfield shows the StorageClass checked in the previous step.
- the PVC is in the
-
Increase the PVC size:
d8 k -n <NAMESPACE> edit pvc <PVC_NAME>For example:
d8 k -n production edit pvc application-dataSpecify the new PVC size in the
spec.resources.requests.storagefield:spec: resources: requests: storage: 30GiIn this example, the PVC size is increased to
30Gi.Save the changes and close the editor.
For the
network-ssd-nonreplicatedandnetwork-ssd-io-m3StorageClasses, the size must be a multiple of 93Gi. -
Wait for the PVC resize operation to complete:
d8 k -n <NAMESPACE> get pvc <PVC_NAME> --watchWhere:
<NAMESPACE>— the namespace containing the PVC;<PVC_NAME>— the name of the PVC being resized.
For example:
d8 k -n production get pvc application-data --watchDuring the resize operation, the
CAPACITYfield may continue to show the previous size:NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS application-data Bound pvc-65e92674-077c-4b4f-b65d-19e92f04e103 20Gi RWO network-ssdThe operation is complete when the
CAPACITYfield shows the new PVC size:NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS application-data Bound pvc-65e92674-077c-4b4f-b65d-19e92f04e103 30Gi RWO network-ssd -
Check the PVC events:
d8 k -n <NAMESPACE> describe pvc <PVC_NAME>For example:
d8 k -n production describe pvc application-dataThe following events may appear during the resize operation:
ExternalExpanding Resizing FileSystemResizeRequiredThe following event indicates that the file system was successfully expanded:
FileSystemResizeSuccessfulFor example:
Normal FileSystemResizeSuccessful kubelet MountVolume.NodeExpandVolume succeeded for volume "pvc-65e92674-077c-4b4f-b65d-19e92f04e103" -
Check the file system size inside the pod:
d8 k -n <NAMESPACE> exec <POD_NAME> -- \ df -hT <MOUNT_PATH>Where:
<NAMESPACE>— the namespace containing the pod;<POD_NAME>— the name of the pod using the PVC;<MOUNT_PATH>— the path inside the container where the PVC is mounted.
For example:
d8 k -n production exec application-0 -- \ df -hT /dataExample output:
Filesystem Type Size Used Avail Use% Mounted on /dev/vde ext4 29.4G 22M 29.4G 1% /dataThe actual file system size may be slightly smaller than the PVC size due to file system metadata.
Adding CloudStatic nodes to a cluster
For VMs that you want to add to the cluster as nodes, add the node-network-cidr key to the metadata (Edit VM -> Metadata) with a value equal to the cluster’s nodeNetworkCIDR.
You can find out the nodeNetworkCIDR of the cluster using the command below:
d8 k -n kube-system get secret d8-provider-cluster-configuration -o json | jq --raw-output '.data."cloud-provider-cluster-configuration.yaml"' | base64 -d | grep '^nodeNetworkCIDR'
Creating a cluster in a new VPC and setting up a bastion host to access the nodes
-
Bootstrap the base-infrastructure of the cluster:
dhctl bootstrap-phase base-infra --config config.yml -
Create a bastion host:
yc compute instance create \ --name bastion \ --hostname bastion \ --create-boot-disk image-family=ubuntu-2404-lts,image-folder-id=standard-images,size=20,type=network-hdd \ --memory 2 \ --cores 2 \ --core-fraction 100 \ --ssh-key ~/.ssh/<SSH_PUBLIC_KEY_FILE> \ --zone ru-central1-a \ --public-address 178.154.226.159Replace
<SSH_PUBLIC_KEY_FILE>here with the name of your public key. For example, for a key with RSA encryption, it will beid_rsa.pub, and for a key with ED25519 encryption, it will be withid_ed25519.pub. -
Continue installing the cluster by specifying the bastion host data. Answer
yto the question about the Terraform cache:dhctl bootstrap --ssh-bastion-host=178.154.226.159 --ssh-bastion-user=yc-user \ --ssh-user=ubuntu --ssh-agent-private-keys=/tmp/.ssh/<SSH_PRIVATE_KEY_FILE> --config=/config.ymlReplace
<SSH_PRIVATE_KEY_FILE>here with the name of your private key. For example, for a key with RSA encryption it can beid_rsa, and for a key with ED25519 encryption it can beid_ed25519.
Long switching to nodes in lower-priority groups
If switching to nodes in lower-priority groups takes a long time (for example, when node groups with preemptible instances are set to the highest priority and, if such instances are unavailable, switching to nodes from other groups takes a very long time), follow the instructions.