Cloud provider — VMware vSphere: Preparing environment

Environment requirements

• vSphere version: 7.x or 8.x with support for the Online volume expansion mechanism.
• vCenter to which master nodes can connect to from within the cluster.
• Datacenter with the following components:
  1. VirtualMachine template.
     • VM image should use Virtual machines with hardware version 15 or later (required for online resize to work).
     • The following packages must be installed in the VM image: open-vm-tools, cloud-init and cloud-init-vmware-guestinfo (if the cloud-init version lower than 21.3 is used).
  2. The network must be available on all ESXi where VirtualMachines will be created.
  3. One or more Datastores connected to all ESXi where VirtualMachines will be created.
     • A tag from the tag category in zoneTagCategory (k8s-zone by default) must be added to Datastores. This tag will indicate the zone. All Clusters of a specific zone must have access to all Datastores within the same zone.
  4. The cluster with the required ESXis.
     • A tag from the tag category in zoneTagCategory (k8s-zone by default) must be added to the Cluster. This tag will indicate the zone.
  5. Folder for VirtualMachines to be created.
     • An optional parameter. By default, the root vm folder is used.
  6. Create a role with the appropriate set of privileges.
  7. Create a user and assign the above role to it.
• A tag from the tag category in regionTagCategory (k8s-region by default) must be added to the Datacenter. This tag will indicate the region.

List of required vSphere resources

• User with required set of permissions.
• Network with DHCP server and access to the Internet
• Datacenter with a tag in k8s-region category.
• Cluster with a tag in k8s-zone category.
• Datastore with required tags.
• Template — prepared VM image.

List of required privileges

Read the documentation on how to create and assign a role to a user.

A detailed list of privileges required for Deckhouse Kubernetes Platform to work in vSphere:

vSphere configuration

Installing govc

You’ll need the vSphere CLI — govc — to proceed with the rest of the guide.

After the installation is complete, set the environment variables required to work with vCenter:

export GOVC_URL=example.com
export GOVC_USERNAME=<username>@vsphere.local
export GOVC_PASSWORD=<password>
export GOVC_INSECURE=1

Creating tags and tag categories

Instead of “regions” and “zones”, VMware vSphere provides Datacenter and Cluster objects. We will use tags to match them with “regions”/”zones”. These tags fall into two categories: one for “regions” tags and the other for “zones” tags.

Create a tag category using the following commands:

govc tags.category.create -d "Kubernetes Region" k8s-region
govc tags.category.create -d "Kubernetes Zone" k8s-zone

Create tags in each category. If you intend to use multiple “zones” (Cluster), create a tag for each one of them:

govc tags.create -d "Kubernetes Region" -c k8s-region test-region
govc tags.create -d "Kubernetes Zone Test 1" -c k8s-zone test-zone-1
govc tags.create -d "Kubernetes Zone Test 2" -c k8s-zone test-zone-2

Attach the “region” tag to Datacenter:

govc tags.attach -c k8s-region test-region /<DatacenterName>

Attach “zone” tags to the Cluster objects:

govc tags.attach -c k8s-zone test-zone-1 /<DatacenterName>/host/<ClusterName1>
govc tags.attach -c k8s-zone test-zone-2 /<DatacenterName>/host/<ClusterName2>

Datastore configuration

Assign the “region” and “zone” tags to the Datastore objects to automatically create a StorageClass in the Kubernetes cluster:

govc tags.attach -c k8s-region test-region /<DatacenterName>/datastore/<DatastoreName1>
govc tags.attach -c k8s-zone test-zone-1 /<DatacenterName>/datastore/<DatastoreName1>

govc tags.attach -c k8s-region test-region /<DatacenterName>/datastore/<DatastoreName2>
govc tags.attach -c k8s-zone test-zone-2 /<DatacenterName>/datastore/<DatastoreName2>

Creating and assigning a role

Create a role with the corresponding permissions:

govc role.create deckhouse \
   Cns.Searchable Datastore.AllocateSpace Datastore.Browse Datastore.FileManagement \
   Global.GlobalTag Global.SystemTag Network.Assign StorageProfile.View \
   $(govc role.ls Admin | grep -F -e 'Folder.' -e 'InventoryService.' -e 'Resource.' -e 'VirtualMachine.')

Assign the role to a user on the vCenter object:

govc permissions.set -principal <username>@vsphere.local -role deckhouse /

VM image requirements

To create a VM template (Template), it is recommended to use a ready-made cloud image/OVA file provided by the OS vendor:

• Ubuntu
• Debian
• CentOS
• Rocky Linux (section Generic Cloud / OpenStack)

Preparing the virtual machine image

1. Install the required packages:

   If you use cloud-init version lower than 21.3 (VMware GuestInfo support is required):

   sudo apt-get update
   sudo apt-get install -y open-vm-tools cloud-init cloud-init-vmware-guestinfo
   

   If you use cloud-init version 21.3 or higher:

   sudo apt-get update
   sudo apt-get install -y open-vm-tools cloud-init
   

2. Verify that the disable_vmware_customization: false parameter is set in /etc/cloud/cloud.cfg.

3. Make sure the default_user parameter is specified in /etc/cloud/cloud.cfg. It is required to add an SSH key when the VM starts.

4. Add the VMware GuestInfo datasource — create /etc/cloud/cloud.cfg.d/99-DataSourceVMwareGuestInfo.cfg:

   datasource:
     VMware:
       vmware_cust_file_max_wait: 10
   

5. Before creating the VM template, reset identifiers and the cloud-init state:

   truncate -s 0 /etc/machine-id rm /var/lib/dbus/machine-id ln -s /etc/machine-id /var/lib/dbus/machine-id
   

6. Clear cloud-init event logs:

   cloud-init clean --logs --seed
   

systemctl is-enabled cloud-config.service cloud-init.service cloud-final.service

enabled
enabled
enabled

Infrastructure

Networking

A VLAN with DHCP and Internet access is required for the running cluster:

• If the VLAN is public (public addresses), then you have to create a second network to deploy cluster nodes (DHCP is not needed in this network).
• If the VLAN is private (private addresses), then this network can be used for cluster nodes.

Inbound traffic

• You can use an internal load balancer (if present) and direct traffic directly to the front nodes of the cluster.
• If there is no load balancer, you can use MetalLB in BGP mode to organize fault-tolerant load balancers (recommended). In this case, front nodes of the cluster will have two interfaces. For this, you will need:
  • A dedicated VLAN for traffic exchange between BGP routers and MetalLB. This VLAN must have DHCP and Internet access.
  • IP addresses of BGP routers.
  • ASN — the AS number on the BGP router.
  • ASN — the AS number in the cluster.
  • A range to announce addresses from.

Using the data store

Various types of storage can be used in the cluster; for the minimum configuration, you will need:

• Datastore for provisioning PersistentVolumes to the Kubernetes cluster.
• Datastore for provisioning root disks for the VMs (it can be the same Datastore as for PersistentVolume).