Cloud provider — VMware vSphere: Preparing environment

Environment requirements

The following prerequisites must be met for Deckhouse Kubernetes Platform to work correctly with VMware vSphere:

• Access to vCenter;
• A user account with the required set of privileges;
• Tags and tag categories created in vSphere;
• Networks with DHCP and Internet access;
• Shared Datastore resources available on all ESXi hosts in use;
• vSphere version 7.x or 8.x with support for Online volume expansion;
• vCenter accessible from inside the cluster from the master nodes;
• A configured Datacenter that includes the following objects:
  1. VirtualMachine template:
     • The virtual machine image must use Virtual machines with hardware version 15 or later — this is required for online resize support.
     • The image must include the open-vm-tools, cloud-init, and cloud-init-vmware-guestinfo packages if the cloud-init version is lower than 21.3.
  2. Network:
     • The network must be available on all ESXi hosts where virtual machines are planned to be created.
  3. Datastore (one or more):
     • The Datastore must be connected to all ESXi hosts where virtual machines are planned to be created.
     • A tag from the category specified in the zoneTagCategory parameter must be assigned to the Datastore (default: k8s-zone). This tag defines the zone.
     • All Cluster objects within the same zone must have access to all Datastore objects in that zone.
  4. Cluster:
     • All ESXi hosts in use must be added to the Cluster.
     • A tag from the category specified in the zoneTagCategory parameter must be assigned to the Cluster (default: k8s-zone). This tag defines the zone.
  5. Folder for the virtual machines being created:
     • This parameter is optional.
     • By default, the root virtual machine folder is used.
  6. Role:
     • The role must include the required set of privileges.
  7. User:
     • The user must be assigned the role specified in the previous item.
• A tag from the category specified in the regionTagCategory parameter must be assigned to the created Datacenter (default: k8s-region). This tag defines the region.

List of required vSphere resources

• User with required set of privileges.
• Network with DHCP server and access to the Internet.
• Datacenter with a tag in k8s-region category.
• Cluster with a tag in k8s-zone category.
• Datastore with required tags.
• Template — prepared VM image.

List of required privileges

Read the Configuration via vSphere Client and Configuration via govc sections for details on how to create and assign a role to a user.

A detailed list of privileges required for Deckhouse Kubernetes Platform to work in vSphere:

vSphere configuration

Configuration in vSphere Client

Creating tags and tag categories in vSphere Client

VMware vSphere does not have built-in concepts of a “region” or a “zone”. In vSphere, a Datacenter acts as a region, and a Cluster acts as a zone. Tags are used to establish this mapping.

1. Open vSphere Client and go to “Menu” → “Tags & Custom Attributes” → “Tags”.

   

2. Open the “Categories” tab and click “NEW”. Create a category for regions (for example, k8s-region): set “Tags Per Object” to “One tag” and specify the applicable object types, including Datacenter.

   

3. Create a second category for zones (for example, k8s-zone) with the object types Host, Cluster, and Datastore.

   

4. Go to the “Tags” tab and create at least one tag in the region category and one tag in the zone category (for example, test-region, test-zone-1).

   

5. In the “Inventory” tab, select the target Datacenter, open the “Summary” panel, then choose “Actions” → “Tags & Custom Attributes” → “Assign Tag” and assign the region tag. Repeat this step for each Cluster that will host nodes, assigning the appropriate zone tags.

   

Configuring Datastore in vSphere Client

In the “Inventory” tab, select the Datastore, open the “Summary” panel, then choose “Actions” → “Tags & Custom Attributes” → “Assign Tag”. Assign the Datastore the same region tag as the corresponding Datacenter, and the same zone tag as the corresponding Cluster.

Creating and assigning a role in vSphere Client

1. Go to “Menu” → “Administration” → “Access Control” → “Roles”.

   

2. Click “NEW”, enter a role name (for example, deckhouse), and add the privileges from the list.

   

3. Assign the role to the Deckhouse service account: go to “Menu” → “Administration” → “Access Control” → “Global Permissions”, click “ADD”, and select the user and the deckhouse role.

   

Configuration via govc

Installing govc

To continue configuring vSphere, install the govc CLI utility.

After installation, set the environment variables required to connect to vCenter:

export GOVC_URL=example.com
export GOVC_USERNAME=<username>@vsphere.local
export GOVC_PASSWORD=<password>
export GOVC_INSECURE=1

Creating tags and tag categories with govc

VMware vSphere does not have built-in concepts of a “region” or a “zone”. In vSphere, a Datacenter acts as a region, and a Cluster acts as a zone. Tags are used to establish this mapping.

Create tag categories with the following commands:

govc tags.category.create -d "Kubernetes Region" k8s-region
govc tags.category.create -d "Kubernetes Zone" k8s-zone

Create tags in each category. If you intend to use multiple “zones” (Cluster), create a tag for each one of them:

govc tags.create -d "Kubernetes Region" -c k8s-region test-region
govc tags.create -d "Kubernetes Zone Test 1" -c k8s-zone test-zone-1
govc tags.create -d "Kubernetes Zone Test 2" -c k8s-zone test-zone-2

Attach the “region” tag to Datacenter:

govc tags.attach -c k8s-region test-region /<DatacenterName>

Attach “zone” tags to the Cluster objects:

govc tags.attach -c k8s-zone test-zone-1 /<DatacenterName>/host/<ClusterName1>
govc tags.attach -c k8s-zone test-zone-2 /<DatacenterName>/host/<ClusterName2>

Datastore configuration with govc

Assign the “region” and “zone” tags to the Datastore objects to automatically create a StorageClass in the Kubernetes cluster:

govc tags.attach -c k8s-region test-region /<DatacenterName>/datastore/<DatastoreName1>
govc tags.attach -c k8s-zone test-zone-1 /<DatacenterName>/datastore/<DatastoreName1>

govc tags.attach -c k8s-region test-region /<DatacenterName>/datastore/<DatastoreName2>
govc tags.attach -c k8s-zone test-zone-2 /<DatacenterName>/datastore/<DatastoreName2>

Creating and assigning a role with govc

Create a role with the required privileges:

govc role.create deckhouse \
    Cns.Searchable \
    Datastore.AllocateSpace Datastore.Browse Datastore.FileManagement \
    Folder.Create Folder.Delete Folder.Move Folder.Rename \
    Global.GlobalTag Global.SystemTag \
    InventoryService.Tagging.AttachTag InventoryService.Tagging.CreateCategory \
    InventoryService.Tagging.CreateTag InventoryService.Tagging.DeleteCategory \
    InventoryService.Tagging.DeleteTag InventoryService.Tagging.EditCategory \
    InventoryService.Tagging.EditTag InventoryService.Tagging.ModifyUsedByForCategory \
    InventoryService.Tagging.ModifyUsedByForTag InventoryService.Tagging.ObjectAttachable \
    Network.Assign \
    Resource.AssignVMToPool Resource.CreatePool Resource.DeletePool Resource.EditPool Resource.RenamePool \
    StorageProfile.View \
    System.Anonymous System.Read System.View \
    VApp.ApplicationConfig VApp.AssignResourcePool VApp.AssignVM VApp.Create VApp.Delete \
    VApp.ExtractOvfEnvironment VApp.Import VApp.InstanceConfig VApp.PowerOff VApp.PowerOn VApp.ResourceConfig \
    VirtualMachine.Config.AddExistingDisk VirtualMachine.Config.AddNewDisk VirtualMachine.Config.AddRemoveDevice \
    VirtualMachine.Config.AdvancedConfig VirtualMachine.Config.Annotation VirtualMachine.Config.CPUCount \
    VirtualMachine.Config.ChangeTracking VirtualMachine.Config.DiskExtend VirtualMachine.Config.DiskLease \
    VirtualMachine.Config.EditDevice VirtualMachine.Config.ManagedBy VirtualMachine.Config.Memory \
    VirtualMachine.Config.QueryUnownedFiles VirtualMachine.Config.RawDevice VirtualMachine.Config.ReloadFromPath \
    VirtualMachine.Config.RemoveDisk VirtualMachine.Config.Rename VirtualMachine.Config.ResetGuestInfo \
    VirtualMachine.Config.Resource VirtualMachine.Config.Settings VirtualMachine.Config.SwapPlacement \
    VirtualMachine.Config.UpgradeVirtualHardware \
    VirtualMachine.GuestOperations.Query \
    VirtualMachine.Interact.AnswerQuestion VirtualMachine.Interact.DeviceConnection \
    VirtualMachine.Interact.GuestControl VirtualMachine.Interact.PowerOff VirtualMachine.Interact.PowerOn \
    VirtualMachine.Interact.Reset VirtualMachine.Interact.SetCDMedia VirtualMachine.Interact.ToolsInstall \
    VirtualMachine.Inventory.Create VirtualMachine.Inventory.CreateFromExisting VirtualMachine.Inventory.Delete \
    VirtualMachine.Inventory.Move \
    VirtualMachine.Provisioning.Clone VirtualMachine.Provisioning.Customize VirtualMachine.Provisioning.DeployTemplate \
    VirtualMachine.Provisioning.GetVmFiles VirtualMachine.Provisioning.PutVmFiles VirtualMachine.Provisioning.ReadCustSpecs \
    VirtualMachine.State.CreateSnapshot VirtualMachine.State.RemoveSnapshot VirtualMachine.State.RenameSnapshot

Assign the role to a user on the vCenter object:

govc permissions.set -principal <username>@vsphere.local -role deckhouse /

VM image requirements

To create a VM template (Template), it is recommended to use a ready-made cloud image/OVA file provided by the OS vendor:

• Ubuntu
• Debian
• CentOS
• Rocky Linux (section Generic Cloud / OpenStack)

Preparing the virtual machine image

1. Install the required packages:

   If you use cloud-init version lower than 21.3 (VMware GuestInfo support is required):

   sudo apt-get update
   sudo apt-get install -y open-vm-tools cloud-init cloud-init-vmware-guestinfo
   

   If you use cloud-init version 21.3 or higher:

   sudo apt-get update
   sudo apt-get install -y open-vm-tools cloud-init
   

2. Verify that the disable_vmware_customization: false parameter is set in /etc/cloud/cloud.cfg.

3. Make sure the default_user parameter is specified in /etc/cloud/cloud.cfg. It is required to add an SSH key when the VM starts.

4. Add the VMware GuestInfo datasource — create /etc/cloud/cloud.cfg.d/99-DataSourceVMwareGuestInfo.cfg:

   datasource:
     VMware:
       vmware_cust_file_max_wait: 10
   

5. Before creating the VM template, reset the cloud-init identifiers and state using the following commands:

   truncate -s 0 /etc/machine-id &&
   rm /var/lib/dbus/machine-id &&
   ln -s /etc/machine-id /var/lib/dbus/machine-id
   

6. Clear cloud-init event logs:

   cloud-init clean --logs --seed
   

systemctl is-enabled cloud-config.service cloud-init.service cloud-final.service

enabled
enabled
enabled

Infrastructure

Networking

A VLAN with DHCP and Internet access is required for the running cluster:

• If the VLAN is public (public addresses), then you have to create a second network to deploy cluster nodes (DHCP is not needed in this network).
• If the VLAN is private (private addresses), then this network can be used for cluster nodes.

Inbound traffic

• You can use an internal load balancer (if present) and direct traffic directly to the front nodes of the cluster.
• If there is no load balancer, you can use MetalLB in BGP mode to organize fault-tolerant load balancers (recommended). In this case, front nodes of the cluster will have two interfaces. For this, you will need:
  • A dedicated VLAN for traffic exchange between BGP routers and MetalLB. This VLAN must have DHCP and Internet access.
  • IP addresses of BGP routers.
  • ASN — the AS number on the BGP router.
  • ASN — the AS number in the cluster.
  • A range to announce addresses from.

Using the datastore

Various types of storage can be used in the cluster; for the minimum configuration, you will need:

• Datastore for provisioning PersistentVolumes to the Kubernetes cluster.
• Datastore for provisioning root disks for the VMs (it can be the same Datastore as for PersistentVolume).