Deckhouse Kubernetes Platform on VMware vSphere

1
Selecting infrastructure
2
Installation information
3
Preparing environment
4
Installation
5
Getting access to the cluster
6
What can I do next?

The Deckhouse Kubernetes Platform on VMware vSphere installation is not available in the Community Edition. See information about commercial editions.

Before installation, ensure the following:

  • Cloud provider quotas for cluster deployment.
  • The cloud-init package is installed on the VMs. After the VM starts, services cloud-config.service, cloud-final.service, cloud-init.service must be running.
  • The virtual machine template contains only one disk.
  • In vSphere, VMware Guest OS Customization (and vApp/OS customization if applicable) is disabled for the cluster template and VMs.

Additional requirements and notes

  • For ContainerdV2 on cluster nodes, the OS on virtual machines must meet the requirements:
    • Linux kernel version 5.8 or newer;
    • CgroupsV2 support;
    • Systemd version 244 or newer;
    • erofs kernel module support.

    For more information, see the ClusterConfiguration resource.

  • From version 1.74, Deckhouse has a module integrity control mechanism (protection against replacement and modification). It turns on automatically when the OS on the nodes supports the erofs kernel module. Without it, Deckhouse runs as before but the mechanism is off — an alert will indicate it is unavailable.

To install Deckhouse Kubernetes Platform on VMware vSphere, you need vSphere version 7.x or 8.x with support for the Online volume expansion mechanism.

List of required vSphere resources

Deckhouse uses the ens192 interface as the default interface for virtual machines in vSphere. Therefore, when using static IP addresses in mainNetwork, you must create an interface named ens192 in the OS image as the default interface.

  • User with required set of privileges.
  • Network with DHCP server and access to the Internet
  • Datacenter with a tag in k8s-region category.
  • Cluster with a tag in k8s-zone category.
  • Datastore with required tags.
  • Template — the prepared VM image.

vSphere configuration

To configure tags, datastore tagging, and the Deckhouse service role through the VMware vSphere Client UI, follow Configuration via vSphere Client in the module documentation. The steps below use govc only.

Installing govc

You’ll need the vSphere CLI — govc — to proceed with the rest of the guide.

After the installation is complete, set the environment variables required to work with vCenter:

export GOVC_URL=example.com
export GOVC_USERNAME=<username>@vsphere.local
export GOVC_PASSWORD=<password>
export GOVC_INSECURE=1

Creating tags and tag categories

Instead of “regions” and “zones”, VMware vSphere provides Datacenter and Cluster objects. We will use tags to match them with “regions”/”zones”. These tags fall into two categories: one for “regions” tags and the other for “zones” tags.

Create a tag category using the following commands:

govc tags.category.create -d "Kubernetes Region" k8s-region
govc tags.category.create -d "Kubernetes Zone" k8s-zone

Create tags in each category. If you intend to use multiple “zones” (Cluster), create a tag for each one of them:

govc tags.create -d "Kubernetes Region" -c k8s-region test-region
govc tags.create -d "Kubernetes Zone Test 1" -c k8s-zone test-zone-1
govc tags.create -d "Kubernetes Zone Test 2" -c k8s-zone test-zone-2

Attach the “region” tag to Datacenter:

govc tags.attach -c k8s-region test-region /<DatacenterName>

Attach “zone” tags to Cluster objects:

govc tags.attach -c k8s-zone test-zone-1 /<DatacenterName>/host/<ClusterName1>
govc tags.attach -c k8s-zone test-zone-2 /<DatacenterName>/host/<ClusterName2>

Datastore configuration

For dynamic PersistentVolume provisioning, a Datastore must be available on each ESXi host (shared datastore).

Assign the “region” and “zone” tags to the Datastore objects to automatically create a StorageClass in the Kubernetes cluster:

govc tags.attach -c k8s-region test-region /<DatacenterName>/datastore/<DatastoreName1>
govc tags.attach -c k8s-zone test-zone-1 /<DatacenterName>/datastore/<DatastoreName1>

govc tags.attach -c k8s-region test-region /<DatacenterName>/datastore/<DatastoreName2>
govc tags.attach -c k8s-zone test-zone-2 /<DatacenterName>/datastore/<DatastoreName2>

Creating and assigning a role

We’ve intentionally skipped User creation since there are many ways to authenticate a user in the vSphere.

The role that you are asked to create next includes the privileges from the section List of required privileges. If you need a more granular Role, please contact your Deckhouse support.

Create a role with the required privileges:

govc role.create deckhouse \
  Cns.Searchable \
  Datastore.AllocateSpace Datastore.Browse Datastore.FileManagement \
  Folder.Create Folder.Delete Folder.Move Folder.Rename \
  Global.GlobalTag Global.SystemTag \
  InventoryService.Tagging.AttachTag InventoryService.Tagging.CreateCategory \
  InventoryService.Tagging.CreateTag InventoryService.Tagging.DeleteCategory \
  InventoryService.Tagging.DeleteTag InventoryService.Tagging.EditCategory \
  InventoryService.Tagging.EditTag InventoryService.Tagging.ModifyUsedByForCategory \
  InventoryService.Tagging.ModifyUsedByForTag InventoryService.Tagging.ObjectAttachable \
  Network.Assign \
  Resource.AssignVMToPool Resource.CreatePool Resource.DeletePool Resource.EditPool Resource.RenamePool \
  StorageProfile.View \
  System.Anonymous System.Read System.View \
  VApp.ApplicationConfig VApp.AssignResourcePool VApp.AssignVM VApp.Create VApp.Delete \
  VApp.ExtractOvfEnvironment VApp.Import VApp.InstanceConfig VApp.PowerOff VApp.PowerOn VApp.ResourceConfig \
  VirtualMachine.Config.AddExistingDisk VirtualMachine.Config.AddNewDisk VirtualMachine.Config.AddRemoveDevice \
  VirtualMachine.Config.AdvancedConfig VirtualMachine.Config.Annotation VirtualMachine.Config.CPUCount \
  VirtualMachine.Config.ChangeTracking VirtualMachine.Config.DiskExtend VirtualMachine.Config.DiskLease \
  VirtualMachine.Config.EditDevice VirtualMachine.Config.ManagedBy VirtualMachine.Config.Memory \
  VirtualMachine.Config.QueryUnownedFiles VirtualMachine.Config.RawDevice VirtualMachine.Config.ReloadFromPath \
  VirtualMachine.Config.RemoveDisk VirtualMachine.Config.Rename VirtualMachine.Config.ResetGuestInfo \
  VirtualMachine.Config.Resource VirtualMachine.Config.Settings VirtualMachine.Config.SwapPlacement \
  VirtualMachine.Config.UpgradeVirtualHardware \
  VirtualMachine.GuestOperations.Query \
  VirtualMachine.Interact.AnswerQuestion VirtualMachine.Interact.DeviceConnection \
  VirtualMachine.Interact.GuestControl VirtualMachine.Interact.PowerOff VirtualMachine.Interact.PowerOn \
  VirtualMachine.Interact.Reset VirtualMachine.Interact.SetCDMedia VirtualMachine.Interact.ToolsInstall \
  VirtualMachine.Inventory.Create VirtualMachine.Inventory.CreateFromExisting VirtualMachine.Inventory.Delete \
  VirtualMachine.Inventory.Move \
  VirtualMachine.Provisioning.Clone VirtualMachine.Provisioning.Customize VirtualMachine.Provisioning.DeployTemplate \
  VirtualMachine.Provisioning.GetVmFiles VirtualMachine.Provisioning.PutVmFiles VirtualMachine.Provisioning.ReadCustSpecs \
  VirtualMachine.State.CreateSnapshot VirtualMachine.State.RemoveSnapshot VirtualMachine.State.RenameSnapshot

Assign the role to a user on the vCenter object:

govc permissions.set -principal <username>@vsphere.local -role deckhouse /

Preparing a virtual machine image

It is recommended to use a pre-built cloud image/OVA file provided by the OS vendor to create a Template:

If you need to use your own image, please refer to the documentation.

Go back Next: Installation