List of required vSphere resources

  • User with required set of permissions.
  • Network with DHCP server and access to the Internet
  • Datacenter with a tag in k8s-region category.
  • Cluster with a tag in k8s-zone category.
  • Datastore with required tags.
  • Template — prepared VM image.

vSphere configuration

Installing govc

You’ll need the vSphere CLI — govc — to proceed with the rest of the guide.

After the installation is complete, set the environment variables required to work with vCenter:

export GOVC_URL=example.com
export GOVC_USERNAME=<username>@vsphere.local
export GOVC_PASSWORD=<password>
export GOVC_INSECURE=1

Creating tags and tag categories

Instead of “regions” and “zones”, VMware vSphere provides Datacenter and Cluster objects. We will use tags to match them with “regions”/”zones”. These tags fall into two categories: one for “regions” tags and the other for “zones” tags.

Create a tag category using the following commands:

govc tags.category.create -d "Kubernetes Region" k8s-region
govc tags.category.create -d "Kubernetes Zone" k8s-zone

Create tags in each category. If you intend to use multiple “zones” (Cluster), create a tag for each one of them:

govc tags.create -d "Kubernetes Region" -c k8s-region test-region
govc tags.create -d "Kubernetes Zone Test 1" -c k8s-zone test-zone-1
govc tags.create -d "Kubernetes Zone Test 2" -c k8s-zone test-zone-2

Attach the “region” tag to Datacenter:

govc tags.attach -c k8s-region test-region /<DatacenterName>

Attach “zone” tags to the Cluster objects:

govc tags.attach -c k8s-zone test-zone-1 /<DatacenterName>/host/<ClusterName1>
govc tags.attach -c k8s-zone test-zone-2 /<DatacenterName>/host/<ClusterName2>

Datastore configuration

For dynamic PersistentVolume provisioning, a Datastore must be available on each ESXi host (shared datastore).

Assign the “region” and “zone” tags to the Datastore objects to automatically create a StorageClass in the Kubernetes cluster:

govc tags.attach -c k8s-region test-region /<DatacenterName>/datastore/<DatastoreName1>
govc tags.attach -c k8s-zone test-zone-1 /<DatacenterName>/datastore/<DatastoreName1>

govc tags.attach -c k8s-region test-region /<DatacenterName>/datastore/<DatastoreName2>
govc tags.attach -c k8s-zone test-zone-2 /<DatacenterName>/datastore/<DatastoreName2>

Creating and assigning a role

We’ve intentionally skipped User creation since there are many ways to authenticate a user in the vSphere.

This all-encompassing Role should be enough for all Deckhouse components. For a detailed list of privileges, refer to the documentation. If you need a more granular Role, please contact your Deckhouse support.

Create a role with the corresponding permissions:

govc role.create deckhouse \
   Cns.Searchable Datastore.AllocateSpace Datastore.Browse Datastore.FileManagement \
   Global.GlobalTag Global.SystemTag Network.Assign StorageProfile.View \
   $(govc role.ls Admin | grep -F -e 'Folder.' -e 'InventoryService.' -e 'Resource.' -e 'VirtualMachine.')

Assign the role to a user on the vCenter object:

govc permissions.set -principal <username>@vsphere.local -role deckhouse /

Preparing a virtual machine image

It is recommended to use a pre-built cloud image/OVA file provided by the OS vendor to create a Template:

Virtual machine image requirements

Deckhouse uses cloud-init to configure a virtual machine after startup. To do this, the following packages must be installed in the image:

Also, after the virtual machine is started, the following services associated with these packages must be started:

  • cloud-config.service
  • cloud-final.service
  • cloud-init.service

To add SSH keys to user’s authorized keys, the default_user parameter must be specified in the /etc/cloud/cloud.cfg file.

Deckhouse creates virtual machine disks of the eagerZeroedThick type, however, the disk type of the created VMs will be changed without any notice to match the VM Storage Policy as configured in vSphere. You can read more in the documentation.

Deckhouse uses the ens192 interface as the default interface for virtual machines in vSphere. Therefore, when using static IP addresses in mainNetwork, you must create an interface named ens192 in the OS image as the default interface.

Infrastructure

Networking

A VLAN with DHCP and Internet access is required for the running cluster:

  • If the VLAN is public (public addresses), then you have to create a second network to deploy cluster nodes (DHCP is not needed in this network).
  • If the VLAN is private (private addresses), then this network can be used for cluster nodes.

Inbound traffic

  • You can use an internal load balancer (if present) and direct traffic directly to the front nodes of the cluster.
  • If there is no load balancer, you can use MetalLB in BGP mode to organize fault-tolerant load balancers (recommended). In this case, front nodes of the cluster will have two interfaces. For this, you will need:
    • A dedicated VLAN for traffic exchange between BGP routers and MetalLB. This VLAN must have DHCP and Internet access.
    • IP addresses of BGP routers.
    • ASN — the AS number on the BGP router.
    • ASN — the AS number in the cluster.
    • A range to announce addresses from.

Using the data store

Various types of storage can be used in the cluster; for the minimum configuration, you will need:

  • Datastore for provisioning PersistentVolumes to the Kubernetes cluster.
  • Datastore for provisioning root disks for the VMs (it can be the same Datastore as for PersistentVolume).