This feature is available in Enterprise Edition only.

List of required vSphere resources

  • User with required set of permissions.
  • Network with DHCP server and access to the Internet
  • Datacenter with a tag in k8s-region category.
  • Cluster with a tag in k8s-zone category.
  • Datastore with required tags.
  • Templateprepared VM image.

vSphere configuration

Installing govc

You’ll need the vSphere CLI — govc — to proceed with the rest of the guide.

After the installation is complete, set the environment variables required to work with vCenter:

export GOVC_URL=example.com
export GOVC_USERNAME=<username>@vsphere.local
export GOVC_PASSWORD=<password>
export GOVC_INSECURE=1

Creating tags and tag categories

Instead of “regions” and “zones”, VMware vSphere provides Datacenter and Cluster objects. We will use tags to match them with “regions”/”zones”. These tags fall into two categories: one for “regions” tags and the other for “zones” tags.

Create a tag category using the following commands:

govc tags.category.create -d "Kubernetes Region" k8s-region
govc tags.category.create -d "Kubernetes Zone" k8s-zone

Create tags in each category. If you intend to use multiple “zones” (Cluster), create a tag for each one of them:

govc tags.create -d "Kubernetes Region" -c k8s-region test-region
govc tags.create -d "Kubernetes Zone Test 1" -c k8s-zone test-zone-1
govc tags.create -d "Kubernetes Zone Test 2" -c k8s-zone test-zone-2

Attach the “region” tag to Datacenter:

govc tags.attach -c k8s-region test-region /<DatacenterName>

Attach “zone” tags to the Cluster objects:

govc tags.attach -c k8s-zone test-zone-1 /<DatacenterName>/host/<ClusterName1>
govc tags.attach -c k8s-zone test-zone-2 /<DatacenterName>/host/<ClusterName2>

Datastore configuration

For dynamic PersistentVolume provisioning, a Datastore must be available on each ESXi host (shared datastore).

Assign the “region” and “zone” tags to the Datastore objects to automatically create a StorageClass in the Kubernetes cluster:

govc tags.attach -c k8s-region test-region /<DatacenterName>/datastore/<DatastoreName1>
govc tags.attach -c k8s-zone test-zone-1 /<DatacenterName>/datastore/<DatastoreName1>

govc tags.attach -c k8s-region test-region /<DatacenterName>/datastore/<DatastoreName1>
govc tags.attach -c k8s-zone test-zone-2 /<DatacenterName>/datastore/<DatastoreName2>

Creating and assigning a role

We’ve intentionally skipped User creation since there are many ways to authenticate a user in the vSphere.

This all-encompassing Role should be enough for all Deckhouse components. For a detailed list of privileges, refer to the documentation. If you need a more granular Role, please contact your Deckhouse support.

Create a role with the corresponding permissions:

govc role.create deckhouse \
   Cns.Searchable Datastore.AllocateSpace Datastore.Browse Datastore.FileManagement \
   Global.GlobalTag Global.SystemTag Network.Assign StorageProfile.View \
   $(govc role.ls Admin | grep -F -e 'Folder.' -e 'InventoryService.' -e 'Resource.' -e 'VirtualMachine.')

Assign the role to a user on the vCenter object:

govc permissions.set -principal <username>@vsphere.local -role deckhouse /

Preparing a virtual machine image

It is recommended to use a pre-built cloud image/OVA file provided by the OS vendor to create a Template:

Virtual machine image requirements

Deckhouse uses cloud-init to configure a virtual machine after startup. To do this, the following packages must be installed in the image:

To add SSH keys to user’s authorized keys, the default_user parameter must be specified in the /etc/cloud/cloud.cfg file.

Infrastructure

Networking

A VLAN with DHCP and Internet access is required for the running cluster:

  • If the VLAN is public (public addresses), then you have to create a second network to deploy cluster nodes (DHCP is not needed in this network).
  • If the VLAN is private (private addresses), then this network can be used for cluster nodes.

Inbound traffic

  • You can use an internal load balancer (if present) and direct traffic directly to the front nodes of the cluster.
  • If there is no load balancer, you can use MetalLB in BGP mode to organize fault-tolerant load balancers (recommended). In this case, front nodes of the cluster will have two interfaces. For this, you will need:
    • A dedicated VLAN for traffic exchange between BGP routers and MetalLB. This VLAN must have DHCP and Internet access.
    • IP addresses of BGP routers.
    • ASN — the AS number on the BGP router.
    • ASN — the AS number in the cluster.
    • A range to announce addresses from.

Using the data store

Various types of storage can be used in the cluster; for the minimum configuration, you will need:

  • Datastore for provisioning PersistentVolumes to the Kubernetes cluster.
  • Datastore for provisioning root disks for the VMs (it can be the same Datastore as for PersistentVolume).