Cloud provider — VMware vSphere: provider configuration

This feature is available in Enterprise Edition only.

If the cluster control plane is hosted on a virtual machines or bare-metal servers, the cloud provider uses the settings from the cloud-provider-vsphere module in the Deckhouse configuration. Otherwise, if the cluster control plane is hosted in a cloud, the cloud provider uses the VsphereClusterConfiguration structure for configuration.

Additional info about Vsphere Cloud Load Balancers.

VsphereClusterConfiguration

Version: deckhouse.io/v1

Describes the configuration of a cloud cluster in vSphere.

Used by the cloud provider if a cluster’s control plane is hosted in the cloud.

Run the following command to change the configuration in a running cluster:

kubectl -n d8-system exec -ti deploy/deckhouse -- deckhouse-controller edit provider-cluster-configuration

Example:

apiVersion: deckhouse.io/v1
kind: VsphereClusterConfiguration
sshPublicKey: "<SSH_PUBLIC_KEY>"
layout: Standard
vmFolderPath: folder/prefix
regionTagCategory: k8s-region
zoneTagCategory: k8s-zone
region: region2
zones:
- region2-a
externalNetworkNames:
- net3-k8s
internalNetworkNames:
- K8S_3
internalNetworkCIDR: 172.16.2.0/24
baseResourcePool: kubernetes/cloud
masterNodeGroup:
  replicas: 1
  instanceClass:
    numCPUs: 4
    memory: 8192
    template: Templates/ubuntu-focal-20.04
    mainNetwork: net3-k8s
    additionalNetworks:
    - K8S_3
    datastore: lun10
    rootDiskSize: 20
    runtimeOptions:
      nestedHardwareVirtualization: false
nodeGroups:
- name: worker
  replicas: 1
  zones:
  - ru-central1-a
  instanceClass:
    numCPUs: 4
    memory: 8192
    template: Templates/ubuntu-focal-20.04
    datastore: lun10
    mainNetwork: net3-k8s
provider:
  server: "<SERVER>"
  username: "<USERNAME>"
  password: "<PASSWORD>"
  insecure: true

apiVersionstring
Required value
Allowed values: deckhouse.io/v1, deckhouse.io/v1alpha1
baseResourcePoolstring
A path (relative to vSphere Cluster) to the existing parent resourcePool for all resourcePool created in each zone.
disableTimesyncboolean
Disable time synchronization on the vSphere side.

Caution! This parameter will not disable the NTP daemons in the guest OS, but only disable the time correction on the part of ESXi.
externalNetworkNamesarray of strings
Names of networks (just the name and not the full path) connected to VirtualMachines and used by vsphere-cloud-controller-manager to insert ExternalIP into the .status.addresses field in the Node API object.

Example:
```
externalNetworkNames:
- MAIN-1
- public
```
internalNetworkCIDRstring
Subnet for master nodes in the internal network.

Addresses are allocated starting with the tenth address. E.g., if you have the 192.168.199.0/24 subnet, addresses will be allocated starting with 192.168.199.10.

The internalNetworkCIDR is used if additionalNetworks are defined in masterInstanceClass.

Pattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(3[0-2]|[1-2][0-9]|[0-9]))$
internalNetworkNamesarray of strings
Names of networks (just the name and not the full path) connected to VirtualMachines and used by vsphere-cloud-controller-manager to insert InternalIP into the .status.addresses field in the Node API object.

Example:
```
internalNetworkNames:
- KUBE-3
- devops-internal
```
kindstring
Required value
Allowed values: VsphereClusterConfiguration
layoutstring
Required value
The way resources are located in the cloud.

Read more about possible provider layouts.
masterNodeGroupobject
Required value
The definition of the master’s NodeGroup.

Caution! After changing the parameters of the section, you need to run dhctl converge for the changes to take effect.
- masterNodeGroup.instanceClassobject
  Required value
  Partial contents of the fields of the VsphereInstanceClass.
  - masterNodeGroup.instanceClass.additionalNetworksarray of strings
    Paths to networks that VirtualMachines’ secondary NICs will connect to. Relative to the datacenter.
    
    Example:
    additionalNetworks: - DEVOPS_32 - DEVOPS_50
  - masterNodeGroup.instanceClass.datastorestring
    Required value
    Path to a Datastore in which VirtualMachines will be cloned. Relative to the datacenter.
    
    Example:
    datastore: lun-1201
  - masterNodeGroup.instanceClass.mainNetworkstring
    Required value
    Path to the network that VirtualMachines’ primary NICs will connect to (default gateway). Relative to the datacenter.
    
    Example:
    mainNetwork: k8s-msk-178
  - masterNodeGroup.instanceClass.mainNetworkIPAddressesarray of objects
    A list of static IP addresses (with a CIDR prefix) sequentially allocated to master nodes in the mainNetwork.
    
    By default, the DHCP client is enabled.
    
    Example:
    mainNetworkIPAddresses: - address: 10.1.14.20/24 gateway: 10.1.14.254 nameservers: addresses: - 8.8.8.8 - 8.8.4.4
    - masterNodeGroup.instanceClass.mainNetworkIPAddresses.addressstring
      Required value
      An IP address with a CIDR prefix.
      
      Pattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(3[0-2]|[1-2][0-9]|[0-9]))$
    - masterNodeGroup.instanceClass.mainNetworkIPAddresses.gatewaystring
      Required value
      The IP address of the default gateway.
      
      It must be located in the subnet specified in the address parameter
      
      Pattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$
    - masterNodeGroup.instanceClass.mainNetworkIPAddresses.nameserversobject
      Required value
      - masterNodeGroup.instanceClass.mainNetworkIPAddresses.nameservers.addressesarray of strings
        A list of DNS servers.
        
        Example:
        addresses: - 8.8.8.8 - 8.8.4.4
        
        Element of the arraystring
        Pattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$
      - masterNodeGroup.instanceClass.mainNetworkIPAddresses.nameservers.searcharray of strings
        A list of DNS search domains.
        
        Example:
        search: - tech.lan
  - masterNodeGroup.instanceClass.memoryinteger
    Required value
    Memory in MiB to allocate to vSphere VirtualMachines.
    
    Example:
    memory: 8192
  - masterNodeGroup.instanceClass.numCPUsinteger
    Required value
    Count of vCPUs to allocate to vSphere VirtualMachines.
    
    Example:
    numCPUs: 2
  - masterNodeGroup.instanceClass.resourcePoolstring
    Path to a Resource Pool in which VirtualMachines will be cloned. Relative to the zone (vSphere Cluster).
    
    Example:
    resourcePool: rp-2012
  - masterNodeGroup.instanceClass.rootDiskSizeinteger
    Root disk size in GiB to use in vSphere VirtualMachines.
    
    The disk will be automatically enlarged if its size in the template is less than specified.
    
    Example:
    rootDiskSize: 20
  - masterNodeGroup.instanceClass.runtimeOptionsobject
    Additional VM’s parameters.
    - masterNodeGroup.instanceClass.runtimeOptions.cpuLimitinteger
      CPU limit in MHz.
    - masterNodeGroup.instanceClass.runtimeOptions.cpuReservationinteger
      CPU reservation in MHz.
    - masterNodeGroup.instanceClass.runtimeOptions.cpuSharesinteger
      The relative amount of CPU Shares for VMs to be created.
    - masterNodeGroup.instanceClass.runtimeOptions.memoryLimitinteger
      Memory limit in MB.
    - masterNodeGroup.instanceClass.runtimeOptions.memoryReservationinteger
      VM memory reservation in percent (relative to .spec.memory).
      
      Allowed values: 0 <= X <= 100
    - masterNodeGroup.instanceClass.runtimeOptions.memorySharesinteger
      The relative amount of Memory Shares for VMs to be created.
      
      Allowed values: 0 <= X <= 100
    - masterNodeGroup.instanceClass.runtimeOptions.nestedHardwareVirtualizationboolean
      Whether to enable or disable nested hardware virtualization.
  - masterNodeGroup.instanceClass.templatestring
    Required value
    Path to the template to be cloned. Relative to the datacenter.
    
    Example:
    template: dev/golden_image
- masterNodeGroup.replicasinteger
  Required value
  The number of master nodes to create. It is important to have an odd number of masters to ensure a quorum.
  
  Allowed values: 1 <= X
- masterNodeGroup.zonesarray of strings
  A limited set of zones in which nodes can be created.
nodeGroupsarray of objects
An array of additional NodeGroups for creating static nodes (e.g., for dedicated front nodes or gateways).
- nodeGroups.instanceClassobject
  Required value
  Partial contents of the fields of the VsphereInstanceClass.
  - nodeGroups.instanceClass.additionalNetworksarray of strings
    Paths to networks that VirtualMachines’ secondary NICs will connect to. Relative to the datacenter.
    
    Example:
    additionalNetworks: - DEVOPS_32 - DEVOPS_50
  - nodeGroups.instanceClass.datastorestring
    Required value
    Path to a Datastore in which VirtualMachines will be cloned. Relative to the datacenter.
    
    Example:
    datastore: lun-1201
  - nodeGroups.instanceClass.mainNetworkstring
    Required value
    Path to the network that VirtualMachines’ primary NICs will connect to (default gateway). Relative to the datacenter.
    
    Example:
    mainNetwork: k8s-msk-178
  - nodeGroups.instanceClass.mainNetworkIPAddressesarray of objects
    A list of static IP addresses (with a CIDR prefix) sequentially allocated to master nodes in the mainNetwork.
    
    By default, the DHCP client is enabled.
    
    Example:
    mainNetworkIPAddresses: - address: 10.1.14.20/24 gateway: 10.1.14.254 nameservers: addresses: - 8.8.8.8 - 8.8.4.4
    - nodeGroups.instanceClass.mainNetworkIPAddresses.addressstring
      Required value
      An IP address with a CIDR prefix.
      
      Pattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(3[0-2]|[1-2][0-9]|[0-9]))$
    - nodeGroups.instanceClass.mainNetworkIPAddresses.gatewaystring
      Required value
      The IP address of the default gateway.
      
      It must be located in the subnet specified in the address parameter
      
      Pattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$
    - nodeGroups.instanceClass.mainNetworkIPAddresses.nameserversobject
      Required value
      - nodeGroups.instanceClass.mainNetworkIPAddresses.nameservers.addressesarray of strings
        A list of DNS servers.
        
        Example:
        addresses: - 8.8.8.8 - 8.8.4.4
        
        Element of the arraystring
        Pattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$
      - nodeGroups.instanceClass.mainNetworkIPAddresses.nameservers.searcharray of strings
        A list of DNS search domains.
        
        Example:
        search: - tech.lan
  - nodeGroups.instanceClass.memoryinteger
    Required value
    Memory in MiB to allocate to vSphere VirtualMachines.
    
    Example:
    memory: 8192
  - nodeGroups.instanceClass.numCPUsinteger
    Required value
    Count of vCPUs to allocate to vSphere VirtualMachines.
    
    Example:
    numCPUs: 2
  - nodeGroups.instanceClass.resourcePoolstring
    Path to a Resource Pool in which VirtualMachines will be cloned. Relative to the zone (vSphere Cluster).
    
    Example:
    resourcePool: rp-2012
  - nodeGroups.instanceClass.rootDiskSizeinteger
    Root disk size in GiB to use in vSphere VirtualMachines.
    
    The disk will be automatically enlarged if its size in the template is less than specified.
    
    Example:
    rootDiskSize: 20
  - nodeGroups.instanceClass.runtimeOptionsobject
    Additional VM’s parameters.
    - nodeGroups.instanceClass.runtimeOptions.cpuLimitinteger
      CPU limit in MHz.
    - nodeGroups.instanceClass.runtimeOptions.cpuReservationinteger
      CPU reservation in MHz.
    - nodeGroups.instanceClass.runtimeOptions.cpuSharesinteger
      The relative amount of CPU Shares for VMs to be created.
    - nodeGroups.instanceClass.runtimeOptions.memoryLimitinteger
      Memory limit in MB.
    - nodeGroups.instanceClass.runtimeOptions.memoryReservationinteger
      VM memory reservation in percent (relative to .spec.memory).
      
      Allowed values: 0 <= X <= 100
    - nodeGroups.instanceClass.runtimeOptions.memorySharesinteger
      The relative amount of Memory Shares for VMs to be created.
      
      Allowed values: 0 <= X <= 100
    - nodeGroups.instanceClass.runtimeOptions.nestedHardwareVirtualizationboolean
      Whether to enable or disable nested hardware virtualization.
  - nodeGroups.instanceClass.templatestring
    Required value
    Path to the template to be cloned. Relative to the datacenter.
    
    Example:
    template: dev/golden_image
- nodeGroups.namestring
  Required value
  The name of the NodeGroup to use for generating node names.
- nodeGroups.nodeTemplate
  Parameters of Node objects in Kubernetes to add after registering the node.
  - nodeGroups.nodeTemplate.annotationsobject
    The same as the metadata.annotations standard field.
    
    Example:
    annotations: ai.fleet.com/discombobulate: 'true'
  - nodeGroups.nodeTemplate.labelsobject
    A list of labels to attach to cluster resources.
    
    The same as the metadata.labels standard field.
    
    Example:
    labels: environment: production app: warp-drive-ai
  - nodeGroups.nodeTemplate.taintsarray of objects
    The same as the .spec.taints field of the Node object.
    
    Caution! Only the effect, key, values fields are available.
    
    Example:
    taints: - effect: NoExecute key: ship-class value: frigate
    - nodeGroups.nodeTemplate.taints.effectstring
      Allowed values: NoSchedule, PreferNoSchedule, NoExecute
    - nodeGroups.nodeTemplate.taints.keystring
    - nodeGroups.nodeTemplate.taints.valuestring
- nodeGroups.replicasinteger
  Required value
  The number of nodes to create.
- nodeGroups.zonesarray of strings
  A limited set of zones in which nodes can be created.
nsxtobject
Kubernetes load balancer support using NSX-T for the vSphere cloud controller manager.
- nsxt.defaultIpPoolNamestring
  Required value
  Name of the default IP pool used for the SVC’s without loadbalancer.vmware.io/class annotation set.
  
  Example:
```
defaultIpPoolName: pool1
```
- nsxt.defaultTcpAppProfileNamestring
  Name of default NSX-T application profile used for TCP connections.
  
  Default: "default-tcp-lb-app-profile"
  Examples:
```
defaultTcpAppProfileName: default-tcp-lb-app-profile
```
```
defaultTcpAppProfileName: tcp-profile1
```
- nsxt.defaultUdpAppProfileNamestring
  Name of default NSX-T application profile used for UDP connections.
  
  Default: "default-udp-lb-app-profile"
  Examples:
```
defaultUdpAppProfileName: default-udp-lb-app-profile
```
```
defaultUdpAppProfileName: udp-profile1
```
- nsxt.hoststring
  Required value
  NSX-T host.
  
  Example:
```
host: 1.2.3.4
```
- nsxt.insecureFlagboolean
  To be set to true if NSX-T uses self-signed certificate.
  
  Examples:
```
insecureFlag: true
```
```
insecureFlag: false
```
- nsxt.loadBalancerClassarray
  Additional section to define Load Balancer Classes (to use class, set annotation loadbalancer.vmware.io/class: <class name> to SVC).
  
  Examples:
```
loadBalancerClass: []
```
```
loadBalancerClass:
  name: LBC1
  ipPoolName: pool2
```
```
loadBalancerClass:
  name: LBC1
  ipPoolName: pool2
  tcpAppProfileName: profile2
  udpAppProfileName: profile3
```
  - nsxt.loadBalancerClass.ipPoolNamestring
    Required value
    Name of the IP pool.
  - nsxt.loadBalancerClass.namestring
    Required value
    Load Balancer Class name to use in SVC annotation loadbalancer.vmware.io/class: <class name>.
  - nsxt.loadBalancerClass.tcpAppProfileNamestring
    Name of application profile used for TCP connections.
    
    Default: "defaultTcpAppProfileName"
  - nsxt.loadBalancerClass.udpAppProfileNamestring
    Name of application profile used for UDP connections.
    
    Default: "defaultUdpAppProfileName"
- nsxt.passwordstring
  Required value
  NSX-T password.
  
  Example:
```
password: password
```
- nsxt.sizestring
  Size of load balancer service.
  
  Default: "MEDIUM"
  Allowed values: SMALL, MEDIUM, LARGE, XLARGE
  Example:
```
size: SMALL
```
- nsxt.tier1GatewayPathstring
  Required value
  Policy path for the NSX-T tier1 gateway.
  
  Example:
```
tier1GatewayPath: "/path/tier1"
```
- nsxt.userstring
  Required value
  NSX-T user name.
  
  Example:
```
user: user
```
providerobject
Required value
Parameters for connecting to the vCenter.
- provider.insecureboolean
  Set to true if vCenter has a self-signed certificate.
  
  Default: false
- provider.passwordstring
  Required value
  The user’s password.
- provider.serverstring
  Required value
  The host or the IP address of the vCenter server.
- provider.usernamestring
  Required value
  The login ID.
regionstring
Required value
Is a tag added to the vSphere Datacenter where all actions will occur: provisioning VirtualMachines, storing virtual disks on datastores, connecting to the network.
regionTagCategorystring
Required value
The name of the tag category used to identify the region (vSphere Datacenter).

Default: "k8s-region"
sshPublicKeystring
Required value
A public key for accessing nodes.
useNestedResourcePoolboolean
Create nested resource pool (true) or use main resource pool (false).

Default: true
vmFolderPathstring
Required value
The path to the VirtualMachine Folder where the cloned VMs will be created.

Example:
```
vmFolderPath: dev/test
```
zoneTagCategorystring
Required value
The name of the tag category used to identify the zone (vSphere Cluster).

Default: "k8s-zone"
zonesarray of strings
Required value
The globally restricted set of zones that this Cloud Provider works with.