If the cluster control plane is hosted on a virtual machines or bare-metal servers, the cloud provider uses the settings from the
cloud-provider-vsphere
module in the Deckhouse configuration. Otherwise, if the cluster control plane is hosted in a cloud, the cloud provider uses the VsphereClusterConfiguration structure for configuration.Additional info about Vsphere Cloud Load Balancers.
VsphereClusterConfiguration
Version: deckhouse.io/v1
Describes the configuration of a cloud cluster in vSphere.
Used by the cloud provider if a cluster’s control plane is hosted in the cloud.
Run the following command to change the configuration in a running cluster:
kubectl -n d8-system exec -ti deploy/deckhouse -- deckhouse-controller edit provider-cluster-configuration
Example:
apiVersion: deckhouse.io/v1
kind: VsphereClusterConfiguration
sshPublicKey: "<SSH_PUBLIC_KEY>"
layout: Standard
vmFolderPath: folder/prefix
regionTagCategory: k8s-region
zoneTagCategory: k8s-zone
region: region2
zones:
- region2-a
externalNetworkNames:
- net3-k8s
internalNetworkNames:
- K8S_3
internalNetworkCIDR: 172.16.2.0/24
baseResourcePool: kubernetes/cloud
masterNodeGroup:
replicas: 1
instanceClass:
numCPUs: 4
memory: 8192
template: Templates/ubuntu-focal-20.04
mainNetwork: net3-k8s
additionalNetworks:
- K8S_3
datastore: lun10
rootDiskSize: 20
runtimeOptions:
nestedHardwareVirtualization: false
nodeGroups:
- name: worker
replicas: 1
zones:
- ru-central1-a
instanceClass:
numCPUs: 4
memory: 8192
template: Templates/ubuntu-focal-20.04
datastore: lun10
mainNetwork: net3-k8s
provider:
server: "<SERVER>"
username: "<USERNAME>"
password: "<PASSWORD>"
insecure: true
- apiVersion
Required value
Allowed values:
deckhouse.io/v1
,deckhouse.io/v1alpha1
- baseResourcePool
A path (relative to vSphere Cluster) to the existing parent
resourcePool
for allresourcePool
created in each zone. - disableTimesync
Disable time synchronization on the vSphere side.
Caution! This parameter will not disable the NTP daemons in the guest OS, but only disable the time correction on the part of ESXi.
Default:
true
- externalNetworkNames
Names of networks (just the name and not the full path) connected to
VirtualMachines
and used byvsphere-cloud-controller-manager
to insert ExternalIP into the.status.addresses
field in the Node API object.Example:
externalNetworkNames: - MAIN-1 - public
- internalNetworkCIDR
Subnet for master nodes in the internal network.
Addresses are allocated starting with the tenth address. E.g., if you have the
192.168.199.0/24
subnet, addresses will be allocated starting with192.168.199.10
.The
internalNetworkCIDR
is used ifadditionalNetworks
are defined inmasterInstanceClass
.Pattern:
^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(3[0-2]|[1-2][0-9]|[0-9]))$
- internalNetworkNames
Names of networks (just the name and not the full path) connected to
VirtualMachines
and used byvsphere-cloud-controller-manager
to insert InternalIP into the.status.addresses
field in the Node API object.Example:
internalNetworkNames: - KUBE-3 - devops-internal
- kind
Required value
Allowed values:
VsphereClusterConfiguration
- layout
Required value
The way resources are located in the cloud.
Read more about possible provider layouts.
- masterNodeGroup
Required value
The definition of the master’s NodeGroup.
Caution! After changing the parameters of the section, you need to run
dhctl converge
for the changes to take effect.- masterNodeGroup.instanceClass
Required value
Partial contents of the fields of the VsphereInstanceClass.
- masterNodeGroup.instanceClass.additionalNetworks
Paths to networks that VirtualMachines’ secondary NICs will connect to. Relative to the datacenter.
Example:
additionalNetworks: - DEVOPS_32 - DEVOPS_50
- masterNodeGroup.instanceClass.datastore
Required value
Path to a Datastore in which VirtualMachines will be cloned. Relative to the datacenter.
Example:
datastore: lun-1201
- masterNodeGroup.instanceClass.mainNetwork
Required value
Path to the network that VirtualMachines’ primary NICs will connect to (default gateway). Relative to the datacenter.
Example:
mainNetwork: k8s-msk-178
- masterNodeGroup.instanceClass.mainNetworkIPAddresses
A list of static IP addresses (with a CIDR prefix) sequentially allocated to nodes in the
mainNetwork
.By default, the DHCP client is enabled.
Example:
mainNetworkIPAddresses: - address: 10.1.14.20/24 gateway: 10.1.14.254 nameservers: addresses: - 8.8.8.8 - 8.8.4.4
- masterNodeGroup.instanceClass.mainNetworkIPAddresses.address
Required value
An IP address with a CIDR prefix.
Pattern:
^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(3[0-2]|[1-2][0-9]|[0-9]))$
- masterNodeGroup.instanceClass.mainNetworkIPAddresses.gateway
Required value
The IP address of the default gateway.
It must be located in the subnet specified in the
address
parameterPattern:
^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$
- masterNodeGroup.instanceClass.mainNetworkIPAddresses.nameservers
Required value
- masterNodeGroup.instanceClass.mainNetworkIPAddresses.nameservers.addresses
A list of DNS servers.
Example:
addresses: - 8.8.8.8 - 8.8.4.4
- Element of the array
Pattern:
^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$
- masterNodeGroup.instanceClass.mainNetworkIPAddresses.nameservers.search
A list of DNS search domains.
Example:
search: - tech.lan
- masterNodeGroup.instanceClass.memory
Required value
Memory in MiB to allocate to vSphere VirtualMachines.
Example:
memory: 8192
- masterNodeGroup.instanceClass.numCPUs
Required value
Count of vCPUs to allocate to vSphere VirtualMachines.
Example:
numCPUs: 2
- masterNodeGroup.instanceClass.resourcePool
Path to a Resource Pool in which VirtualMachines will be cloned. Relative to the zone (vSphere Cluster).
Example:
resourcePool: rp-2012
- masterNodeGroup.instanceClass.rootDiskSize
Root disk size in GiB to use in vSphere VirtualMachines.
The disk will be automatically enlarged if its size in the template is less than specified.
Example:
rootDiskSize: 20
- masterNodeGroup.instanceClass.runtimeOptions
Additional VM’s parameters.
- masterNodeGroup.instanceClass.runtimeOptions.cpuLimit
CPU limit in MHz.
- masterNodeGroup.instanceClass.runtimeOptions.cpuReservation
CPU reservation in MHz.
- integer
The relative amount of CPU Shares for VMs to be created.
- masterNodeGroup.instanceClass.runtimeOptions.memoryLimit
Memory limit in MB.
- masterNodeGroup.instanceClass.runtimeOptions.memoryReservation
VM memory reservation in percent (relative to
.spec.memory
).Allowed values:
0 <= X <= 100
- integer
The relative amount of Memory Shares for VMs to be created.
Allowed values:
0 <= X <= 100
- masterNodeGroup.instanceClass.runtimeOptions.nestedHardwareVirtualization
Whether to enable or disable nested hardware virtualization.
- masterNodeGroup.instanceClass.template
Required value
Path to the template to be cloned. Relative to the datacenter.
Example:
template: dev/golden_image
- masterNodeGroup.replicas
Required value
The number of master nodes to create. It is important to have an odd number of masters to ensure a quorum.
Allowed values:
1 <= X
- masterNodeGroup.zones
A limited set of zones in which nodes can be created.
- nodeGroups
An array of additional NodeGroups for creating static nodes (e.g., for dedicated front nodes or gateways).
- nodeGroups.instanceClass
Required value
Partial contents of the fields of the VsphereInstanceClass.
- nodeGroups.instanceClass.additionalNetworks
Paths to networks that VirtualMachines’ secondary NICs will connect to. Relative to the datacenter.
Example:
additionalNetworks: - DEVOPS_32 - DEVOPS_50
- nodeGroups.instanceClass.datastore
Required value
Path to a Datastore in which VirtualMachines will be cloned. Relative to the datacenter.
Example:
datastore: lun-1201
- nodeGroups.instanceClass.mainNetwork
Required value
Path to the network that VirtualMachines’ primary NICs will connect to (default gateway). Relative to the datacenter.
Example:
mainNetwork: k8s-msk-178
- nodeGroups.instanceClass.mainNetworkIPAddresses
A list of static IP addresses (with a CIDR prefix) sequentially allocated to nodes in the
mainNetwork
.By default, the DHCP client is enabled.
Example:
mainNetworkIPAddresses: - address: 10.1.14.20/24 gateway: 10.1.14.254 nameservers: addresses: - 8.8.8.8 - 8.8.4.4
- nodeGroups.instanceClass.mainNetworkIPAddresses.address
Required value
An IP address with a CIDR prefix.
Pattern:
^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(3[0-2]|[1-2][0-9]|[0-9]))$
- nodeGroups.instanceClass.mainNetworkIPAddresses.gateway
Required value
The IP address of the default gateway.
It must be located in the subnet specified in the
address
parameterPattern:
^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$
- nodeGroups.instanceClass.mainNetworkIPAddresses.nameservers
Required value
- nodeGroups.instanceClass.mainNetworkIPAddresses.nameservers.addresses
A list of DNS servers.
Example:
addresses: - 8.8.8.8 - 8.8.4.4
- Element of the array
Pattern:
^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$
- nodeGroups.instanceClass.mainNetworkIPAddresses.nameservers.search
A list of DNS search domains.
Example:
search: - tech.lan
- nodeGroups.instanceClass.memory
Required value
Memory in MiB to allocate to vSphere VirtualMachines.
Example:
memory: 8192
- nodeGroups.instanceClass.numCPUs
Required value
Count of vCPUs to allocate to vSphere VirtualMachines.
Example:
numCPUs: 2
- nodeGroups.instanceClass.resourcePool
Path to a Resource Pool in which VirtualMachines will be cloned. Relative to the zone (vSphere Cluster).
Example:
resourcePool: rp-2012
- nodeGroups.instanceClass.rootDiskSize
Root disk size in GiB to use in vSphere VirtualMachines.
The disk will be automatically enlarged if its size in the template is less than specified.
Example:
rootDiskSize: 20
- nodeGroups.instanceClass.runtimeOptions
Additional VM’s parameters.
- nodeGroups.instanceClass.runtimeOptions.cpuLimit
CPU limit in MHz.
- nodeGroups.instanceClass.runtimeOptions.cpuReservation
CPU reservation in MHz.
- integer
The relative amount of CPU Shares for VMs to be created.
- nodeGroups.instanceClass.runtimeOptions.memoryLimit
Memory limit in MB.
- nodeGroups.instanceClass.runtimeOptions.memoryReservation
VM memory reservation in percent (relative to
.spec.memory
).Allowed values:
0 <= X <= 100
- integer
The relative amount of Memory Shares for VMs to be created.
Allowed values:
0 <= X <= 100
- nodeGroups.instanceClass.runtimeOptions.nestedHardwareVirtualization
Whether to enable or disable nested hardware virtualization.
- nodeGroups.instanceClass.template
Required value
Path to the template to be cloned. Relative to the datacenter.
Example:
template: dev/golden_image
- nodeGroups.name
Required value
The name of the NodeGroup to use for generating node names.
- nodeGroups.nodeTemplate
Parameters of Node objects in Kubernetes to add after registering the node.
- nodeGroups.nodeTemplate.annotations
The same as the
metadata.annotations
standard field.Example:
annotations: ai.fleet.com/discombobulate: 'true'
- nodeGroups.nodeTemplate.labels
A list of labels to attach to cluster resources.
The same as the
metadata.labels
standard field.Example:
labels: environment: production app: warp-drive-ai
- nodeGroups.nodeTemplate.taints
The same as the
.spec.taints
field of the Node object.Caution! Only the
effect
,key
,values
fields are available.Example:
taints: - effect: NoExecute key: ship-class value: frigate
- nodeGroups.nodeTemplate.taints.effect
Allowed values:
NoSchedule
,PreferNoSchedule
,NoExecute
- nodeGroups.nodeTemplate.taints.key
- nodeGroups.nodeTemplate.taints.value
- nodeGroups.replicas
Required value
The number of nodes to create.
- nodeGroups.zones
A limited set of zones in which nodes can be created.
- nsxt
Kubernetes load balancer support using NSX-T for the vSphere cloud controller manager.
- nsxt.defaultIpPoolName
Required value
Name of the default IP pool used for the SVC’s without
loadbalancer.vmware.io/class
annotation set.Example:
defaultIpPoolName: pool1
- nsxt.defaultTcpAppProfileName
Name of default NSX-T application profile used for TCP connections.
Default:
"default-tcp-lb-app-profile"
Examples:
defaultTcpAppProfileName: default-tcp-lb-app-profile
defaultTcpAppProfileName: tcp-profile1
- nsxt.defaultUdpAppProfileName
Name of default NSX-T application profile used for UDP connections.
Default:
"default-udp-lb-app-profile"
Examples:
defaultUdpAppProfileName: default-udp-lb-app-profile
defaultUdpAppProfileName: udp-profile1
- nsxt.host
Required value
NSX-T host.
Example:
host: 1.2.3.4
- nsxt.insecureFlag
To be set to true if NSX-T uses self-signed certificate.
Examples:
insecureFlag: true
insecureFlag: false
- nsxt.loadBalancerClass
Additional section to define Load Balancer Classes (set annotation
loadbalancer.vmware.io/class: <CLASS NAME>
to SVC to use the class).Examples:
loadBalancerClass: []
loadBalancerClass: name: LBC1 ipPoolName: pool2
loadBalancerClass: name: LBC1 ipPoolName: pool2 tcpAppProfileName: profile2 udpAppProfileName: profile3
- nsxt.loadBalancerClass.ipPoolName
Required value
Name of the IP pool.
- nsxt.loadBalancerClass.name
Required value
Load Balancer Class name to use in SVC annotation
loadbalancer.vmware.io/class: <CLASS NAME>
. - nsxt.loadBalancerClass.tcpAppProfileName
Name of application profile used for TCP connections.
Default:
"defaultTcpAppProfileName"
- nsxt.loadBalancerClass.udpAppProfileName
Name of application profile used for UDP connections.
Default:
"defaultUdpAppProfileName"
- nsxt.password
Required value
NSX-T password.
Example:
password: password
- nsxt.size
Size of load balancer service.
Default:
"MEDIUM"
Allowed values:
SMALL
,MEDIUM
,LARGE
,XLARGE
Example:
size: SMALL
- nsxt.tier1GatewayPath
Required value
Policy path for the NSX-T tier1 gateway.
Example:
tier1GatewayPath: "/path/tier1"
- nsxt.user
Required value
NSX-T user name.
Example:
user: user
- provider
Required value
Parameters for connecting to the vCenter.
- provider.insecure
Set to
true
if vCenter has a self-signed certificate.Default:
false
- provider.password
Required value
The user’s password.
- provider.server
Required value
The host or the IP address of the vCenter server.
- provider.username
Required value
The login ID.
- region
Required value
Is a tag added to the vSphere Datacenter where all actions will occur: provisioning VirtualMachines, storing virtual disks on datastores, connecting to the network.
- regionTagCategory
Required value
The name of the tag category used to identify the region (vSphere Datacenter).
Default:
"k8s-region"
- sshPublicKey
Required value
A public key for accessing nodes.
- useNestedResourcePool
Create nested resource pool (
true
) or use main resource pool (false
).Default:
true
- vmFolderExists
Set the value to
true
if the path specified invmFolderPath
exists. Installing more than one cluster in a folder is not possible.Default:
false
Example:
vmFolderExists: dev/test
- vmFolderPath
Required value
The path to the VirtualMachine Folder where the cloned VMs will be created.
Example:
vmFolderPath: dev/test
- zoneTagCategory
Required value
The name of the tag category used to identify the zone (vSphere Cluster).
Default:
"k8s-zone"
- zones
Required value
The globally restricted set of zones that this Cloud Provider works with.