If the cluster control plane is hosted on a virtual machines or bare-metal servers, the cloud provider uses the settings from the
cloud-provider-vsphere
module in the Deckhouse configuration. Otherwise, if the cluster control plane is hosted in a cloud, the cloud provider uses the VsphereClusterConfiguration structure for configuration.Additional info about Vsphere Cloud Load Balancers.
VsphereClusterConfiguration
Version: deckhouse.io/v1
Describes the configuration of a cloud cluster in vSphere.
Used by the cloud provider if a cluster’s control plane is hosted in the cloud.
Run the following command to change the configuration in a running cluster:
kubectl -n d8-system exec -ti deploy/deckhouse -- deckhouse-controller edit provider-cluster-configuration
Example:
apiVersion: deckhouse.io/v1
kind: VsphereClusterConfiguration
sshPublicKey: "<SSH_PUBLIC_KEY>"
layout: Standard
vmFolderPath: folder/prefix
regionTagCategory: k8s-region
zoneTagCategory: k8s-zone
region: region2
zones:
- region2-a
externalNetworkNames:
- net3-k8s
internalNetworkNames:
- K8S_3
internalNetworkCIDR: 172.16.2.0/24
baseResourcePool: kubernetes/cloud
masterNodeGroup:
replicas: 1
instanceClass:
numCPUs: 4
memory: 8192
template: Templates/ubuntu-focal-20.04
mainNetwork: net3-k8s
additionalNetworks:
- K8S_3
datastore: lun10
rootDiskSize: 20
runtimeOptions:
nestedHardwareVirtualization: false
nodeGroups:
- name: worker
replicas: 1
zones:
- ru-central1-a
instanceClass:
numCPUs: 4
memory: 8192
template: Templates/ubuntu-focal-20.04
datastore: lun10
mainNetwork: net3-k8s
provider:
server: "<SERVER>"
username: "<USERNAME>"
password: "<PASSWORD>"
insecure: true
- apiVersionstring
Required value
Allowed values:
deckhouse.io/v1
,deckhouse.io/v1alpha1
- baseResourcePoolstring
A path (relative to vSphere Cluster) to the existing parent
resourcePool
for allresourcePool
created in each zone. - disableTimesyncboolean
Disable time synchronization on the vSphere side.
Caution! This parameter will not disable the NTP daemons in the guest OS, but only disable the time correction on the part of ESXi.
Default:
true
- externalNetworkNamesarray of strings
Names of networks (just the name and not the full path) connected to
VirtualMachines
and used byvsphere-cloud-controller-manager
to insert ExternalIP into the.status.addresses
field in the Node API object.Example:
externalNetworkNames: - MAIN-1 - public
- internalNetworkCIDRstring
Subnet for master nodes in the internal network.
Addresses are allocated starting with the tenth address. E.g., if you have the
192.168.199.0/24
subnet, addresses will be allocated starting with192.168.199.10
.The
internalNetworkCIDR
is used ifadditionalNetworks
are defined inmasterInstanceClass
.Pattern:
^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(3[0-2]|[1-2][0-9]|[0-9]))$
- internalNetworkNamesarray of strings
Names of networks (just the name and not the full path) connected to
VirtualMachines
and used byvsphere-cloud-controller-manager
to insert InternalIP into the.status.addresses
field in the Node API object.Example:
internalNetworkNames: - KUBE-3 - devops-internal
- kindstring
Required value
Allowed values:
VsphereClusterConfiguration
- layoutstring
Required value
The way resources are located in the cloud.
Read more about possible provider layouts.
- masterNodeGroupobject
Required value
The definition of the master’s NodeGroup.
Caution! After changing the parameters of the section, you need to run
dhctl converge
for the changes to take effect.- masterNodeGroup.instanceClassobject
Required value
Partial contents of the fields of the VsphereInstanceClass.
- masterNodeGroup.instanceClass.additionalNetworksarray of strings
Paths to networks that VirtualMachines’ secondary NICs will connect to. Relative to the datacenter.
Example:
additionalNetworks: - DEVOPS_32 - DEVOPS_50
- masterNodeGroup.instanceClass.datastorestring
Required value
Path to a Datastore in which VirtualMachines will be cloned. Relative to the datacenter.
Example:
datastore: lun-1201
- masterNodeGroup.instanceClass.mainNetworkstring
Required value
Path to the network that VirtualMachines’ primary NICs will connect to (default gateway). Relative to the datacenter.
Example:
mainNetwork: k8s-msk-178
- masterNodeGroup.instanceClass.mainNetworkIPAddressesarray of objects
A list of static IP addresses (with a CIDR prefix) sequentially allocated to master nodes in the
mainNetwork
.By default, the DHCP client is enabled.
Example:
mainNetworkIPAddresses: - address: 10.1.14.20/24 gateway: 10.1.14.254 nameservers: addresses: - 8.8.8.8 - 8.8.4.4
- masterNodeGroup.instanceClass.mainNetworkIPAddresses.addressstring
Required value
An IP address with a CIDR prefix.
Pattern:
^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(3[0-2]|[1-2][0-9]|[0-9]))$
- masterNodeGroup.instanceClass.mainNetworkIPAddresses.gatewaystring
Required value
The IP address of the default gateway.
It must be located in the subnet specified in the
address
parameterPattern:
^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$
- masterNodeGroup.instanceClass.mainNetworkIPAddresses.nameserversobject
Required value
- masterNodeGroup.instanceClass.mainNetworkIPAddresses.nameservers.addressesarray of strings
A list of DNS servers.
Example:
addresses: - 8.8.8.8 - 8.8.4.4
- Element of the arraystring
Pattern:
^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$
- Element of the arraystring
- masterNodeGroup.instanceClass.mainNetworkIPAddresses.nameservers.searcharray of strings
A list of DNS search domains.
Example:
search: - tech.lan
- masterNodeGroup.instanceClass.mainNetworkIPAddresses.nameservers.addressesarray of strings
- masterNodeGroup.instanceClass.mainNetworkIPAddresses.addressstring
- masterNodeGroup.instanceClass.memoryinteger
Required value
Memory in MiB to allocate to vSphere VirtualMachines.
Example:
memory: 8192
- masterNodeGroup.instanceClass.numCPUsinteger
Required value
Count of vCPUs to allocate to vSphere VirtualMachines.
Example:
numCPUs: 2
- masterNodeGroup.instanceClass.resourcePoolstring
Path to a Resource Pool in which VirtualMachines will be cloned. Relative to the zone (vSphere Cluster).
Example:
resourcePool: rp-2012
- masterNodeGroup.instanceClass.rootDiskSizeinteger
Root disk size in GiB to use in vSphere VirtualMachines.
The disk will be automatically enlarged if its size in the template is less than specified.
Example:
rootDiskSize: 20
- masterNodeGroup.instanceClass.runtimeOptionsobject
Additional VM’s parameters.
- masterNodeGroup.instanceClass.runtimeOptions.cpuLimitinteger
CPU limit in MHz.
- masterNodeGroup.instanceClass.runtimeOptions.cpuReservationinteger
CPU reservation in MHz.
- integer
The relative amount of CPU Shares for VMs to be created.
- masterNodeGroup.instanceClass.runtimeOptions.memoryLimitinteger
Memory limit in MB.
- masterNodeGroup.instanceClass.runtimeOptions.memoryReservationinteger
VM memory reservation in percent (relative to
.spec.memory
).Allowed values:
0 <= X <= 100
- integer
The relative amount of Memory Shares for VMs to be created.
Allowed values:
0 <= X <= 100
- masterNodeGroup.instanceClass.runtimeOptions.nestedHardwareVirtualizationboolean
Whether to enable or disable nested hardware virtualization.
- masterNodeGroup.instanceClass.runtimeOptions.cpuLimitinteger
- masterNodeGroup.instanceClass.templatestring
Required value
Path to the template to be cloned. Relative to the datacenter.
Example:
template: dev/golden_image
- masterNodeGroup.instanceClass.additionalNetworksarray of strings
- masterNodeGroup.replicasinteger
Required value
The number of master nodes to create. It is important to have an odd number of masters to ensure a quorum.
Allowed values:
1 <= X
- masterNodeGroup.zonesarray of strings
A limited set of zones in which nodes can be created.
- masterNodeGroup.instanceClassobject
- nodeGroupsarray of objects
An array of additional NodeGroups for creating static nodes (e.g., for dedicated front nodes or gateways).
- nodeGroups.instanceClassobject
Required value
Partial contents of the fields of the VsphereInstanceClass.
- nodeGroups.instanceClass.additionalNetworksarray of strings
Paths to networks that VirtualMachines’ secondary NICs will connect to. Relative to the datacenter.
Example:
additionalNetworks: - DEVOPS_32 - DEVOPS_50
- nodeGroups.instanceClass.datastorestring
Required value
Path to a Datastore in which VirtualMachines will be cloned. Relative to the datacenter.
Example:
datastore: lun-1201
- nodeGroups.instanceClass.mainNetworkstring
Required value
Path to the network that VirtualMachines’ primary NICs will connect to (default gateway). Relative to the datacenter.
Example:
mainNetwork: k8s-msk-178
- nodeGroups.instanceClass.mainNetworkIPAddressesarray of objects
A list of static IP addresses (with a CIDR prefix) sequentially allocated to master nodes in the
mainNetwork
.By default, the DHCP client is enabled.
Example:
mainNetworkIPAddresses: - address: 10.1.14.20/24 gateway: 10.1.14.254 nameservers: addresses: - 8.8.8.8 - 8.8.4.4
- nodeGroups.instanceClass.mainNetworkIPAddresses.addressstring
Required value
An IP address with a CIDR prefix.
Pattern:
^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(3[0-2]|[1-2][0-9]|[0-9]))$
- nodeGroups.instanceClass.mainNetworkIPAddresses.gatewaystring
Required value
The IP address of the default gateway.
It must be located in the subnet specified in the
address
parameterPattern:
^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$
- nodeGroups.instanceClass.mainNetworkIPAddresses.nameserversobject
Required value
- nodeGroups.instanceClass.mainNetworkIPAddresses.nameservers.addressesarray of strings
A list of DNS servers.
Example:
addresses: - 8.8.8.8 - 8.8.4.4
- Element of the arraystring
Pattern:
^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$
- Element of the arraystring
- nodeGroups.instanceClass.mainNetworkIPAddresses.nameservers.searcharray of strings
A list of DNS search domains.
Example:
search: - tech.lan
- nodeGroups.instanceClass.mainNetworkIPAddresses.nameservers.addressesarray of strings
- nodeGroups.instanceClass.mainNetworkIPAddresses.addressstring
- nodeGroups.instanceClass.memoryinteger
Required value
Memory in MiB to allocate to vSphere VirtualMachines.
Example:
memory: 8192
- nodeGroups.instanceClass.numCPUsinteger
Required value
Count of vCPUs to allocate to vSphere VirtualMachines.
Example:
numCPUs: 2
- nodeGroups.instanceClass.resourcePoolstring
Path to a Resource Pool in which VirtualMachines will be cloned. Relative to the zone (vSphere Cluster).
Example:
resourcePool: rp-2012
- nodeGroups.instanceClass.rootDiskSizeinteger
Root disk size in GiB to use in vSphere VirtualMachines.
The disk will be automatically enlarged if its size in the template is less than specified.
Example:
rootDiskSize: 20
- nodeGroups.instanceClass.runtimeOptionsobject
Additional VM’s parameters.
- nodeGroups.instanceClass.runtimeOptions.cpuLimitinteger
CPU limit in MHz.
- nodeGroups.instanceClass.runtimeOptions.cpuReservationinteger
CPU reservation in MHz.
- integer
The relative amount of CPU Shares for VMs to be created.
- nodeGroups.instanceClass.runtimeOptions.memoryLimitinteger
Memory limit in MB.
- nodeGroups.instanceClass.runtimeOptions.memoryReservationinteger
VM memory reservation in percent (relative to
.spec.memory
).Allowed values:
0 <= X <= 100
- integer
The relative amount of Memory Shares for VMs to be created.
Allowed values:
0 <= X <= 100
- nodeGroups.instanceClass.runtimeOptions.nestedHardwareVirtualizationboolean
Whether to enable or disable nested hardware virtualization.
- nodeGroups.instanceClass.runtimeOptions.cpuLimitinteger
- nodeGroups.instanceClass.templatestring
Required value
Path to the template to be cloned. Relative to the datacenter.
Example:
template: dev/golden_image
- nodeGroups.instanceClass.additionalNetworksarray of strings
- nodeGroups.namestring
Required value
The name of the NodeGroup to use for generating node names.
- nodeGroups.nodeTemplate
Parameters of Node objects in Kubernetes to add after registering the node.
- nodeGroups.nodeTemplate.annotationsobject
The same as the
metadata.annotations
standard field.Example:
annotations: ai.fleet.com/discombobulate: 'true'
- nodeGroups.nodeTemplate.labelsobject
A list of labels to attach to cluster resources.
The same as the
metadata.labels
standard field.Example:
labels: environment: production app: warp-drive-ai
- nodeGroups.nodeTemplate.taintsarray of objects
The same as the
.spec.taints
field of the Node object.Caution! Only the
effect
,key
,values
fields are available.Example:
taints: - effect: NoExecute key: ship-class value: frigate
- nodeGroups.nodeTemplate.taints.effectstring
Allowed values:
NoSchedule
,PreferNoSchedule
,NoExecute
- nodeGroups.nodeTemplate.taints.keystring
- nodeGroups.nodeTemplate.taints.valuestring
- nodeGroups.nodeTemplate.taints.effectstring
- nodeGroups.nodeTemplate.annotationsobject
- nodeGroups.replicasinteger
Required value
The number of nodes to create.
- nodeGroups.zonesarray of strings
A limited set of zones in which nodes can be created.
- nodeGroups.instanceClassobject
- nsxtobject
Kubernetes load balancer support using NSX-T for the vSphere cloud controller manager.
- nsxt.defaultIpPoolNamestring
Required value
Name of the default IP pool used for the SVC’s without
loadbalancer.vmware.io/class
annotation set.Example:
defaultIpPoolName: pool1
- nsxt.defaultTcpAppProfileNamestring
Name of default NSX-T application profile used for TCP connections.
Default:
"default-tcp-lb-app-profile"
Examples:
defaultTcpAppProfileName: default-tcp-lb-app-profile
defaultTcpAppProfileName: tcp-profile1
- nsxt.defaultUdpAppProfileNamestring
Name of default NSX-T application profile used for UDP connections.
Default:
"default-udp-lb-app-profile"
Examples:
defaultUdpAppProfileName: default-udp-lb-app-profile
defaultUdpAppProfileName: udp-profile1
- nsxt.hoststring
Required value
NSX-T host.
Example:
host: 1.2.3.4
- nsxt.insecureFlagboolean
To be set to true if NSX-T uses self-signed certificate.
Examples:
insecureFlag: true
insecureFlag: false
- nsxt.loadBalancerClassarray
Additional section to define Load Balancer Classes (to use class, set annotation
loadbalancer.vmware.io/class: <class name>
to SVC).Examples:
loadBalancerClass: []
loadBalancerClass: name: LBC1 ipPoolName: pool2
loadBalancerClass: name: LBC1 ipPoolName: pool2 tcpAppProfileName: profile2 udpAppProfileName: profile3
- nsxt.loadBalancerClass.ipPoolNamestring
Required value
Name of the IP pool.
- nsxt.loadBalancerClass.namestring
Required value
Load Balancer Class name to use in SVC annotation
loadbalancer.vmware.io/class: <class name>
. - nsxt.loadBalancerClass.tcpAppProfileNamestring
Name of application profile used for TCP connections.
Default:
"defaultTcpAppProfileName"
- nsxt.loadBalancerClass.udpAppProfileNamestring
Name of application profile used for UDP connections.
Default:
"defaultUdpAppProfileName"
- nsxt.loadBalancerClass.ipPoolNamestring
- nsxt.passwordstring
Required value
NSX-T password.
Example:
password: password
- nsxt.sizestring
Size of load balancer service.
Default:
"MEDIUM"
Allowed values:
SMALL
,MEDIUM
,LARGE
,XLARGE
Example:
size: SMALL
- nsxt.tier1GatewayPathstring
Required value
Policy path for the NSX-T tier1 gateway.
Example:
tier1GatewayPath: "/path/tier1"
- nsxt.userstring
Required value
NSX-T user name.
Example:
user: user
- nsxt.defaultIpPoolNamestring
- providerobject
Required value
Parameters for connecting to the vCenter.
- provider.insecureboolean
Set to
true
if vCenter has a self-signed certificate.Default:
false
- provider.passwordstring
Required value
The user’s password.
- provider.serverstring
Required value
The host or the IP address of the vCenter server.
- provider.usernamestring
Required value
The login ID.
- provider.insecureboolean
- regionstring
Required value
Is a tag added to the vSphere Datacenter where all actions will occur: provisioning VirtualMachines, storing virtual disks on datastores, connecting to the network.
- regionTagCategorystring
Required value
The name of the tag category used to identify the region (vSphere Datacenter).
Default:
"k8s-region"
- sshPublicKeystring
Required value
A public key for accessing nodes.
- useNestedResourcePoolboolean
Create nested resource pool (
true
) or use main resource pool (false
).Default:
true
- vmFolderPathstring
Required value
The path to the VirtualMachine Folder where the cloned VMs will be created.
Example:
vmFolderPath: dev/test
- zoneTagCategorystring
Required value
The name of the tag category used to identify the zone (vSphere Cluster).
Default:
"k8s-zone"
- zonesarray of strings
Required value
The globally restricted set of zones that this Cloud Provider works with.