If the cluster control plane is hosted on a virtual machines or bare-metal servers, the cloud provider uses the settings from the cloud-provider-vsphere module in the Deckhouse configuration. Otherwise, if the cluster control plane is hosted in a cloud, the cloud provider uses the VsphereClusterConfiguration structure for configuration.

Additional info about Vsphere Cloud Load Balancers.

VsphereClusterConfiguration

Version: deckhouse.io/v1

Describes the configuration of a cloud cluster in vSphere.

Used by the cloud provider if a cluster’s control plane is hosted in the cloud.

Run the following command to change the configuration in a running cluster:

kubectl -n d8-system exec -ti svc/deckhouse-leader -c deckhouse -- deckhouse-controller edit provider-cluster-configuration

After updating the node parameters, you need to run the dhctl converge command to apply the changes.

Example:

apiVersion: deckhouse.io/v1
kind: VsphereClusterConfiguration
sshPublicKey: "<SSH_PUBLIC_KEY>"
layout: Standard
vmFolderPath: folder/prefix
regionTagCategory: k8s-region
zoneTagCategory: k8s-zone
region: region2
zones:
- region2-a
externalNetworkNames:
- net3-k8s
internalNetworkNames:
- K8S_3
internalNetworkCIDR: 172.16.2.0/24
baseResourcePool: kubernetes/cloud
masterNodeGroup:
  replicas: 1
  instanceClass:
    numCPUs: 4
    memory: 8192
    template: Templates/ubuntu-focal-20.04
    mainNetwork: net3-k8s
    additionalNetworks:
    - K8S_3
    datastore: lun10
    rootDiskSize: 50
    runtimeOptions:
      nestedHardwareVirtualization: false
nodeGroups:
- name: worker
  replicas: 1
  zones:
  - ru-central1-a
  instanceClass:
    numCPUs: 4
    memory: 8192
    template: Templates/ubuntu-focal-20.04
    datastore: lun10
    mainNetwork: net3-k8s
provider:
  server: "<SERVER>"
  username: "<USERNAME>"
  password: "<PASSWORD>"
  insecure: true
  • apiVersion
    string

    Required value

    Allowed values: deckhouse.io/v1, deckhouse.io/v1alpha1

  • baseResourcePool
    string

    A path (relative to vSphere Cluster) to the existing parent resourcePool for all resourcePool created in each zone.

  • disableTimesync
    boolean

    Disable time synchronization on the vSphere side.

    Caution! This parameter will not disable the NTP daemons in the guest OS, but only disable the time correction on the part of ESXi.

    Default: true

  • externalNetworkNames
    array of strings

    Names of networks (just the name and not the full path) connected to VirtualMachines and used by vsphere-cloud-controller-manager to insert ExternalIP into the .status.addresses field in the Node API object.

    Example:

    externalNetworkNames:
- MAIN-1
- public
  • internalNetworkCIDR
    string

    Subnet for master nodes in the internal network.

    Addresses are allocated starting with the tenth address. E.g., if you have the 192.168.199.0/24 subnet, addresses will be allocated starting with 192.168.199.10.

    The internalNetworkCIDR is used if additionalNetworks are defined in masterInstanceClass.

    Pattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(3[0-2]|[1-2][0-9]|[0-9]))$

  • internalNetworkNames
    array of strings

    Names of networks (just the name and not the full path) connected to VirtualMachines and used by vsphere-cloud-controller-manager to insert InternalIP into the .status.addresses field in the Node API object.

    Example:

    internalNetworkNames:
- KUBE-3
- devops-internal
  • kind
    string

    Required value

    Allowed values: VsphereClusterConfiguration

  • layout
    string

    Required value

    The way resources are located in the cloud.

    Read more about possible provider layouts.

  • masterNodeGroup
    object

    Required value

    The definition of the master’s NodeGroup.

    For the changes to take effect, run dhctl converge after modifying the parameters of the masterNodeGroup section.

    • masterNodeGroup.instanceClass
      object

      Required value

      Partial contents of the fields of the VsphereInstanceClass.

      • masterNodeGroup.instanceClass.additionalNetworks
        array of strings

        Paths to networks that VirtualMachines’ secondary NICs will connect to. Relative to the datacenter.

        Example:

        additionalNetworks:
- DEVOPS_32
- DEVOPS_50
      • masterNodeGroup.instanceClass.datastore
        string

        Required value

        Path to a Datastore in which VirtualMachines will be cloned. Relative to the datacenter.

        Example:

        datastore: lun-1201
      • masterNodeGroup.instanceClass.mainNetwork
        string

        Required value

        Path to the network that VirtualMachines’ primary NICs will connect to (default gateway). Relative to the datacenter.

        Example:

        mainNetwork: k8s-msk-178
      • masterNodeGroup.instanceClass.mainNetworkIPAddresses
        array of objects

        A list of static IP addresses (with a CIDR prefix) sequentially allocated to nodes in the mainNetwork.

        By default, the DHCP client is enabled.

        Example:

        mainNetworkIPAddresses:
- address: 10.1.14.20/24
  gateway: 10.1.14.254
  nameservers:
    addresses:
    - 8.8.8.8
    - 8.8.4.4
        • masterNodeGroup.instanceClass.mainNetworkIPAddresses.address
          string

          Required value

          An IP address with a CIDR prefix.

          Pattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(3[0-2]|[1-2][0-9]|[0-9]))$

        • masterNodeGroup.instanceClass.mainNetworkIPAddresses.gateway
          string

          Required value

          The IP address of the default gateway.

          It must be located in the subnet specified in the address parameter

          Pattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$

        • masterNodeGroup.instanceClass.mainNetworkIPAddresses.nameservers
          object

          Required value

          • masterNodeGroup.instanceClass.mainNetworkIPAddresses.nameservers.addresses
            array of strings

            A list of DNS servers.

            Example:

            addresses:
- 8.8.8.8
- 8.8.4.4
            • Element of the array
              string

              Pattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$

          • masterNodeGroup.instanceClass.mainNetworkIPAddresses.nameservers.search
            array of strings

            A list of DNS search domains.

            Example:

            search:
- tech.lan
      • masterNodeGroup.instanceClass.memory
        integer

        Required value

        Memory in MiB to allocate to vSphere VirtualMachines.

        Example:

        memory: 8192
      • masterNodeGroup.instanceClass.numCPUs
        integer

        Required value

        Count of vCPUs to allocate to vSphere VirtualMachines.

        Example:

        numCPUs: 2
      • masterNodeGroup.instanceClass.resourcePool
        string

        Path to a Resource Pool in which VirtualMachines will be cloned. Relative to the zone (vSphere Cluster).

        Example:

        resourcePool: rp-2012
      • masterNodeGroup.instanceClass.rootDiskSize
        integer

        Root disk size in GiB to use in vSphere VirtualMachines.

        The disk will be automatically enlarged if its size in the template is less than specified.

        Example:

        rootDiskSize: 50
      • masterNodeGroup.instanceClass.runtimeOptions
        object

        Additional VM’s parameters.

        • masterNodeGroup.instanceClass.runtimeOptions.cpuLimit
          integer

          CPU limit in MHz.

        • masterNodeGroup.instanceClass.runtimeOptions.cpuReservation
          integer

          CPU reservation in MHz.

        • masterNodeGroup.instanceClass.runtimeOptions.cpuShares
          integer

          The relative amount of CPU Shares for VMs to be created.

        • masterNodeGroup.instanceClass.runtimeOptions.memoryLimit
          integer

          Memory limit in MB.

        • masterNodeGroup.instanceClass.runtimeOptions.memoryReservation
          integer

          VM memory reservation in percent (relative to .spec.memory).

          Allowed values: 0 <= X <= 100

        • masterNodeGroup.instanceClass.runtimeOptions.memoryShares
          integer

          The relative amount of Memory Shares for VMs to be created.

          Allowed values: 0 <= X <= 100

        • masterNodeGroup.instanceClass.runtimeOptions.nestedHardwareVirtualization
          boolean

          Whether to enable or disable nested hardware virtualization.

      • masterNodeGroup.instanceClass.template
        string

        Required value

        Path to the template to be cloned. Relative to the datacenter.

        Example:

        template: dev/golden_image
    • masterNodeGroup.replicas
      integer

      Required value

      The number of master nodes to create. It is important to have an odd number of masters to ensure a quorum.

      Allowed values: 1 <= X

    • masterNodeGroup.zones
      array of strings

      A limited set of zones in which master nodes can be created.

      The order in which zones are specified does not influence the sequence of node creation. Nodes are assigned to zones in alphabetical order: the first node is placed in the alphabetically first zone, the second node in the second zone, and so forth. If there are more nodes than zones, the assignment process restarts from the beginning of the zone list after placing a node in the last alphabetical zone.

  • nodeGroups
    array of objects

    An array of additional NodeGroups for creating static nodes (e.g., for dedicated front nodes or gateways).

    • nodeGroups.instanceClass
      object

      Required value

      Partial contents of the fields of the VsphereInstanceClass.

      • nodeGroups.instanceClass.additionalNetworks
        array of strings

        Paths to networks that VirtualMachines’ secondary NICs will connect to. Relative to the datacenter.

        Example:

        additionalNetworks:
- DEVOPS_32
- DEVOPS_50
      • nodeGroups.instanceClass.datastore
        string

        Required value

        Path to a Datastore in which VirtualMachines will be cloned. Relative to the datacenter.

        Example:

        datastore: lun-1201
      • nodeGroups.instanceClass.mainNetwork
        string

        Required value

        Path to the network that VirtualMachines’ primary NICs will connect to (default gateway). Relative to the datacenter.

        Example:

        mainNetwork: k8s-msk-178
      • nodeGroups.instanceClass.mainNetworkIPAddresses
        array of objects

        A list of static IP addresses (with a CIDR prefix) sequentially allocated to nodes in the mainNetwork.

        By default, the DHCP client is enabled.

        Example:

        mainNetworkIPAddresses:
- address: 10.1.14.20/24
  gateway: 10.1.14.254
  nameservers:
    addresses:
    - 8.8.8.8
    - 8.8.4.4
        • nodeGroups.instanceClass.mainNetworkIPAddresses.address
          string

          Required value

          An IP address with a CIDR prefix.

          Pattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(3[0-2]|[1-2][0-9]|[0-9]))$

        • nodeGroups.instanceClass.mainNetworkIPAddresses.gateway
          string

          Required value

          The IP address of the default gateway.

          It must be located in the subnet specified in the address parameter

          Pattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$

        • nodeGroups.instanceClass.mainNetworkIPAddresses.nameservers
          object

          Required value

          • nodeGroups.instanceClass.mainNetworkIPAddresses.nameservers.addresses
            array of strings

            A list of DNS servers.

            Example:

            addresses:
- 8.8.8.8
- 8.8.4.4
            • Element of the array
              string

              Pattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])$

          • nodeGroups.instanceClass.mainNetworkIPAddresses.nameservers.search
            array of strings

            A list of DNS search domains.

            Example:

            search:
- tech.lan
      • nodeGroups.instanceClass.memory
        integer

        Required value

        Memory in MiB to allocate to vSphere VirtualMachines.

        Example:

        memory: 8192
      • nodeGroups.instanceClass.numCPUs
        integer

        Required value

        Count of vCPUs to allocate to vSphere VirtualMachines.

        Example:

        numCPUs: 2
      • nodeGroups.instanceClass.resourcePool
        string

        Path to a Resource Pool in which VirtualMachines will be cloned. Relative to the zone (vSphere Cluster).

        Example:

        resourcePool: rp-2012
      • nodeGroups.instanceClass.rootDiskSize
        integer

        Root disk size in GiB to use in vSphere VirtualMachines.

        The disk will be automatically enlarged if its size in the template is less than specified.

        Example:

        rootDiskSize: 50
      • nodeGroups.instanceClass.runtimeOptions
        object

        Additional VM’s parameters.

        • nodeGroups.instanceClass.runtimeOptions.cpuLimit
          integer

          CPU limit in MHz.

        • nodeGroups.instanceClass.runtimeOptions.cpuReservation
          integer

          CPU reservation in MHz.

        • nodeGroups.instanceClass.runtimeOptions.cpuShares
          integer

          The relative amount of CPU Shares for VMs to be created.

        • nodeGroups.instanceClass.runtimeOptions.memoryLimit
          integer

          Memory limit in MB.

        • nodeGroups.instanceClass.runtimeOptions.memoryReservation
          integer

          VM memory reservation in percent (relative to .spec.memory).

          Allowed values: 0 <= X <= 100

        • nodeGroups.instanceClass.runtimeOptions.memoryShares
          integer

          The relative amount of Memory Shares for VMs to be created.

          Allowed values: 0 <= X <= 100

        • nodeGroups.instanceClass.runtimeOptions.nestedHardwareVirtualization
          boolean

          Whether to enable or disable nested hardware virtualization.

      • nodeGroups.instanceClass.template
        string

        Required value

        Path to the template to be cloned. Relative to the datacenter.

        Example:

        template: dev/golden_image
    • nodeGroups.name
      string

      Required value

      The name of the NodeGroup to use for generating node names.

    • nodeGroups.nodeTemplate

      Parameters of Node objects in Kubernetes to add after registering the node.

      • nodeGroups.nodeTemplate.annotations
        object

        The same as the metadata.annotations standard field.

        Example:

        annotations:
  ai.fleet.com/discombobulate: 'true'
      • nodeGroups.nodeTemplate.labels
        object

        A list of labels to attach to cluster resources.

        The same as the metadata.labels standard field.

        Example:

        labels:
  environment: production
  app: warp-drive-ai
      • nodeGroups.nodeTemplate.taints
        array of objects

        The same as the .spec.taints field of the Node object.

        Available fields: effect, key, and values.

        Example:

        taints:
- effect: NoExecute
  key: ship-class
  value: frigate
        • nodeGroups.nodeTemplate.taints.effect
          string

          Allowed values: NoSchedule, PreferNoSchedule, NoExecute

        • nodeGroups.nodeTemplate.taints.key
          string
        • nodeGroups.nodeTemplate.taints.value
          string
    • nodeGroups.replicas
      integer

      Required value

      The number of nodes to create.

    • nodeGroups.zones
      array of strings

      A limited set of zones in which nodes can be created.

  • nsxt
    object

    Kubernetes load balancer support using NSX-T for the vSphere cloud controller manager.

    • nsxt.defaultIpPoolName
      string

      Required value

      Name of the default IP pool used for the SVC’s without loadbalancer.vmware.io/class annotation set.

      Example:

      defaultIpPoolName: pool1
    • nsxt.defaultTcpAppProfileName
      string

      Name of default NSX-T application profile used for TCP connections.

      Default: "default-tcp-lb-app-profile"

      Examples:

      defaultTcpAppProfileName: default-tcp-lb-app-profile

      defaultTcpAppProfileName: tcp-profile1
    • nsxt.defaultUdpAppProfileName
      string

      Name of default NSX-T application profile used for UDP connections.

      Default: "default-udp-lb-app-profile"

      Examples:

      defaultUdpAppProfileName: default-udp-lb-app-profile

      defaultUdpAppProfileName: udp-profile1
    • nsxt.host
      string

      Required value

      NSX-T host.

      Example:

      host: 1.2.3.4
    • nsxt.insecureFlag
      boolean

      To be set to true if NSX-T uses self-signed certificate.

      Examples:

      insecureFlag: true

      insecureFlag: false
    • nsxt.loadBalancerClass
      array

      Additional section to define Load Balancer Classes (set annotation loadbalancer.vmware.io/class: <CLASS NAME> to SVC to use the class).

      Examples:

      loadBalancerClass: []

      loadBalancerClass:
  name: LBC1
  ipPoolName: pool2

      loadBalancerClass:
  name: LBC1
  ipPoolName: pool2
  tcpAppProfileName: profile2
  udpAppProfileName: profile3
      • nsxt.loadBalancerClass.ipPoolName
        string

        Required value

        Name of the IP pool.

      • nsxt.loadBalancerClass.name
        string

        Required value

        Load Balancer Class name to use in SVC annotation loadbalancer.vmware.io/class: <CLASS NAME>.

      • nsxt.loadBalancerClass.tcpAppProfileName
        string

        Name of application profile used for TCP connections.

        Default: "defaultTcpAppProfileName"

      • nsxt.loadBalancerClass.udpAppProfileName
        string

        Name of application profile used for UDP connections.

        Default: "defaultUdpAppProfileName"

    • nsxt.password
      string

      Required value

      NSX-T password.

      Example:

      password: password
    • nsxt.size
      string

      Size of load balancer service.

      Default: "MEDIUM"

      Allowed values: SMALL, MEDIUM, LARGE, XLARGE

      Example:

      size: SMALL
    • nsxt.tier1GatewayPath
      string

      Required value

      Policy path for the NSX-T tier1 gateway.

      Example:

      tier1GatewayPath: "/path/tier1"
    • nsxt.user
      string

      Required value

      NSX-T user name.

      Example:

      user: user
  • provider
    object

    Required value

    Parameters for connecting to the vCenter.

    • provider.insecure
      boolean

      Set to true if vCenter has a self-signed certificate.

      Default: false

    • provider.password
      string

      Required value

      The user’s password.

    • provider.server
      string

      Required value

      The host or the IP address of the vCenter server.

    • provider.username
      string

      Required value

      The login ID.

  • region
    string

    Required value

    Is a tag added to the vSphere Datacenter where all actions will occur: provisioning VirtualMachines, storing virtual disks on datastores, connecting to the network.

  • regionTagCategory
    string

    Required value

    The name of the tag category used to identify the region (vSphere Datacenter).

    Default: "k8s-region"

  • sshPublicKey
    string

    Required value

    A public key for accessing nodes.

  • useNestedResourcePool
    boolean

    Create nested resource pool (true) or use main resource pool (false).

    Default: true

  • vmFolderExists
    boolean

    Set the value to true if the path specified in vmFolderPath exists. Installing more than one cluster in a folder is not possible.

    Default: false

  • vmFolderPath
    string

    Required value

    The path to the VirtualMachine Folder where the cloned VMs will be created.

    Example:

    vmFolderPath: dev/test
  • zoneTagCategory
    string

    Required value

    The name of the tag category used to identify the zone (vSphere Cluster).

    Default: "k8s-zone"

  • zones
    array of strings

    Required value

    The globally restricted set of zones that this Cloud Provider works with.