The module is automatically enabled for all cloud clusters deployed in vSphere.
If the cluster control plane is hosted on a virtual machines or bare-metal servers, the cloud provider uses the settings from the cloud-provider-vsphere
module in the Deckhouse configuration (see below). Otherwise, if the cluster control plane is hosted in a cloud, the cloud provider uses the VsphereClusterConfiguration structure for configuration.
You can configure the number and parameters of ordering machines in the cloud via the NodeGroup
custom resource of the node-manager module. Also, in this custom resource, you can specify the instance class’s name for the above group of nodes (the cloudInstances.ClassReference
parameter of NodeGroup). In the case of the vSphere cloud provider, the instance class is the VsphereInstanceClass
custom resource that stores specific parameters of the machines.
The module is configured using the ModuleConfig custom resource named cloud-provider-vsphere
(learn more about setting up Deckhouse…).
Example of the ModuleConfig/cloud-provider-vsphere
resource for configuring the module:
apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
name: cloud-provider-vsphere
spec:
version: 1
enabled: true
settings: # <-- Module parameters from the "Parameters" section below.
Parameters
Schema version: 1
- disableTimesyncboolean
Disable time synchronization on the vSphere side.
Caution! This parameter will not disable the NTP daemons in the guest OS, but only disable the time correction on the part of ESXi.
- externalNetworkNamesarray of strings
Names of networks (just the name and not the full path) connected to
VirtualMachines
and used byvsphere-cloud-controller-manager
to insert ExternalIP into the.status.addresses
field in the Node API object. - hoststring
The domain of the vCenter server.
- insecureboolean
Set to
true
if vCenter has a self-signed certificate. - internalNetworkNamesarray of strings
Names of networks (just the name and not the full path) connected to
VirtualMachines
and used byvsphere-cloud-controller-manager
to insert InternalIP into the.status.addresses
field in the Node API object. - nsxtobject
Kubernetes load balancer support using NSX-T for the vSphere cloud controller manager.
- nsxt.defaultIpPoolNamestring
Required value
Name of the default IP pool used for the SVC’s without
loadbalancer.vmware.io/class
annotation set.Example:
defaultIpPoolName: pool1
- nsxt.defaultTcpAppProfileNamestring
Name of default NSX-T application profile used for TCP connections.
Default:
"default-tcp-lb-app-profile"
Examples:
defaultTcpAppProfileName: default-tcp-lb-app-profile
defaultTcpAppProfileName: tcp-profile1
- nsxt.defaultUdpAppProfileNamestring
Name of default NSX-T application profile used for UDP connections.
Default:
"default-udp-lb-app-profile"
Examples:
defaultUdpAppProfileName: default-udp-lb-app-profile
defaultUdpAppProfileName: udp-profile1
- nsxt.hoststring
Required value
NSX-T host.
Example:
host: 1.2.3.4
- nsxt.insecureFlagboolean
To be set to true if NSX-T uses self-signed certificate.
Examples:
insecureFlag: true
insecureFlag: false
- nsxt.loadBalancerClassarray
Additional section to define Load Balancer Classes (to use class, set annotation
loadbalancer.vmware.io/class: <class name>
to SVC).Examples:
loadBalancerClass: []
loadBalancerClass: name: LBC1 ipPoolName: pool2
loadBalancerClass: name: LBC1 ipPoolName: pool2 tcpAppProfileName: profile2 udpAppProfileName: profile3
- nsxt.loadBalancerClass.ipPoolNamestring
Required value
Name of the IP pool.
- nsxt.loadBalancerClass.namestring
Required value
Load Balancer Class name to use in SVC annotation
loadbalancer.vmware.io/class: <class name>
. - nsxt.loadBalancerClass.tcpAppProfileNamestring
Name of application profile used for TCP connections.
Default:
"defaultTcpAppProfileName"
- nsxt.loadBalancerClass.udpAppProfileNamestring
Name of application profile used for UDP connections.
Default:
"defaultUdpAppProfileName"
- nsxt.loadBalancerClass.ipPoolNamestring
- nsxt.passwordstring
Required value
NSX-T password.
Example:
password: password
- nsxt.sizestring
Size of load balancer service.
Default:
"MEDIUM"
Allowed values:
SMALL
,MEDIUM
,LARGE
,XLARGE
Example:
size: SMALL
- nsxt.tier1GatewayPathstring
Required value
Policy path for the NSX-T tier1 gateway.
Example:
tier1GatewayPath: "/path/tier1"
- nsxt.userstring
Required value
NSX-T user name.
Example:
user: user
- nsxt.defaultIpPoolNamestring
- passwordstring
The user’s password.
- regionstring
Is a tag added to the vSphere Datacenter where all actions will occur: provisioning VirtualMachines, storing virtual disks on datastores, connecting to the network.
- regionTagCategorystring
The name of the tag category used to identify the region (vSphere Datacenter).
- sshKeysarray of strings
A list of public SSH keys in plain-text format.
- storageClassobject
- storageClass.compatibilityFlagstring
A flag allowing the use of the old CSI version:
legacy
— use the old version of the driver. FCD discs only, no online-resizing;migration
— in this case, both drivers will be available in the cluster at the same time. This mode is used to migrate from an old driver.
Allowed values:
legacy
,migration
- storageClass.defaultstring
The name of StorageClass that will be used by default in the cluster.
If the parameter is omitted, the default StorageClass will be one of the following:
- An arbitrary StorageClass present in the cluster that has the default annotation.
- The first (in lexicographic order) StorageClass of those created by the module.
Example:
default: fast-lun102-7d0bf578
- storageClass.excludearray of strings
A list of StorageClass names (or regex expressions for names) to exclude from the creation in the cluster.
Example:
exclude: - ".*-lun101-.*" - slow-lun103-1c280603
- storageClass.compatibilityFlagstring
- usernamestring
The login ID.
- vmFolderPathstring
The path to the VirtualMachine Folder where the cloned VMs will be created.
- zoneTagCategorystring
The name of the tag category used to identify the region (vSphere Cluster).
- zonesarray of strings
The globally restricted set of zones that this Cloud Provider works with.
Storage
The module automatically creates a StorageClass for each Datastore and DatastoreCluster in the zone (or zones).
Also, it can set the name of StorageClass that will be used in the cluster by default (the default parameter), and filter out the unnecessary StorageClasses (the exclude parameter).
CSI
By default, the storage subsystem uses CNS volumes with the ability of online-resize. FCD volumes are also supported, but only in the legacy or migration modes. You can set this via the compatibilityFlag parameter.
Important information concerning the increase of the PVC size
Due to the nature f volume-resizer, CSI, and vSphere API, you have to do the following after increasing the PVC size:
- On the node where the Pod is located, run the
kubectl cordon <node_name>
command. - Delete the Pod.
- Make sure that the resize was successful. The PVC object must not have the
Resizing
condition.The
FileSystemResizePending
state is OK. - On the node where the Pod is located, run the
kubectl uncordon <node_name>
command.
Environment requirements
- vSphere version required:
v7.0U2
(required for theOnline volume expansion
work). - vCenter to which master nodes can connect to from within the cluster.
- Datacenter with the following components:
- VirtualMachine template with a specific cloud-init datasource.
- VM image should use
Virtual machines with hardware version 15 or later
(required for online resize to work).
- VM image should use
- The network must be available on all ESXi where VirtualMachines will be created.
- One or more Datastores connected to all ESXi where VirtualMachines will be created.
- A tag from the tag category in
zoneTagCategory
(k8s-zone
by default) must be added to Datastores. This tag will indicate the zone. All Clusters of a specific zone must have access to all Datastores within the same zone.
- A tag from the tag category in
- The cluster with the required ESXis.
- A tag from the tag category in
zoneTagCategory
(k8s-zone
by default) must be added to the Cluster. This tag will indicate the zone.
- A tag from the tag category in
- Folder for VirtualMachines to be created.
- An optional parameter. By default, the root vm folder is used.
- Create a role with the appropriate set of privileges.
- Create a user and assign the above role to it.
- VirtualMachine template with a specific cloud-init datasource.
- A tag from the tag category in
regionTagCategory
(k8s-region
by default) must be added to the Datacenter. This tag will indicate the region.
List of privileges for using the module
Datastore.AllocateSpace
Datastore.FileManagement
Global.GlobalTag
Global.SystemTag
InventoryService.Tagging.AttachTag
InventoryService.Tagging.CreateCategory
InventoryService.Tagging.CreateTag
InventoryService.Tagging.DeleteCategory
InventoryService.Tagging.DeleteTag
InventoryService.Tagging.EditCategory
InventoryService.Tagging.EditTag
InventoryService.Tagging.ModifyUsedByForCategory
InventoryService.Tagging.ModifyUsedByForTag
Network.Assign
Resource.AssignVMToPool
StorageProfile.View
System.Anonymous
System.Read
System.View
VirtualMachine.Config.AddExistingDisk
VirtualMachine.Config.AddNewDisk
VirtualMachine.Config.AddRemoveDevice
VirtualMachine.Config.AdvancedConfig
VirtualMachine.Config.Annotation
VirtualMachine.Config.CPUCount
VirtualMachine.Config.ChangeTracking
VirtualMachine.Config.DiskExtend
VirtualMachine.Config.DiskLease
VirtualMachine.Config.EditDevice
VirtualMachine.Config.HostUSBDevice
VirtualMachine.Config.ManagedBy
VirtualMachine.Config.Memory
VirtualMachine.Config.MksControl
VirtualMachine.Config.QueryFTCompatibility
VirtualMachine.Config.QueryUnownedFiles
VirtualMachine.Config.RawDevice
VirtualMachine.Config.ReloadFromPath
VirtualMachine.Config.RemoveDisk
VirtualMachine.Config.Rename
VirtualMachine.Config.ResetGuestInfo
VirtualMachine.Config.Resource
VirtualMachine.Config.Settings
VirtualMachine.Config.SwapPlacement
VirtualMachine.Config.ToggleForkParent
VirtualMachine.Config.UpgradeVirtualHardware
VirtualMachine.GuestOperations.Execute
VirtualMachine.GuestOperations.Modify
VirtualMachine.GuestOperations.ModifyAliases
VirtualMachine.GuestOperations.Query
VirtualMachine.GuestOperations.QueryAliases
VirtualMachine.Hbr.ConfigureReplication
VirtualMachine.Hbr.MonitorReplication
VirtualMachine.Hbr.ReplicaManagement
VirtualMachine.Interact.AnswerQuestion
VirtualMachine.Interact.Backup
VirtualMachine.Interact.ConsoleInteract
VirtualMachine.Interact.CreateScreenshot
VirtualMachine.Interact.CreateSecondary
VirtualMachine.Interact.DefragmentAllDisks
VirtualMachine.Interact.DeviceConnection
VirtualMachine.Interact.DisableSecondary
VirtualMachine.Interact.DnD
VirtualMachine.Interact.EnableSecondary
VirtualMachine.Interact.GuestControl
VirtualMachine.Interact.MakePrimary
VirtualMachine.Interact.Pause
VirtualMachine.Interact.PowerOff
VirtualMachine.Interact.PowerOn
VirtualMachine.Interact.PutUsbScanCodes
VirtualMachine.Interact.Record
VirtualMachine.Interact.Replay
VirtualMachine.Interact.Reset
VirtualMachine.Interact.SESparseMaintenance
VirtualMachine.Interact.SetCDMedia
VirtualMachine.Interact.SetFloppyMedia
VirtualMachine.Interact.Suspend
VirtualMachine.Interact.TerminateFaultTolerantVM
VirtualMachine.Interact.ToolsInstall
VirtualMachine.Interact.TurnOffFaultTolerance
VirtualMachine.Inventory.Create
VirtualMachine.Inventory.CreateFromExisting
VirtualMachine.Inventory.Delete
VirtualMachine.Inventory.Move
VirtualMachine.Inventory.Register
VirtualMachine.Inventory.Unregister
VirtualMachine.Namespace.Event
VirtualMachine.Namespace.EventNotify
VirtualMachine.Namespace.Management
VirtualMachine.Namespace.ModifyContent
VirtualMachine.Namespace.Query
VirtualMachine.Namespace.ReadContent
VirtualMachine.Provisioning.Clone
VirtualMachine.Provisioning.CloneTemplate
VirtualMachine.Provisioning.CreateTemplateFromVM
VirtualMachine.Provisioning.Customize
VirtualMachine.Provisioning.DeployTemplate
VirtualMachine.Provisioning.DiskRandomAccess
VirtualMachine.Provisioning.DiskRandomRead
VirtualMachine.Provisioning.FileRandomAccess
VirtualMachine.Provisioning.GetVmFiles
VirtualMachine.Provisioning.MarkAsTemplate
VirtualMachine.Provisioning.MarkAsVM
VirtualMachine.Provisioning.ModifyCustSpecs
VirtualMachine.Provisioning.PromoteDisks
VirtualMachine.Provisioning.PutVmFiles
VirtualMachine.Provisioning.ReadCustSpecs
VirtualMachine.State.CreateSnapshot
VirtualMachine.State.RemoveSnapshot
VirtualMachine.State.RenameSnapshot
VirtualMachine.State.RevertToSnapshot