Templating clusters

Deckhouse Commander is a web app that lets one create uniform clusters using UI, and control the lifecycle of those clusters.

Features

• Creation, updating and deleting clusters on major cloud platforms as well as on static resources
• Unification and actualization of clusters configuration using cluster templates
• The control of changes and the reconciliation to the desired configuration
• Cluster operation wia embedded admin console
• Catalogs of resources data utilized in clusters
• Integration API

Coming soon:

• Access control: users and permissions
• Cross-cluster projects
• Cloud resources overview used by cluster

Concepts

dhctl

To install the Deckhouse Kubernetes Platform manually, use the dhctl utility. It accepts three sets of data as input:

1. The cluster configuration and cluster installation in the form of a file, hereinafter referred to as config.yaml.
2. The SSH connection configuration to the machine that will become the first master node. The configuration is used as dhctl command line keys (it is possible to specify it in the form of YAML). Hereinafter referred to as SSHConfig.
3. An optional set of resources that need to be created at the last step of the installation, hereinafter referred to as resources.yaml.

What logical parts are contained in this data?

1. config.yaml
   1. InitConfiguration — cluster setup configuration
   2. ModuleConfig resources — configuration of built-in modules: explicit enabling or disabling, as well as default settings override
   3. ClusterConfiguration — Kubernetes configuration: version, pod subnets, services, etc.
   4. Deployment parameters
      1. <Provider>ClusterConfiguration — parameters of cluster deployment in the cloud or a Virtualization API;
      2. or StaticClusterConfiguration if Deckhouse Kubernetes Platform is being installed on static resources;
2. resources.yaml
   1. Arbitrary Kubernetes resources, including ModuleConfig for non-built-in modules in Deckhouse Kubernetes Platform
3. SSHConfig
   1. User name, password and key to connect to an existing machine or the one that will be created during cluster creation
   2. IP address of the machine, if the cluster is deployed on static resources and the address of the future master node is known in advance
   3. You can see the rest of the parameters in the help of the dhctl command, in this documentation, additional details are not essential.

For manual cluster management using dhctl, the above configuration types are needed in different combinations for cluster installation, modification, and removal.

As you can see, all the provided configuration is used to create a cluster. To do this, use dhctl bootstrap with all the provided configuration.

Changes that can be made to the cluster using the same tool relate to either the deployment parameters (for example, resources of permanent nodes created by Terraform) or the Kubernetes parameters. Only the connection settings and general cluster parameters are available for modification with dhctl converge. However, it is not possible to apply changes to the platform configuration or additional cluster resources in this way.

Finally, to delete a cluster, it is enough to have an access to it: the dhctl destroy operation uses only SSHConfig.

Commander

Commander makes use of the same dataset for configuring clusters as dhctl does, however Commander adds the ability to synchronize a complete desired configuration with the cluster. If we imagine Commander as an enhanced version of dhctl, then the table would look like this:

As you can see, Commander has a complete coverage of the Deckhouse Kubernetes Platform configuration for managing it after installation. Only InitConfiguration does not participate in modification, because this part of the configuration does not bring new information to the existing cluster.

Commander is the source of truth for the cluster configuration. Commander monitors that the cluster configuration matches the desired one. If Commander detects a discrepancy, it attempts to bring the cluster to the desired configuration. For this purpose, we will use the term “synchronization” further on.

In Commander, it is possible to specify the initial configuration of cluster resources, which will be applied during the cluster installation but will not be synchronized later on. This is useful when you need to create recommended or initial resources, but want to give control over them to the cluster operators.

Commander divides the Deckhouse Kubernetes Platform configuration based on the principle of traceability. Moreover, the user decides which part of the configuration should be synchronized and which should be set once when creating the cluster. Here is how this configuration looks from Commander’s perspective:

Note that the last two lines describe a configuration that will not be monitored or synchronized after the cluster is created.

Templates

The Idea

Commander is designed to manage typical clusters. Since all types of configuration in Commander are represented in YAML format, clustering templatization is a markup of the required YAML configuration with parameters and a description of the input parameters scheme of the template. To templatize YAML, the go template syntax and the sprig function set are used. A custom syntax for fields is used to describe the scheme of input parameters.

The cluster configuration is created by substituting the input parameters into the configuration templates. The input parameters are validated by the scheme specified for them.

Template versions

An important feature of a template is evolution. It is not enough to create a cluster fleet based on templates. Templates are improved and updated to meet the new software versions and new requirements for cluster operation. An updated template allows not only creating new clusters that meet modern requirements, but also updating existing clusters.

To evolve templates in Commander, a versioning mechanism is provided. When a template receives updates, a new version is created for it. The version can be accompanied by a comment. Based on the template version, you can create a cluster and test its performance. If the template version is unsuitable for use, it can be marked as unavailable for use. Then, cluster administrators will not be able to switch the cluster to the template version.

In Commander, each cluster is tied to a specific template version. However, technically, the cluster can be transferred to any other template and any available template version, subject to an invalid configuration that Commander will not allow to save. When the cluster is transferred to a new version or template, it is necessary to update the input parameters so that the updated configuration is created for the cluster. Commander will detect that the target configuration does not match the last applied configuration and create a task to synchronize the cluster.

Complexity of the template

Creating and testing a template is an engineering task, while creating clusters based on a template does not require a deep dive into technical details in general.

The input parameters of a template are presented to the user in the form of an online form, where the user enters or selects the parameters necessary to create a cluster. The entire set of input parameters is defined by the author of the template: which parameters are available, which are mandatory, in what order they are filled in, what test they are accompanied by, and how they are formatted for ease of perception by the end user.

Only the author of the template determines how easy or difficult it will be for the end user to use the template, and what decisions the user needs to make in order to successfully create a cluster. The more complex the template, the more complex the templating code and the more complex the form of the template parameters. Commander users themselves determine the ratio of complexity of the template and the number of templates for different scenarios. Commander is a flexible enough tool. With it, you can create both one template for all occasions and many templates for each individual use case.

Creating a template

You can add a template to Commander in two ways: by importing an existing one (for example, one created earlier in another installation of Commander) or by creating a new one from scratch. Ultimately, the templated configuration must comply with the features of dhctl and Deckhouse Kubernetes Platform of the version that will be installed using the template.

Where to find documentation for configuration types:

• Input parameters: usage.html#cluster-and-resource-parameter-scheme
• Kubernetes:
  • ClusterConfiguration
• Deployment:
  • Static resources
  • Yandex Cloud: cluster_configuration
  • VMware vSphere: cluster_configuration
  • OpenStack: cluster_configuration
  • Amazon Web Services: cluster_configuration
  • Google Cloud Platform: cluster_configuration
  • Microsoft Azure: cluster_configuration
• SSH parameters: see examples below
• Resources: arbitrary Kubernetes and Deckhouse resource manifests. Commander will synchronize these resources.
• Primary resources: arbitrary Kubernetes and Deckhouse resource manifests. Commander will not synchronize these resources.
• Installation: InitConfiguration

Special variables

There are several special variables in the cluster templates.

Required manifests

At the moment, Commander does not create invisible configuration, so the author of the template needs to take into account several manifests in the template to get a full experience using Commander. In the future, Commander will be improved to reduce the impact of technical features on the experience of working with it.

SSH parameters for a cloud cluster

For a cloud cluster, you can use the private key created by Commander if you do not provide a predefined key in the OS image. Also, in the virtual machine images, a user will be created under which Commander will connect to the created machine to start it up as a master node.

apiVersion: dhctl.deckhouse.io/v1
kind: SSHConfig
# The name of the user for SSH is defined in the "Hosting" section of the OS image
sshUser: ubuntu
sshPort: 22
# The private key that will be used to connect to VMs via SSH
sshAgentPrivateKeys:
  - key: |
      {{ .dc_sshPrivateKey | nindent 4 }}      

SSH and resources for static cluster

Since the machines are created in advance and SSH server, user and key are configured on them, these data must be provided in the input parameters of the cluster. Unlike the cloud configuration above, we use not an embedded parameter but one explicitly passed by the user. Some data can always be set within the template if their parameterization is not considered appropriate.

Pay attention to the SSHHost manifests. They declare IP addresses to which Commander has access. In this example, it is assumed that the input parameter .masterHosts is a list of IP addresses based on which the configuration will contain SSH hosts. Since these are masters, they should be specified in the quantity of 1 or 3.

apiVersion: dhctl.deckhouse.io/v1
kind: SSHConfig
# username and port for SSH configured on the machines
sshUser: {{ .sshUser }}
sshPort: {{ .sshPort }}
# private key used on machines is passed as an input parameter to the cluster
sshAgentPrivateKeys:
- key: |
    {{ .sshPrivateKey | nindent 4 }}    

{{- range $masterHost := .masterHosts }}
---
apiVersion: dhctl.deckhouse.io/v1
kind: SSHHost
host: {{ $masterHost.ext_ip }}
{{- end }}

Commander will connect to only one SSH host in the list provided, trying hosts in order until a successful connection is made. The first connected host will become the master node of the cluster. Once Deckhouse is installed on the first master node, it will be able to add the remaining two master nodes to the cluster itself if they are declared in the template. To do this, you need to tell Deckhouse that the machines exist, how to access them, and that they need to be added to the cluster. To do this, create a StaticInstance for the two masters, define SSHCredentials for them, and explicitly write the group of master nodes with the parameter spec.staticInstances.count=2. This will ensure that not only are the two static master nodes known to Deckhouse, but they are also claimed as master nodes. It is advisable to define this part of the template in the Resources. Below is the code of the template for this task:

---
apiVersion: deckhouse.io/v1alpha1
kind: SSHCredentials
metadata:
  name: commander-ssh-credentials
  labels:
    heritage: deckhouse-commander
spec:
  sshPort: {{ .sshPort }}
  user: {{ .sshUser }}
  privateSSHKey: {{ .sshPrivateKey | b64enc }}

{{- if gt (len .masterHosts) 1 }}
{{-   range $masterInstance := slice .masterHosts 1 }}
---
apiVersion: deckhouse.io/v1alpha1
kind: StaticInstance
metadata:
  labels:
    type: master
    heritage: deckhouse-commander
  name: {{ $masterInstance.hostname | quote }}
spec:
  address: {{ $masterInstance.ip | quote }}
  credentialsRef:
    apiVersion: deckhouse.io/v1alpha1
    kind: SSHCredentials
    name: commander-ssh-credentials
{{-   end }}
{{- end }}

{{- if gt (len .masterHosts) 1 }}
---
apiVersion: deckhouse.io/v1
kind: NodeGroup
metadata:
  name: master
  labels:
    heritage: deckhouse-commander
spec:
  disruptions:
    approvalMode: Manual
  nodeTemplate:
    labels:
      node-role.kubernetes.io/control-plane: ""
      node-role.kubernetes.io/master: ""
    taints:
    - effect: NoSchedule
      key: node-role.kubernetes.io/master
    - effect: NoSchedule
      key: node-role.kubernetes.io/control-plane
  nodeType: Static
  staticInstances:
    count: 2
    labelSelector:
      matchLabels:
        type: master
{{- end }}

Resources: deckhouse-commander-agent module

Commander synchronizes resources using the deckhouse-commander-agent module. This module is installed on the target cluster. The commander-agent application requests the current list of resources for the cluster and updates them in the cluster where it is running. To configure the agent correctly, you need to create a manifest in the resources that includes the module.

Please pay attention to commanderUrl. You will have to specify the scheme of this address: HTTP or HTTPS.

Cluster and Resource Parameter Scheme

There are two forms based on input parameter scheme:

• cluster parameter form, the input scheme is specified in a cluster template;
• resource form, the input scheme is specified in its catalog.

The scheme defines an object. There is always an option to edit it in the visual form editor.

Common fields

Text input

A string with default value

- key: something
  type: string
  title: Something
  default: No input

Optional parameter

- key: something
  type: string
  title: Something
  optional: true

The parameter is filled once during the creation of the resource or cluster, and then it cannot be edited (immutable).

- key: something
  type: string
  title: Something
  immutable: true

The password and its description (see format and description)

- key: password
  type: string
  format: password
  minLength: 8
  span: 2
  title: Password
  description: |
    Create a good password.

    The password must contain such and such elements and be updated every N days.    

Number input

One can specify the maximum value and mark the field as optional

- key: ordinarySize
  type: number
  optional: true
  max: 13

One can set a minimum and make the input field span over 4 columns (it is maximum width).

- key: eliteSize
  type: number
  description: In cm
  min: 18
  span: 4

One can set maximum and minimum values.

- key: elephantSize
  type: number
  description: In meters
  min: 0.7
  max: 2.5
  span: 1

Predefined values

Simple values

Selection of predefined values. The user sees in the interface the value that they select.

- key: kubeVersion
  type: string
  title: Kubernetes Version
  enum:
    - Automatic
    - "1.25"
    - "1.26"
    - "1.27"

Complex values

Selection of pre-defined object values. In the interface, the user sees a value from text, while technically value is chosen for the template. Note that in value only object (key-value) type values are available. This value is not described by a schema, the structure of the object is arbitrary.

- key: kubeVersion
  title: Kubernetes Version
  select:
    - text: Default
      value:
        version: Automatic
        isSupported: true
    - text: 1.25 (EOL in March)
      value:
        version: 1.25.8
        isSupported: true
    - text: 1.26
      value:
        version: 1.26.4
        isSupported: true
    - text: 1.27
      value:
        version: 1.27.3
        isSupported: true
    - text: 1.28 (experimental support)
      value:
        version: 1.28.0
        isSupported: false

Fields supported in cluster template

Single resource from a catalog

Catalogs have an unchangeable technical name (slug). It is specified in the catalog property:

- key: slot
  catalog: yandex-cloud-slot
  title: Cloud slot for the cluster
  immutable: true
  description: >
    Select a letter. It will determine the domain, prefix in the cloud, and IP address.
    This slot is unique for all clusters regardless of the template.

    Let's say you chose 'N'. The domain template will be '%s.X.kube.example.com'.
    We recommend naming the cluster 'dev-X'.

    The login and password are always 'admin@example.com'.    

Multiple resources from a catalog

Multiple choice is provided by a pair of properties: minItems and maxItems. Any field can be made a list of data if both of these fields are specified.

- key: slot
  catalog: virtual-machines
  title: Worker nodes
  description: Any count of worker nodes
  minItems: 0
  maxItems: 10000

Auto-selection

Sometimes it is not important to the user which resource is selected from the pool of resources. Therefore, auto-selection makes a substitution of a free resource automatically. The automatically selected resource can be replaced manually with another one.

- key: publicAddressesForFrontendNodes
  title: Public addresses
  catalog: public-ip-addresses
  minItems: 3
  maxItems: 3
  autoselect: true

Parts

Separators

header

• Type: string

The only type of a divider is a header. It supports only text. It has no other properties.

- header: Access to container images

Properties of input fields

key

• Type: string
• Required

It is necessary to identify the value of an input field in a template. Therefore, there must always be a key field property — this field name will be used in the template during configuration rendering.

In the schema:

- key: podSubnet
  title: Pod subnet
  type: string

In a template:

podSubnet: { { .podSubnet | quote } }

type

• Type: string
• Required
• Supported values:
  • string
  • number
  • boolean

The value has a predefined type: string, number, or boolean.

title

• Type: string
• Required

The field has a title that conveys meaning. It is one line of text. It is displayed in the parameter form and in the audit.

description

• Type: string

A field may have a comment that reveals the meaning, explains boundary conditions, recording format or exceptions. There may be several lines of text.

default

• Type: depends on type

The Default value is filled in if the field is marked optional. Also, this value is shown to the user, for example, in the form of a placeholder. The value type of this property must match the type.

format

• Type: string
• Supported values:
  • password
  • date-time
  • url
  • email
  • uuid
  • cuid
  • cuid2
  • ulid
  • emoji

String parameters can have a format that determines the specifics of their display and validation.

span

• Type: number
• Supported values: 1, 2, 3, 4
• Default: 1

This is a decorative property that specifies how much width to occupy on the screen in fractions: from 1 to 4. Input fields fill the form one line at a time horizontally, like text. In this case, the width of a “line” in the form is 4 elements.

optional

• Type: boolean
• Default: false

This flag indicates that the field is optional. An empty value will be ignored and the property will not be passed to the template.

immutable

• Type: boolean
• Default: false

This flag indicates that the field is filled only once when it appears in the input parameters. The field becomes unavailable for editing if it has already been filled. This means that when you update a cluster to a new template with an immutable field, you can fill it in.

Immunity depends on the life cycle of the parameter in the form, not the cluster.

enum

• Type: array

Lists the possible values that the field accepts. The field is represented by a select regardless of the value type selected in type.

selector

• Type: array

This is a more complex version of enum. It provides a string representation of an object (text) and an arbitrarily complex value in value that will be selected for the template. The text is provided for humans, and the values are for the template.

Example:

- key: kubeVersion
  title: Kubernetes Version
  select:
    - text: Default
      value:
        version: Automatic
        isSupported: true
    - text: 1.25 (EOL in March)
      value:
        version: 1.25.8
        isSupported: true
    - text: 1.26
      value:
        version: 1.26.4
        isSupported: true
    - text: 1.27
      value:
        version: 1.27.3
        isSupported: true
    - text: 1.28 (experimental support)
      value:
        version: 1.28.0
        isSupported: false

catalog

• Type: string

Selecting one value from the resources catalog. Write the slug of the resources catalog in the value. The type field does not need to be specified, because in fact it is object, the schema of which is described in the specified catalog. To select several values (and get a list of resources at the entrance to the template), use minItems and maxItems.

Example:

- key: workerMachine
  title: Virtual machine
  catalog: virtual-machines

- key: workerMachines
  title: Virtual machines
  catalog: virtual-machines
  minItems: 1
  maxItems: 10

maxLength (for strings)

• Type: number

For type: string, this field adds validation on the string length.

minItems, maxItems (for resources selection)

• Type: number

Validation of the number of items selected from the resource catalog. This pair of fields is optional, but it is prohibited to use them separately: if used, then both at once.

autoselect (for resources selection)

• Type: boolean
• Default: false

Sometimes, it is not so important for the user which specific resource is chosen. In this case, the form chooses an available resource for the user. However, the user always has the option to change them.

Example:

- key: publicAddressesForFrontendNodes
  title: Public addresses
  catalog: public-ip-addresses
  minItems: 3
  maxItems: 3
  autoselect: true

identifier

• Type: boolean
• Default: depends on values

A resource is a flat object. A resource has a compact one-line representation made up of the field values of the resource (without keys). The resource values are separated by commas in the order specified by the schema. This compact representation can be seen both in the list of resources inside the directory and in the selection of a resource in a cluster view (dropdown lists). To select a limited set of fields for a compact display of a resource, use the identifier property.

For example, consider a resource and its possible schema options.

## Resource
login: John
password: E3xE#%DH@hW
age: 42

- key: login
  type: string
  title: Username
  unique: true
  pattern: ^[a-z0-9.-]+$

- key: password
  type: string
  title: Password
  format: password

- key: age
  type: number
  title: Age

 - key: login
   type: string
   title: Username
   unique: true
   pattern: ^[a-z0-9.-]+$

 - key: password
   type: string
   title: Password
   format: password
+  identifier: false

 - key: age
   type: number
   title: Age

 - key: login
   type: string
   title: Username
   unique: true
   pattern: ^[a-z0-9.-]+$
+  identifier: true

 - key: password
   type: string
   title: Password
   format: password

 - key: age
   type: number
   title: Age
+  identifier: true

unique

• Type: boolean
• Default: false
• This flag marks resource fields that must be unique within a catalog. It is prohibited to create, restore from archive, or import non-unique data.

This flag also marks cluster fields that must be unique among all clusters. Saving a cluster with non-unique data will result in a validation error and prevent creation and editing of the cluster.

The selection from a catalog for clusters is unique to the extent that resources are unique. It is not necessary to mark a selection from a catalog with this flag.

- key: subnetCIDR
  type: string
  title: Subnet CIDR
  unique: true