NodeGroup

Scope: Cluster

Describes the runtime parameters of the node group.

  • spec (object)

    Required value.

    • chaos (object)

      Chaos monkey settings.

      Example:

      mode: DrainAndReboot
      period: 24h
      
      • mode (string)

        The chaos monkey mode.

        • DrainAndDelete — drains and deletes a node when triggered
        • Disabled — leaves this NodeGroup intact.

        Default: "Disabled"

        Allowed values: Disabled, DrainAndDelete

      • period (string)

        The time interval to use for the chaos monkey (can be specified in the golang format).

        Default: "6h"

        Pattern: ^[0-9]+[mh]{1}$

    • cloudInstances (object)

      Parameter for provisioning the cloud-based VMs.

      Caution! Can only be used together with nodeType: CloudEphemeral.

      • classReference (object)

        The reference to the InstanceClass object (unique for each cloud-provider- module).

        Required value.

        • kind (string)

          The object type (e.g., OpenStackInstanceClass). The object type is specified in the documentation of the corresponding cloud-provider- module.

          Allowed values: OpenStackInstanceClass, GCPInstanceClass, VsphereInstanceClass, AWSInstanceClass, YandexInstanceClass, AzureInstanceClass

        • name (string)

          The name of the required InstanceClass object (e.g., finland-medium).

      • maxPerZone (integer)

        The maximum number of instances for the group in each zone.

        This value is used as the upper bound in cluster-autoscaler.

        Allowed values: 0 <= X

        Required value.

      • maxSurgePerZone (integer)

        The maximum number of instances to rollout simultaneously in the group in each zone.

        Default: 1

        Allowed values: 0 <= X

      • maxUnavailablePerZone (integer)

        The maximum number of unavailable instances (during rollout) in the group in each zone.

        Default: 0

        Allowed values: 0 <= X

      • minPerZone (integer)

        The minimum number of instances for the group in each zone.

        This value is used in the MachineDeployment object and as a lower bound in cluster-autoscaler.

        Allowed values: 0 <= X

        Required value.

      • standby (integer or string)

        The number of overprovisioned Nodes for this NodeGroup.

        The value can be an absolute number (for example, 2) or a percentage of desired Nodes (for example, 10%). The absolute number is calculated from percentage of maximum Nodes amount by rounding down.

        Pattern: ^[0-9]+%?$

      • standbyHolder (object)

        Parameters of the “idle” process (resource consumer) for the prepared instances.

        • notHeldResources (object)

          Describes the resources that will not be held (consumed) by the standby holder.

          • cpu (integer or string)

            Describes the amount of CPU that will not be held by standby holder on Nodes from this NodeGroup.

            The value can be an absolute number of cpus (for example, 2) as well as a milli representation (for example, 1500m).

            Pattern: ^[0-9]+m?$

          • memory (integer or string)

            Describes the amount of memory that will not be held by standby holder on Nodes from this NodeGroup.

            The value can be an absolute number of bytes (for example, 128974848) as well as a fixed-point number using one of memory suffixes: G, Gi, M, Mi.

            Pattern: ^[0-9]+(E|P|T|G|M|K|Ei|Pi|Ti|Gi|Mi|Ki)?$

      • zones (array of strings)

        List of availability zones to create instances in.

        The default value depends on the cloud provider selected and usually corresponds to all zones of the region being used.

        Example:

        - Helsinki
        - Espoo
        - Tampere
        
    • cri (object)

      Container runtime parameters.

      • containerd (object)

        Containerd runtime parameters.

        If used, cri.type must be set to Containerd.

        • maxConcurrentDownloads (integer)

          Set the max concurrent downloads for each pull.

          Default: 3

      • docker (object)

        Docker settings for nodes.

        • manage (boolean)

          Enable docker maintenance from bashible.

          Default: true

        • maxConcurrentDownloads (integer)

          Set the max concurrent downloads for each pull.

          Default: 3

      • type (string)

        Container runtime type.

        Value defaultCRI from the initial cluster configration (cluster-configuration.yaml parameter from the d8-cluster-configuration secret in the kube-system namespace) is used if not specified.

        Optional parameter.

        Allowed values: Docker, Containerd, NotManaged

    • disruptions (object)

      Disruptions settings for nodes.

      • approvalMode (string)

        The approval mode for disruptive updates.

        • Manual — disable automatic disruption approval; the alert will be displayed if disruption is needed.
        • Automatic — automatically approve disruption-involving updates (the default value).

        Allowed values: Manual, Automatic

        Required value.

      • automatic (object)

        Additional parameters for the Automatic mode.

        • drainBeforeApproval (boolean)

          Drain pods from the nodes before approving disruption.

          Default: true

    • kubelet (object)

      Kubelet settings for nodes.

      • maxPods (integer)

        Set the max count of pods per node.

        Default: 110

      • rootDir (string)

        Directory path for managing kubelet files (volume mounts,etc).

        Default: "/var/lib/kubelet"

    • nodeTemplate (object)

      Specification of some of the fields that will be maintained in all nodes of the group.

      Example:

      labels:
        environment: production
        app: warp-drive-ai
      annotations:
        ai.fleet.com/discombobulate: "true"
      taints:
      - effect: NoExecute
        key: ship-class
        value: frigate
      
      • annotations (object)

        Similar to the standard metadata.annotations field.

        Example:

        annotations:
          ai.fleet.com/discombobulate: "true"
        
      • labels (object)

        Similar to the standard metadata.labels field.

        Example:

        labels:
          environment: production
          app: warp-drive-ai
        
      • taints (array of objects)

        Similar to the .spec.taints field of the Node object.

        Caution! Only effect, key, value fields are available.

        Example:

        taints:
        - effect: NoExecute
          key: ship-class
          value: frigate
        
        • effect (string)

          Allowed values: NoSchedule, PreferNoSchedule, NoExecute

        • key (string)
        • value (string)
    • nodeType (string)

      The type of nodes this group provides.

      • CloudEphemeral - nodes for this group will be automatically created (and deleted) in the cloud of the specified cloud provider.
      • CloudPermanent - nodes from ProviderClusterConfiguration will be created via dhctl.
      • CloudStatic - a static node (created manually or using any external tools) hosted in the cloud integrated with one of the cloud providers; this node has the CSI running, and it is managed by the cloud-controller-manager (the Node object automatically gets the information about the zone and region based on the cloud data; if a node gets deleted from the cloud, its corresponding Node object will be deleted in Kubernetes).
      • Static — a static node hosted on a bare metal or virtual machine. The cloud-controller-manager does not manage the node even if one of the cloud providers is enabled.

      Allowed values: CloudEphemeral, CloudPermanent, CloudStatic, Static

      Required value.

    • operatingSystem (object)

      Operating System settings for nodes.

      • manageKernel (boolean)

        Enable kernel maintenance from bashible.

        Default: true

Describes the runtime parameters of the node group.

  • spec (object)

    Required value.

    • chaos (object)

      Chaos monkey settings.

      Example:

      mode: DrainAndReboot
      period: 24h
      
      • mode (string)

        The chaos monkey mode.

        • DrainAndDelete — drains and deletes a node when triggered
        • Disabled — leaves this NodeGroup intact.

        Default: "Disabled"

        Allowed values: Disabled, DrainAndDelete

      • period (string)

        The time interval to use for the chaos monkey (can be specified in the golang format).

        Default: "6h"

        Pattern: ^[0-9]+[mh]{1}$

    • cloudInstances (object)

      Parameter for provisioning the cloud-based VMs.

      Caution! Can only be used together with nodeType: CloudEphemeral.

      • classReference (object)

        The reference to the InstanceClass object (unique for each cloud-provider- module).

        Required value.

        • kind (string)

          The object type (e.g., OpenStackInstanceClass). The object type is specified in the documentation of the corresponding cloud-provider- module.

          Allowed values: OpenStackInstanceClass, GCPInstanceClass, VsphereInstanceClass, AWSInstanceClass, YandexInstanceClass, AzureInstanceClass

        • name (string)

          The name of the required InstanceClass object (e.g., finland-medium).

      • maxPerZone (integer)

        The maximum number of instances for the group in each zone.

        This value is used as the upper bound in cluster-autoscaler.

        Allowed values: 0 <= X

        Required value.

      • maxSurgePerZone (integer)

        The maximum number of instances to rollout simultaneously in the group in each zone.

        Default: 1

        Allowed values: 0 <= X

      • maxUnavailablePerZone (integer)

        The maximum number of unavailable instances (during rollout) in the group in each zone.

        Default: 0

        Allowed values: 0 <= X

      • minPerZone (integer)

        The minimum number of instances for the group in each zone.

        This value is used in the MachineDeployment object and as a lower bound in cluster-autoscaler.

        Allowed values: 0 <= X

        Required value.

      • standby (integer or string)

        The number of overprovisioned Nodes for this NodeGroup.

        The value can be an absolute number (for example, 2) or a percentage of desired Nodes (for example, 10%). The absolute number is calculated from percentage of maximum Nodes amount by rounding down.

        Pattern: ^[0-9]+%?$

      • standbyHolder (object)

        Parameters of the “idle” process (resource consumer) for the prepared instances.

        • notHeldResources (object)

          Describes the resources that will not be held (consumed) by the standby holder.

          • cpu (integer or string)

            Describes the amount of CPU that will not be held by standby holder on Nodes from this NodeGroup.

            The value can be an absolute number of cpus (for example, 2) as well as a milli representation (for example, 1500m).

            Pattern: ^[0-9]+m?$

          • memory (integer or string)

            Describes the amount of memory that will not be held by standby holder on Nodes from this NodeGroup.

            The value can be an absolute number of bytes (for example, 128974848) as well as a fixed-point number using one of memory suffixes: G, Gi, M, Mi.

            Pattern: ^[0-9]+(E|P|T|G|M|K|Ei|Pi|Ti|Gi|Mi|Ki)?$

      • zones (array of strings)

        List of availability zones to create instances in.

        The default value depends on the cloud provider selected and usually corresponds to all zones of the region being used.

        Example:

        - Helsinki
        - Espoo
        - Tampere
        
    • cri (object)

      Container runtime parameters.

      • containerd (object)

        Containerd runtime parameters.

        If used, cri.type must be set to Containerd.

        • maxConcurrentDownloads (integer)

          Set the max concurrent downloads for each pull.

          Default: 3

      • docker (object)

        Docker settings for nodes.

        • manage (boolean)

          Enable docker maintenance from bashible.

          Default: true

        • maxConcurrentDownloads (integer)

          Set the max concurrent downloads for each pull.

          Default: 3

      • type (string)

        Container runtime type.

        Value defaultCRI from the initial cluster configration (cluster-configuration.yaml parameter from the d8-cluster-configuration secret in the kube-system namespace) is used if not specified.

        Optional parameter.

        Allowed values: Docker, Containerd, NotManaged

    • disruptions (object)

      Disruptions settings for nodes.

      • approvalMode (string)

        The approval mode for disruptive updates.

        • Manual — disable automatic disruption approval; the alert will be displayed if disruption is needed.
        • Automatic — automatically approve disruption-involving updates (the default value).

        Allowed values: Manual, Automatic

        Required value.

      • automatic (object)

        Additional parameters for the Automatic mode.

        • drainBeforeApproval (boolean)

          Drain pods from the nodes before approving disruption.

          Default: true

    • kubelet (object)

      Kubelet settings for nodes.

      • maxPods (integer)

        Set the max count of pods per node.

        Default: 110

      • rootDir (string)

        Directory path for managing kubelet files (volume mounts,etc).

        Default: "/var/lib/kubelet"

    • nodeTemplate (object)

      Specification of some of the fields that will be maintained in all nodes of the group.

      Example:

      labels:
        environment: production
        app: warp-drive-ai
      annotations:
        ai.fleet.com/discombobulate: "true"
      taints:
      - effect: NoExecute
        key: ship-class
        value: frigate
      
      • annotations (object)

        Similar to the standard metadata.annotations field.

        Example:

        annotations:
          ai.fleet.com/discombobulate: "true"
        
      • labels (object)

        Similar to the standard metadata.labels field.

        Example:

        labels:
          environment: production
          app: warp-drive-ai
        
      • taints (array of objects)

        Similar to the .spec.taints field of the Node object.

        Caution! Only effect, key, value fields are available.

        Example:

        taints:
        - effect: NoExecute
          key: ship-class
          value: frigate
        
        • effect (string)

          Allowed values: NoSchedule, PreferNoSchedule, NoExecute

        • key (string)
        • value (string)
    • nodeType (string)

      The type of nodes this group provides.

      • Cloud — nodes for this group will be automatically created (and deleted) in the cloud of the specified cloud provider.
      • Static — a static node hosted on a bare metal or virtual machine. The cloud-controller-manager does not manage the node even of one of the cloud providers is enabled.
      • Hybrid – a static node (created manually or using any external tools) hosted in the cloud integrated with one of the cloud providers; this node has the CSI running, and it is managed by the cloud-controller-manager (the Node object automatically gets the information about the zone and region based on the cloud data; if a node gets deleted from the cloud, its corresponding Node object will be deleted in Kubernetes).

      Allowed values: Cloud, Static, Hybrid

      Required value.

    • operatingSystem (object)

      Operating System settings for nodes.

      • manageKernel (boolean)

        Enable kernel maintenance from bashible.

        Default: true

Defines the runtime parameters of a node group.

  • spec (object)

    Required value.

    • chaos (object)

      Chaos monkey settings.

      Example:

      mode: DrainAndReboot
      period: 24h
      
      • mode (string)

        The chaos monkey mode.

        • DrainAndDelete — drains and deletes a node when triggered
        • Disabled — leaves this NodeGroup intact.

        Default: "Disabled"

        Allowed values: Disabled, DrainAndDelete

      • period (string)

        The time interval to use for the chaos monkey (can be specified in the golang format).

        Default: "6h"

        Pattern: ^[0-9]+[mh]{1}$

    • cloudInstances (object)

      Parameter for provisioning the cloud-based VMs.

      Caution! Can only be used together with nodeType: CloudEphemeral.

      • classReference (object)

        The reference to the InstanceClass object (unique for each cloud-provider- module).

        Required value.

        • kind (string)

          The object type (e.g., OpenStackInstanceClass). The object type is specified in the documentation of the corresponding cloud-provider- module.

          Allowed values: OpenStackInstanceClass, GCPInstanceClass, VsphereInstanceClass, AWSInstanceClass, YandexInstanceClass, AzureInstanceClass

        • name (string)

          The name of the required InstanceClass object (e.g., finland-medium).

      • maxPerZone (integer)

        The maximum number of instances for the group in each zone.

        This value is used as the upper bound in cluster-autoscaler.

        Allowed values: 0 <= X

        Required value.

      • maxSurgePerZone (integer)

        The maximum number of instances to rollout simultaneously in the group in each zone.

        Default: 1

        Allowed values: 0 <= X

      • maxUnavailablePerZone (integer)

        The maximum number of unavailable instances (during rollout) in the group in each zone.

        Default: 0

        Allowed values: 0 <= X

      • minPerZone (integer)

        The minimum number of instances for the group in each zone.

        This value is used in the MachineDeployment object and as a lower bound in cluster-autoscaler.

        Allowed values: 0 <= X

        Required value.

      • standby (integer or string)

        The number of overprovisioned Nodes for this NodeGroup.

        The value can be an absolute number (for example, 2) or a percentage of desired Nodes (for example, 10%). The absolute number is calculated from percentage of maximum Nodes amount by rounding down.

        Pattern: ^[0-9]+%?$

      • standbyHolder (object)

        Parameters of the “idle” process (resource consumer) for the prepared instances.

        • notHeldResources (object)

          Describes the resources that will not be held (consumed) by the standby holder.

          • cpu (integer or string)

            Describes the amount of CPU that will not be held by standby holder on Nodes from this NodeGroup.

            The value can be an absolute number of cpus (for example, 2) as well as a milli representation (for example, 1500m).

            Pattern: ^[0-9]+m?$

          • memory (integer or string)

            Describes the amount of memory that will not be held by standby holder on Nodes from this NodeGroup.

            The value can be an absolute number of bytes (for example, 128974848) as well as a fixed-point number using one of memory suffixes: G, Gi, M, Mi.

            Pattern: ^[0-9]+(E|P|T|G|M|K|Ei|Pi|Ti|Gi|Mi|Ki)?$

      • zones (array of strings)

        List of availability zones to create instances in.

        The default value depends on the cloud provider selected and usually corresponds to all zones of the region being used.

        Example:

        - Helsinki
        - Espoo
        - Tampere
        
    • cri (object)

      Container runtime parameters.

      • containerd (object)

        Containerd runtime parameters.

        If used, cri.type must be set to Containerd.

        • maxConcurrentDownloads (integer)

          Set the max concurrent downloads for each pull.

          Default: 3

      • type (string)

        Container runtime type.

        Value defaultCRI from the initial cluster configration (cluster-configuration.yaml parameter from the d8-cluster-configuration secret in the kube-system namespace) is used if not specified.

        Optional parameter.

        Allowed values: Docker, Containerd, NotManaged

    • disruptions (object)

      Disruptions settings for nodes.

      • approvalMode (string)

        The approval mode for disruptive updates.

        • Manual — disable automatic disruption approval; the alert will be displayed if disruption is needed.
        • Automatic — automatically approve disruption-involving updates (the default value).

        Allowed values: Manual, Automatic

        Required value.

      • automatic (object)

        Additional parameters for the Automatic mode.

        • drainBeforeApproval (boolean)

          Drain pods from the nodes before approving disruption.

          Default: true

    • docker (object)

      Docker settings for nodes.

      If used, cri.type must be set to Docker.

      • manage (boolean)

        Enable docker maintenance from bashible.

        Default: true

      • maxConcurrentDownloads (integer)

        Set the max concurrent downloads for each pull.

        Default: 3

    • kubelet (object)

      Kubelet settings for nodes.

      • maxPods (integer)

        Set the max count of pods per node.

        Default: 110

      • rootDir (string)

        Directory path for managing kubelet files (volume mounts,etc).

        Default: "/var/lib/kubelet"

    • kubernetesVersion (string)

      The desired minor version of Kubernetes.

      By default, it corresponds to the version selected for the cluster globally (see installation documentation) or to the current version of the control-plane (if the global version is not defined).

      Allowed values: 1.14, 1.15, 1.16, 1.17, 1.18, 1.19, 1.20, 1.21

      Example: "1.19"

    • nodeTemplate (object)

      Specification of some of the fields that will be maintained in all nodes of the group.

      Example:

      labels:
        environment: production
        app: warp-drive-ai
      annotations:
        ai.fleet.com/discombobulate: "true"
      taints:
      - effect: NoExecute
        key: ship-class
        value: frigate
      
      • annotations (object)

        Similar to the standard metadata.annotations field.

        Example:

        annotations:
          ai.fleet.com/discombobulate: "true"
        
      • labels (object)

        Similar to the standard metadata.labels field.

        Example:

        labels:
          environment: production
          app: warp-drive-ai
        
      • taints (array of objects)

        Similar to the .spec.taints field of the Node object.

        Caution! Only effect, key, value fields are available.

        Example:

        taints:
        - effect: NoExecute
          key: ship-class
          value: frigate
        
        • effect (string)

          Allowed values: NoSchedule, PreferNoSchedule, NoExecute

        • key (string)
        • value (string)
    • nodeType (string)

      The type of nodes this group provides.

      • Cloud — nodes for this group will be automatically created (and deleted) in the cloud of the specified cloud provider.
      • Static — a static node hosted on a bare metal or virtual machine. The cloud-controller-manager does not manage the node even of one of the cloud providers is enabled.
      • Hybrid – a static node (created manually or using any external tools) hosted in the cloud integrated with one of the cloud providers; this node has the CSI running, and it is managed by the cloud-controller-manager (the Node object automatically gets the information about the zone and region based on the cloud data; if a node gets deleted from the cloud, its corresponding Node object will be deleted in Kubernetes).

      Allowed values: Cloud, Static, Hybrid

      Required value.

    • operatingSystem (object)

      Operating System settings for nodes.

      • manageKernel (boolean)

        Enable kernel maintenance from bashible.

        Default: true

    • static (object)

      Static node parameters

      • internalNetworkCIDRs (array of strings)

        Subnet CIDR

NodeUser

Scope: Cluster

Defines the linux users to create on all nodes.

The user’s home directory is created in the /home/deckhouse/ directory.

  • spec (object)

    Required value.

    • extraGroups (array of strings)

      Node user additional system groups.

    • isSudoer (boolean)

      Persistence of node user in sudo group.

      Default: false

      Example: true

    • passwordHash (string)

      Hashed user password.

      The format corresponds to the password hashes in /etc/shadow. Yoou can get it using the following command: openssl passwd -6.

      Example: "$2a$10$F9ey7zW.sVliT224RFxpWeMsgzO.D9YRG54a8T36/K2MCiT41nzmC"

      Required value.

    • sshPublicKey (string)

      Node user ssh public key.

      Example: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQ"

      Required value.

    • uid (number)

      Node user ID.

      This parameter does not change during the entire resource life.

      Allowed values: 1001 <= X

      Example: 1001

      Required value.

Defines the linux users to create on all nodes.

The user’s home directory is created in the /home/deckhouse/ directory.

  • spec (object)

    Required value.

    • extraGroups (array of strings)

      Node user additional system groups.

    • isSudoer (boolean)

      Persistence of node user in sudo group.

      Default: false

      Example: true

    • passwordHash (string)

      Hashed user password.

      The format corresponds to the password hashes in /etc/shadow. Yoou can get it using the following command: openssl passwd -6.

      Example: "$2a$10$F9ey7zW.sVliT224RFxpWeMsgzO.D9YRG54a8T36/K2MCiT41nzmC"

      Required value.

    • sshPublicKey (string)

      Node user ssh public key.

      Example: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQ"

      Required value.

    • uid (number)

      Node user ID.

      This parameter does not change during the entire resource life.

      Allowed values: 1001 <= X

      Example: 1001

      Required value.