Custom resources | node-manager

The Deckhouse Platform installs CRDs but does not remove them when a module is disabled. If you no longer need the created CRDs, delete them.

The module lifecycle stage: General Availability

Instance

Scope: Cluster

v1alpha2
v1alpha1

Instance describes the full lifecycle of a server in Deckhouse Platform, aggregating status from the Machine object (infrastructure) and Bashible service on the node (configuration).

spec
object
Specification of the Instance, including references to related resources.
- spec.classReference
  object
  The reference to the InstanceClass object. It is unique for each cloud-provider-* module.
  - spec.classReference.kind
    string
    The object type (e.g., OpenStackInstanceClass). The object type is specified in the documentation of the corresponding cloud-provider- module.
  - spec.classReference.name
    string
    The name of the required InstanceClass object (e.g., finland-medium).
- spec.machineRef
  object
  Reference to instance in the cloud.
  - spec.machineRef.apiVersion
    string
    APIVersion defines the versioned schema of this representation of an reference object.
  - spec.machineRef.kind
    string
    Kind is a string value representing the REST resource referenced object represents.
  - spec.machineRef.name
    string
    Name of referenced object.
  - spec.machineRef.namespace
    string
    Namespace of referenced object.
- spec.nodeGroupRef
  object
  Reference to the NodeGroup that ordered this instance. It is known before the node joins the cluster.
  - spec.nodeGroupRef.name
    string
    NodeGroup name.
- spec.nodeRef
  object
  Reference to the kubernetes node object.
  - spec.nodeRef.name
    string
    Node name.
status
object
Most recently observed status of the Instance.
- status.bashibleStatus
  string
  Aggregated status of the bashible configuration process.
  
  Derived from BashibleReady condition.
  
  Allowed values: Error, Unknown, Ready, WaitingApproval
- status.bootstrapStatus
  object
  Information about instance bootstrapping process.
  - status.bootstrapStatus.description
    string
    Description about getting logs.
  - status.bootstrapStatus.logsEndpoint
    string
    http endpoint for getting bootstrap logs.
- status.conditions
  array of objects
  Detailed conditions of the instance (MachineReady, BashibleReady, etc.).
  
  On a node that has a NodeConfig, NodeAgentReady is True while the node agent reports and Unknown with reason NoReport after 10 minutes without a report.
  - status.conditions.lastHeartbeatTime
    string
    Last time the condition heartbeat was updated by its source.
  - status.conditions.lastTransitionTime
    string
    Last time the condition transitioned from one status to another.
  - status.conditions.message
    string
    A human-readable message indicating details about the transition.
  - status.conditions.observedGeneration
    integer
    The generation observed by the controller.
  - status.conditions.reason
    string
    Required value
    The reason for the condition’s last transition.
  - status.conditions.severity
    string
    Severity of the condition (Error, Warning, Info).
  - status.conditions.status
    string
    Required value
    Status of the condition (True, False, Unknown).
  - status.conditions.type
    string
    Required value
    Type of the condition (e.g., MachineReady, BashibleReady).
- status.machineStatus
  string
  Aggregated status of the infrastructure machine.
  
  Derived from MachineReady condition.
  
  Allowed values: Progressing, Error, Ready, Blocked, Rebooting
- status.message
  string
  A human-readable message indicating details about the current status or error.
  
  Displayed in CLI only when using -o wide for d8 k.
- status.phase
  string
  The high-level lifecycle phase of the Instance.
  
  Mapping: Pending -> Provisioning -> Provisioned -> Running -> Terminating.
  
  Allowed values: Pending, Provisioning, Provisioned, Running, Terminating, Unknown

Describes an implementation-independent ephemeral machine resource.

status
object
- status.bootstrapStatus
  object
  Information about instance bootstrapping process.
  - status.bootstrapStatus.description
    string
    Description about getting logs.
  - status.bootstrapStatus.logsEndpoint
    string
    http endpoint for getting bootstrap logs.
- status.classReference
  object
  The reference to the InstanceClass object. It is unique for each cloud-provider-* module.
  - status.classReference.kind
    string
    The object type (e.g., OpenStackInstanceClass). The object type is specified in the documentation of the corresponding cloud-provider- module.
    
    Allowed values: OpenStackInstanceClass, GCPInstanceClass, VsphereInstanceClass, AWSInstanceClass, YandexInstanceClass, AzureInstanceClass, VCDInstanceClass, ZvirtInstanceClass, DynamixInstanceClass, HuaweiCloudInstanceClass, DVPInstanceClass
  - status.classReference.name
    string
    The name of the required InstanceClass object (e.g., finland-medium).
- status.currentStatus
  object
  Current status of the instance object.
  - status.currentStatus.lastUpdateTime
    string
    Last update time of current status.
  - status.currentStatus.phase
    string
    InstancePhase is a label for the condition of a instance at the current time.
- status.lastOperation
  object
  Last operation refers to the status of the last operation performed.
  - status.lastOperation.description
    string
    Description of the current operation.
  - status.lastOperation.lastUpdateTime
    string
    Last update time of current operation.
  - status.lastOperation.state
    string
    State of operation.
  - status.lastOperation.type
    string
    Type of operation.
- status.machineRef
  object
  Reference to instance in the cloud.
  - status.machineRef.apiVersion
    string
    APIVersion defines the versioned schema of this representation of an reference object.
  - status.machineRef.kind
    string
    Kind is a string value representing the REST resource referenced object represents.
  - status.machineRef.name
    string
    Name of referenced object.
  - status.machineRef.namespace
    string
    Namespace of referenced object.
- status.nodeRef
  object
  Reference to the kubernetes node object.
  - status.nodeRef.name
    string
    Node name.

InstanceTypesCatalog

Scope: Cluster
Version: v1alpha1

Contains information about discovered data from cloud.

instanceTypes
array of objects
Required value
List of available instance types.
- instanceTypes.cpu
  integer or string
  Required value
  The number of CPUs for the instance type.
  
  Pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
- instanceTypes.memory
  integer or string
  Required value
  The amount of RAM a instance type has.
  
  Pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$
- instanceTypes.name
  string
  Instance type name.
- instanceTypes.rootDisk
  integer or string
  Root disk size.
  
  Pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$

NodeGroup

Short names: ng

Scope: Cluster

v1
v1alpha2
v1alpha1

Describes the runtime parameters of the node group.

Example:

# NodeGroup for cloud nodes in AWS.
apiVersion: deckhouse.io/v1
kind: NodeGroup
metadata:
  name: test
spec:
  nodeType: CloudEphemeral
  cloudInstances:
    zones:
      - eu-west-1a
      - eu-west-1b
    minPerZone: 1
    maxPerZone: 2
    classReference:
      kind: AWSInstanceClass
      name: test
  nodeTemplate:
    labels:
      tier: test
---
# NodeGroup for static nodes on bare metal servers (or VMs).
apiVersion: deckhouse.io/v1
kind: NodeGroup
metadata:
  name: worker
spec:
  nodeType: Static

metadata
object
- metadata.name
  string
  Pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$
  Maximum length: 42
spec
object
Required value
- spec.chaos
  object
  Chaos monkey settings.
  
  Example:
  chaos: mode: DrainAndDelete period: 24h
  - spec.chaos.mode
    string
    The chaos monkey mode:
    
    DrainAndDelete — drains and deletes a node when triggered;
    
    Disabled — leaves this NodeGroup intact.
    Default: Disabled
    Allowed values: Disabled, DrainAndDelete
  - spec.chaos.period
    string
    The time interval to use for the chaos monkey.
    
    It is specified as a string containing the time unit in hours and minutes: 30m, 1h, 2h30m, 24h.
    
    Default: 6h
    Pattern: ^([0-9]+h([0-9]+m)?|[0-9]+m)$
- spec.cloudInstances
  object
  Parameter for provisioning the cloud-based VMs.
  
  Caution. Can only be used together with nodeType: CloudEphemeral.
  - spec.cloudInstances.classReference
    object
    Required value
    The reference to the InstanceClass object. It is unique for each cloud-provider-* module.
    - spec.cloudInstances.classReference.kind
      string
      The kind of the InstanceClass object (e.g., OpenStackInstanceClass). Supported kinds are defined by the installed cloud provider modules and specified in their documentation.
    - spec.cloudInstances.classReference.name
      string
      The name of the required InstanceClass object (e.g., finland-medium).
  - spec.cloudInstances.maxPerZone
    integer
    Required value
    The maximum number of instances for the group in each zone.
    
    This value is used as the upper bound in cluster-autoscaler.
    
    Allowed values: 0 <= X
  - spec.cloudInstances.maxSurgePerZone
    integer
    The maximum number of instances the platform creates in each zone above the desired number while it replaces the instances of the group.
    
    Replacement happens when parameters that require recreating a virtual machine change, for example in the InstanceClass. The parameter does not limit the number of instances that the cluster autoscaler orders when it scales the group up. For details on what NodeGroup parameters affect, see NodeGroup parameters and their result.
    
    The value 0 is treated as if the parameter is not set, so the default value applies.
    
    Default: 1
    Allowed values: 0 <= X
  - spec.cloudInstances.maxUnavailablePerZone
    integer
    The maximum number of unavailable instances (during rollout) in the group in each zone.
    
    Default: 0
    Allowed values: 0 <= X
  - spec.cloudInstances.minPerZone
    integer
    Required value
    The minimum number of instances for the group in each zone.
    
    This value is used in the MachineDeployment object and as a lower bound in cluster-autoscaler.
    
    Allowed values: 0 <= X
  - spec.cloudInstances.priority
    integer
    Priority of the node group.
    
    When scaling a cluster, the autoscaler will first select node groups with a higher priority set. If several node groups have the same priority, the autoscaler randomly selects a group of them. You can control the rate at which nodes are switched to lower-priority groups on demand (for more details, see the FAQ).
    
    Using priorities can be convenient to prefer ordering cheaper nodes (for example, spot instances) over more expensive ones.
  - spec.cloudInstances.quickShutdown
    Deprecated
    boolean
    Lowers CloudEphemeral machine drain timeout to 5 minutes.
  - spec.cloudInstances.standby
    integer or string
    The summary number of overprovisioned nodes for this NodeGroup in all zones.
    
    An overprovisioned node is a cluster node on which resources are reserved that are available at any time for scaling. The presence of such a node allows the cluster autoscaler not to wait for node initialization (which may take several minutes), but to immediately place a load on it.
    
    The value can be an absolute number (for example, 2) or a percentage of desired nodes (for example, 10%). If a percentage is specified, the absolute number is calculated based on the percentage of the maximum number of nodes (the maxPerZone parameter) rounded down, but not less than one.
    
    Pattern: ^[0-9]+%?$
  - spec.cloudInstances.standbyHolder
    object
    Amount of reserved resources.
    
    Used to determine whether to order overprovisioned nodes.
    - spec.cloudInstances.standbyHolder.notHeldResources
      Deprecated
      object
      Deprecated: the parameter is no longer used. Use the overprovisioningRate parameter.
      
      Describes the resources that will not be held (consumed) by the standby holder.
      - spec.cloudInstances.standbyHolder.notHeldResources.cpu
        integer or string
        Describes the amount of CPU that will not be held by standby holder on Nodes from this NodeGroup.
        
        The value can be an absolute number of cpus (for example, 2) as well as a milli representation (for example, 1500m).
        
        Pattern: ^[0-9]+m?$
      - spec.cloudInstances.standbyHolder.notHeldResources.memory
        integer or string
        Describes the amount of memory that will not be held by standby holder on Nodes from this NodeGroup.
        
        The value can be an absolute number of bytes (for example, 128974848) as well as a fixed-point number using one of memory suffixes: G, Gi, M, Mi.
        
        Pattern: ^[0-9]+(\.[0-9]+)?(E|P|T|G|M|K|Ei|Pi|Ti|Gi|Mi|Ki)?$
    - spec.cloudInstances.standbyHolder.overprovisioningRate
      integer
      Percentage of reserved resources calculated from the capacity of a node of a NodeGroup.
      
      Default: 50
      Allowed values: 1 <= X <= 80
  - spec.cloudInstances.zones
    array of strings
    List of availability zones to create instances in.
    
    The default value depends on the cloud provider selected and usually corresponds to all zones of the region being used.
    
    Example:
    zones: - Helsinki - Espoo - Tampere
- spec.cri
  object
  Container runtime parameters.
  - spec.cri.containerd
    object
    Containerd runtime parameters.
    
    If used, cri.type must be set to Containerd.
    - spec.cri.containerd.maxConcurrentDownloads
      integer
      Set the max concurrent downloads for each pull.
      
      Default: 3
  - spec.cri.containerdV2
    object
    ContainerdV2 runtime parameters.
    
    If used, cri.type must be set to ContainerdV2.
    
    When using ContainerdV2, CgroupsV2 will be used (providing improved security and resource management).
    
    To use ContainerdV2 as a container runtime, the nodes in the group must meet the following requirements:
    
    Support for CgroupsV2.
    
    Linux kernel version 5.8 or newer.
    
    Systemd version 244 or newer.
    
    Support for erofs kernel module.
    - spec.cri.containerdV2.maxConcurrentDownloads
      integer
      Set the max concurrent downloads for each pull.
      
      Default: 3
  - spec.cri.docker
    Deprecated
    object
    Docker settings for nodes.
    - spec.cri.docker.manage
      boolean
      Enable Docker maintenance from bashible.
      
      Default: true
    - spec.cri.docker.maxConcurrentDownloads
      integer
      Set the max concurrent downloads for each pull.
      
      Default: 3
  - spec.cri.notManaged
    object
    Settings for not managed CRI for nodes.
    - spec.cri.notManaged.criSocketPath
      string
      Path to CRI socket.
  - spec.cri.type
    string
    Container runtime type.
    
    If not specified, the defaultCRI parameter value of ClusterConfiguration is used.
    
    Note. Docker support is deprecated.
    
    Note. The CSE edition supports containerd v2 only.
    
    Allowed values: Docker, Containerd, ContainerdV2, NotManaged
- spec.disruptions
  object
  Disruptions settings for nodes.
  
  Example:
  disruptions: approvalMode: Automatic automatic: drainBeforeApproval: false windows: - from: '06:00' to: '08:00' days: - Tue - Sun
  - spec.disruptions.approvalMode
    string
    The approval mode for disruptive updates:
    
    Manual — disable automatic disruption approval; the alert will be displayed if disruption is needed. Nodes are not drained automatically in Manual mode. If necessary, manually drain nodes before setting the update.node.deckhouse.io/disruption-approved= annotation. Caution. The master node group update mode must be Manual to avoid issues with draining.
    
    Automatic — automatically approve disruption-involving updates. Nodes could be drained automatically before reboot, as configured by the drainBeforeApproval option.
    
    RollingUpdate — in this mode, a new node with new settings will be created; then, the old node will be deleted. Available only for cloud nodes.
    
    If the RollingUpdate mode is not used, when updating, the node is first drained and then updated (rebooted) and put back into operation (uncordoned). Note that in this case, the cluster must have sufficient resources to accommodate the load while the node being updated is unavailable. In the RollingUpdate mode, the node is replaced by the updated node, i.e., an extra node appears in the cluster for the duration of the update. In cloud infrastructures, the RollingUpdate mode is convenient, for example, if there are no resources in the cluster to temporarily host the load from the node being updated.
    Default: Automatic
    Allowed values: Manual, Automatic, RollingUpdate
  - spec.disruptions.automatic
    object
    Additional parameters for the Automatic mode.
    - spec.disruptions.automatic.drainBeforeApproval
      boolean
      Drain Pods from the nodes before approving disruption.
      
      Caution. This setting ignores (nodes will be approved without draining Pods):
      
      for the nodeGroup master with a single node;
      
      for a single ready node in a nodeGroup picked out for Deckhouse placement.
      
      Default: true
    - spec.disruptions.automatic.windows
      array of objects
      Time windows for node disruptive updates.
      - spec.disruptions.automatic.windows.days
        array of strings
        Days of the week when node could be updated.
        
        Examples:
        days: Mon
        
        days: Wed
        
        Element of the array
        string
        Day of the week.
        
        Allowed values: Mon, Tue, Wed, Thu, Fri, Sat, Sun
      - spec.disruptions.automatic.windows.from
        string
        Required value
        Start time of disruptive update window (UTC timezone).
        
        Pattern: ^(?:\d|[01]\d|2[0-3]):[0-5]\d$
        Example:
        from: '13:00'
      - spec.disruptions.automatic.windows.to
        string
        Required value
        End time of disruptive update window (UTC timezone).
        
        Pattern: ^(?:\d|[01]\d|2[0-3]):[0-5]\d$
        Example:
        to: '18:30'
  - spec.disruptions.rollingUpdate
    object
    Additional parameters for the RollingUpdate mode.
    - spec.disruptions.rollingUpdate.windows
      array of objects
      Time windows for node disruptive updates.
      - spec.disruptions.rollingUpdate.windows.days
        array of strings
        Days of the week when node could be updated.
        
        Examples:
        days: Mon
        
        days: Wed
        
        Element of the array
        string
        Day of the week.
        
        Allowed values: Mon, Tue, Wed, Thu, Fri, Sat, Sun
      - spec.disruptions.rollingUpdate.windows.from
        string
        Required value
        Start time of disruptive update window (UTC timezone).
        
        Pattern: ^(?:\d|[01]\d|2[0-3]):[0-5]\d$
        Example:
        from: '13:00'
      - spec.disruptions.rollingUpdate.windows.to
        string
        Required value
        End time of disruptive update window (UTC timezone).
        
        Pattern: ^(?:\d|[01]\d|2[0-3]):[0-5]\d$
        Example:
        to: '18:30'
- spec.fencing
  object
  Enable the fencing-controller for a node group.
  - spec.fencing.mode
    string
    Required value
    Specifies the operating mode of the fencing-controller.
    
    Supported modes:
    
    Watchdog: Uses the watchdog kernel module to automatically reboot nodes that lose connectivity with the cluster. In this mode, the fencing-agent also monitors node availability in the cluster.
    
    When fencing is enabled, the NodeGroupConfiguration script loads the softdog kernel module with the soft_margin parameter set to the value of spec.fencing.watchdog.timeout (60 seconds by default) and soft_panic=1.
    
    How the fencing-agent works:
    
    The agent opens the /dev/watchdog device and sets the node-manager.deckhouse.io/fencing-enabled label on the node, as well as the node-manager.deckhouse.io/fencing-mode label with the value Watchdog.
    
    The fencing-agent joins a gossip cluster (memberlist) with agents from the same node group.
    
    The agent regularly checks quorum (availability of the majority of nodes in the group):
    
    If quorum is lost (quorum=false), the agent performs an additional check of Kubernetes API availability.
    
    If quorum is present or the Kubernetes API is reachable, the agent resets the watchdog timer.
    
    If neither quorum nor access to the Kubernetes API is available, the watchdog is no longer reset, the timer expires, and a kernel panic occurs.
    
    The agent respects maintenance annotations node-manager.deckhouse.io/fencing-disable, update.node.deckhouse.io/approved, and update.node.deckhouse.io/disruption-approved, temporarily disabling the watchdog during planned operations.
    
    When fencing is enabled, automatic node reboot after a kernel panic is disabled at the OS level. This prevents a node from returning in an undefined state before it is properly handled (manually for static nodes, or via the cloud-provider-controller for cloud nodes).
    
    The fencing-controller takes the node type (node.deckhouse.io/type) into account when handling a failed node:
    
    For cloud nodes (CloudEphemeral, CloudPermanent), the controller deletes the Node object, after which the cloud-provider-controller recreates the corresponding virtual machine.
    
    For static nodes (Static, CloudStatic), the controller only evicts workloads while keeping the Node object in the cluster.
    
    For details on how fencing handles different node types, see FAQ.
    Allowed values: Watchdog
  - spec.fencing.watchdog
    object
    - spec.fencing.watchdog.timeout
      string
      Required value
      Watchdog timeout duration. Should be specified in seconds, minutes or hours.
      
      Default: 60s
      Pattern: ^[0-9]+[smh]{1}$
- spec.gpu
  object
  Available in editions: Ultimate/EE
  GPU parameters.
  - spec.gpu.exclusive
    object
    Without GPU sharing.
  - spec.gpu.mig
    object
    MIG sharing strategy parameters.
    - spec.gpu.mig.customConfigs
      array of objects
      Custom MIG partitioning by GPU index.
      
      Used only when partedConfig: custom is set.
      
      Represents an array of devices (a map keyed by index), where each entry refers to a GPU by its index and describes the slices to be created.
      
      Examples:
      customConfigs: profile: 3g.40gb count: 2
      
      customConfigs: profile: 1g.10gb.me count: 1
      - spec.gpu.mig.customConfigs.index
        integer
        Required value
        GPU index on the node.
        
        Allowed values: 0 <= X
      - spec.gpu.mig.customConfigs.slices
        array of objects
        Required value
        List of MIG slices to create.
        
        spec.gpu.mig.customConfigs.slices.count
        integer
        Required value
        Number of slices in the selected profile.
        
        Default: 1
        Allowed values: 1 <= X <= 7
        
        spec.gpu.mig.customConfigs.slices.profile
        string
        Required value
        MIG profile name.
        
        Pattern: ^[0-9]+g\.[0-9]+gb(\.me)?$
        Examples:
        profile: 3g.40gb
        
        profile: 1g.10gb
        
        profile: 1g.10gb.me
    - spec.gpu.mig.partedConfig
      string
      MIG configuration name.
      
      Standard all-* profiles are supported (the same partitioning applied to all GPUs), as well as the custom value, which allows defining partitioning individually for each GPU.
      
      Default: all-1g.5gb
  - spec.gpu.sharing
    string
    GPU sharing strategy:
    
    Exclusive — no sharing (each GPU is fully allocated to a single Pod).
    
    TimeSlicing — time-sliced sharing (multiple Pods can share one GPU concurrently by splitting GPU time).
    
    MIG — use NVIDIA Multi-Instance GPU (MIG) to split each GPU into multiple instances (if supported by hardware).
    Allowed values: TimeSlicing, MIG, Exclusive
  - spec.gpu.timeSlicing
    object
    TimeSlicing sharing strategy parameters.
    - spec.gpu.timeSlicing.partitionCount
      number
      Count of partition per GPU.
      
      Default: 4
      Allowed values: 1 <= X <= 100
- spec.kubelet
  object
  Kubelet settings for nodes.
  - spec.kubelet.containerLogMaxFiles
    integer
    How many rotated log files to store before deleting them.
    
    Default: 4
    Allowed values: 1 <= X <= 20
  - spec.kubelet.containerLogMaxSize
    string
    Maximum log file size before it is rotated.
    
    Default: 50Mi
    Pattern: \d+[Ei|Pi|Ti|Gi|Mi|Ki|E|P|T|G|M|k|m]
  - spec.kubelet.maxPods
    integer
    Set the max count of pods per node for given NodeGroup. When left unspecified, the system automatically determines this limit according to the value of the podSubnetNodeCIDRPrefix ClusterConfiguration parameter:
    
    120 pods for podSubnetNodeCIDRPrefix ≥ 24;
    
    250 pods for podSubnetNodeCIDRPrefix = 23;
    
    500 pods for podSubnetNodeCIDRPrefix = 22;
    
    1000 pods for podSubnetNodeCIDRPrefix ≤ 21.
  - spec.kubelet.memorySwap
    object
    Configuration of swap memory usage for kubelet.
    - spec.kubelet.memorySwap.limitedSwap
      object
      Configuration for the limited swap mode. Required when swapBehavior is set to LimitedSwap.
      - spec.kubelet.memorySwap.limitedSwap.size
        string
        Required value
        Size of the swap file to create, specified in GB (for example, “2G”, “4G”).
        
        Pattern: ^\d+G$
        Example:
        size: 1G
    - spec.kubelet.memorySwap.swapBehavior
      string
      Defines how swap memory is handled on the node:
      
      NoSwap: Swap is disabled. This is the traditional Kubernetes behavior.
      
      LimitedSwap: Kubelet can use swap memory up to the limit specified in limitedSwap.size.
      
      For details on swap memory usage on nodes, refer to Kubernetes documentation.
      
      Allowed values: NoSwap, LimitedSwap
    - spec.kubelet.memorySwap.swappiness
      integer
      Defines the kernel’s tendency to use the swap.
      
      0: Minimal swap usage (only to avoid OOM).
      
      60: Default balanced behavior.
      
      100: Aggressive swap usage.
      
      This value is set via vm.swappiness sysctl parameter.
      
      Default: 60
      Allowed values: 0 <= X <= 100
  - spec.kubelet.resourceReservation
    object
    Management of resource reservation for system daemons on a node.
    
    More info in the Kubernetes documentation.
    - spec.kubelet.resourceReservation.mode
      string
      Defines the system resource reservation mode:
      
      Off — disables resource reservation.
      
      Auto — reserves resources based on the Node capacity.
      
      Static — reserves resources based on the values provided by the user via the static parameter.
      
      Note that currently we do not use a dedicated group cgroup for resource reservation (-system-reserved-cgroup is not used).
      
      Default: Auto
    - spec.kubelet.resourceReservation.static
      object
      Resource reservation parameters for the ‘Static’ mode.
      - spec.kubelet.resourceReservation.static.cpu
        integer or string
        Pattern: \d+[m]
      - spec.kubelet.resourceReservation.static.ephemeralStorage
        integer or string
        Pattern: \d+[Ei|Pi|Ti|Gi|Mi|Ki|E|P|T|G|M|k|m]
      - spec.kubelet.resourceReservation.static.memory
        integer or string
        Pattern: \d+[Ei|Pi|Ti|Gi|Mi|Ki|E|P|T|G|M|k|m]
  - spec.kubelet.rootDir
    string
    Directory path for managing kubelet files (volume mounts,etc).
    
    Default: /var/lib/kubelet
  - spec.kubelet.seccompDefault
    boolean
    Use RuntimeDefault seccomp profile for workloads that don’t specify seccomp profile explicitly.
  - spec.kubelet.topologyManager
    object
    Controls the kubelet resource managers on NUMA nodes: the CPU manager, the memory manager and the topology manager.
    
    With a policy other than None, containers of Guaranteed pods with an integer CPU request get exclusive CPUs, and the memory of Guaranteed pods is bound to NUMA nodes. With Restricted and SingleNumaNode, they get whole physical cores. Deckhouse calculates NUMA-dependent kubelet settings on each node of the group: it splits reservedMemory between NUMA nodes and sets max-allowable-numa-nodes on nodes with more than 8 NUMA nodes.
    
    A policy other than None requires resourceReservation.mode Auto, or Static with a positive static.cpu. Changing the settings restarts the kubelet and can require a disruption on nodes with Guaranteed pods. See How do I give pods dedicated CPUs and NUMA alignment?
    
    For more information, see the Kubernetes documentation.
    - spec.kubelet.topologyManager.enabled
      Deprecated
      boolean
      Deprecated, has no effect. The resource managers are controlled by policy alone.
    - spec.kubelet.topologyManager.policy
      string
      Resource alignment policy and the related kubelet settings:
      
      None: no alignment. The CPU and memory managers are not used.
      
      BestEffort: Guaranteed pods get exclusive CPUs and NUMA-bound memory. Resources are aligned to NUMA nodes when possible; a pod is admitted even if they cannot be aligned, but it is rejected if its memory cannot be placed.
      
      Restricted: Guaranteed pods get exclusive whole physical cores and NUMA-bound memory. A pod is rejected if its resources cannot be aligned; when a pod needs several NUMA nodes, the closest ones are preferred.
      
      SingleNumaNode: as Restricted, but all resources of a pod (or of a container, see scope) must come from one NUMA node.
      
      More information can be found in the Kubernetes documentation.
      
      Default: None
      Allowed values: None, BestEffort, Restricted, SingleNumaNode
    - spec.kubelet.topologyManager.scope
      string
      Defines granularity with which resources are aligned:
      
      Container: resources are aligned for each container separately.
      
      Pod: all containers of a pod are aligned together.
      
      Default: Pod
      Allowed values: Container, Pod
- spec.nodeDrainTimeoutSecond
  integer
  Maximum duration (in seconds) for which the controller attempts to reschedule pods to a new machine during node draining.
  
  When a machine is deleted, the Drain operation evicts the pods for this long. Then the machine is deleted anyway. In a Static or CloudStatic NodeGroup without this field, the Drain of a deleted machine has no time limit.
  
  The range is the one the module’s own settings have always enforced. A value outside it used to be accepted here and then rejected when the module read it, which stopped node-manager reconciling anything at all.
  
  Allowed values: 30 <= X <= 7200
- spec.nodeTemplate
  object
  Specification of some of the fields that will be maintained in all nodes of the group.
  
  Example:
  nodeTemplate: labels: environment: production app: warp-drive-ai annotations: ai.fleet.com/discombobulate: 'true' taints: - effect: NoExecute key: ship-class value: frigate
  - spec.nodeTemplate.annotations
    object
    Similar to the standard metadata.annotations field.
    
    Example:
    annotations: ai.fleet.com/discombobulate: 'true'
  - spec.nodeTemplate.labels
    object
    Similar to the standard metadata.labels field.
    
    Example:
    labels: environment: production app: warp-drive-ai
  - spec.nodeTemplate.taints
    array of objects
    Similar to the .spec.taints field of the Node object.
    
    Available fields: effect, key, and values.
    
    Example:
    taints: - effect: NoExecute key: ship-class value: frigate
    - spec.nodeTemplate.taints.effect
      string
      Allowed values: NoSchedule, PreferNoSchedule, NoExecute
    - spec.nodeTemplate.taints.key
      string
    - spec.nodeTemplate.taints.value
      string
- spec.nodeType
  string
  Required value
  The type of nodes this group provides:
  - CloudEphemeral — nodes for this group will be automatically created (and deleted) in the cloud of the specified cloud provider;
  - CloudPermanent — nodes from ProviderClusterConfiguration will be created via dhctl;
  - CloudStatic — a static node (created manually or using any external tools) hosted in the cloud integrated with one of the cloud providers. This node has the CSI running, and it is managed by the cloud-controller-manager: the Node object automatically gets the information about the zone and region based on the cloud data; if a node gets deleted from the cloud, its corresponding Node object will be deleted in Kubernetes;
  - Static — a static node hosted on a bare metal or virtual machine. The cloud-controller-manager does not manage the node even if one of the cloud providers is enabled.
  Allowed values: CloudEphemeral, CloudPermanent, CloudStatic, Static
- spec.operatingSystem
  object
  Operating System settings for nodes.
  - spec.operatingSystem.manageKernel
    Deprecated
    boolean
    This parameter has no effect. Earlier, it enabled kernel maintenance on behalf of bashible.
    
    Default: true
- spec.providerType
  string
  Cloud provider type of the node group. None or an empty value — for a static node group.
- spec.staticInstances
  object
  Parameter for provisioning static machines to the cluster.
  - spec.staticInstances.count
    integer
    Target number of StaticInstance resources (matching the labelSelector) that DP bootstraps and adds into the cluster.
    
    In a NodeGroup with Static or CloudStatic nodes, this parameter sets the expected number of nodes. DP uses this value for state control and automation purposes.
    
    Warning. Set this parameter explicitly. The default value is 0, which means DP won’t bootstrap any StaticInstance resource or add any node to the group unless this value is modified. To add static nodes, specify the required number here (greater than 0).
    
    Default: 0
    Allowed values: 0 <= X
  - spec.staticInstances.labelSelector
    object
    Label selector for choosing the StaticInstance resources the node group uses.
    
    If both matchLabels and matchExpressions are set, a StaticInstance resource must satisfy all the conditions at once.
    
    An empty selector ({}) chooses every StaticInstance resource. If the selector is not set, the node group chooses no StaticInstance resources.
    
    Warning. A selector with at least one condition cannot be modified or removed. To change it, create a new node group and move the static nodes to it.
    
    Adding a selector to a node group that already uses StaticInstance resources is possible only when the labels of all those resources satisfy the new selector. Otherwise the webhook rejects the change and lists the resources that do not match the selector in the error message.
    
    After the selector is added, existing static nodes are not recreated and keep their previous configuration, but the platform applies the selector to every new node. A StaticInstance resource that does not match the selector cannot be reused, so the node group ends up with fewer nodes than the count parameter specifies.
    
    To add a selector:
    
    Label all the StaticInstance resources of the node group with the d8 k label staticinstance <STATIC_INSTANCE> <KEY>=<VALUE> command.
    
    Set the selector in the spec.staticInstances parameter of the NodeGroup resource.
    - spec.staticInstances.labelSelector.matchExpressions
      array of objects
      A list of label selector requirements. The requirements are ANDed.
      
      A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.
      - spec.staticInstances.labelSelector.matchExpressions.key
        string
        key is the label key that the selector applies to.
      - spec.staticInstances.labelSelector.matchExpressions.operator
        string
        operator represents a key’s relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.
      - spec.staticInstances.labelSelector.matchExpressions.values
        array of strings
        values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch.
        
        Element of the array
        string
        Pattern: [a-z0-9]([-a-z0-9]*[a-z0-9])?
        Length: 1..63
    - spec.staticInstances.labelSelector.matchLabels
      object
      matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is “key”, the operator is “In”, and the values array contains only “value”. The requirements are ANDed.
- spec.systemType
  string
  How the nodes of the group are managed.
  - Mutable: bashible configures the node on the fly. An unset field means the same.
  - Immutable: The node agent brings the node to the state described in a NodeConfig object, without bashible. The machine must boot from a Deckhouse Engine image.
  Immutable is supported for:
  - Static: Machines must be added with d8 node add. staticInstances cannot be used.
  - CloudEphemeral: Machines must be created by the cloud provider through Cluster API (CAPI).
  - CloudPermanent: Machines are created by dhctl.
  Immutable is not supported for CloudStatic groups.
  
  A group can become Immutable only while bashible has configured none of its nodes. Immutable cannot be changed or removed.
  Allowed values: Mutable, Immutable
- spec.update
  object
  - spec.update.maxConcurrent
    integer or string
    Maximum number of concurrently updating nodes.
    
    Can be set as absolute count or as a percent of total nodes.
    
    Default: 1
    Pattern: ^[1-9][0-9]*%?$
status
object
- status.conditionSummary
  object
  - status.conditionSummary.ready
    string
    Status of the condition summary.
    
    Allowed values: True, False
  - status.conditionSummary.statusMessage
    string
    Status message about group handling.
- status.conditions
  array of objects
  - status.conditions.lastTransitionTime
    string
    Last time the condition transit from one status to another.
  - status.conditions.message
    string
    Human readable message indicating details about last transition.
  - status.conditions.status
    string
    Required value
    Status of the condition, one of True, False.
  - status.conditions.type
    string
    Required value
    Type of node group condition.
- status.deckhouse
  object
  - status.deckhouse.observed
    object
    Contains last timestamp when the resource change was noted by the operator and its checksum.
    - status.deckhouse.observed.checkSum
      string
      The checksum of the last observed resource.
    - status.deckhouse.observed.lastTimestamp
      string
      Last timestamp when a change of the resource was noted by the operator.
  - status.deckhouse.processed
    object
    Contains last timestamp when the resource was applied in the cluster by the operator and its checksum.
    - status.deckhouse.processed.checkSum
      string
      The checksum of the last applied resource.
    - status.deckhouse.processed.lastTimestamp
      string
      Last timestamp when the resource was applied in the cluster.
  - status.deckhouse.synced
    string
    True if last observed version of the resource was successfully applied in the cluster.
- status.desired
  integer
  Number of desired machines in the group.
- status.engine
  string
  Machinery used to manage nodes in this group.
  
  The field value indicates which component manages the nodes:
  - None — nodes in the group are not managed by Machine Controller Manager or CAPI.
  - MCM — nodes in the group are managed by Machine Controller Manager.
  - CAPI — nodes in the group are managed by Cluster API.
  Allowed values: None, MCM, CAPI
- status.error
  string
  Error message about possible problems with the group handling.
- status.instances
  integer
  Number of instances (in any state) in the group.
- status.kubernetesVersion
  string
  The current version of kubernetes on the nodes, or the version to which the nodes will be upgraded.
- status.lastMachineFailures
  array of objects
  - status.lastMachineFailures.lastOperation
    object
    - status.lastMachineFailures.lastOperation.description
      string
      Last operation’s description.
    - status.lastMachineFailures.lastOperation.lastUpdateTime
      string
      Timestamp of last status update for operation.
    - status.lastMachineFailures.lastOperation.state
      string
      Machine’s operation state.
    - status.lastMachineFailures.lastOperation.type
      string
      Type of operation.
  - status.lastMachineFailures.name
    string
    Machine’s name.
  - status.lastMachineFailures.ownerRef
    string
    Machine owner’s name.
  - status.lastMachineFailures.providerID
    string
    Machine’s ProviderID.
- status.max
  integer
  Maximum amount of instances in the group.
- status.min
  integer
  Minimal amount of instances in the group.
- status.nodes
  integer
  Number of Kubernetes nodes (in any state) in the group.
- status.ready
  integer
  Number of ready Kubernetes nodes in the group.
- status.standby
  integer
  Number of overprovisioned instances in the group.
- status.upToDate
  integer
  Number of up-to-date nodes in the group.

Describes the runtime parameters of the node group.

metadata
object
- metadata.name
  string
  Pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$
  Maximum length: 42
spec
object
Required value
- spec.chaos
  object
  Chaos monkey settings.
  
  Example:
  chaos: mode: DrainAndDelete period: 24h
  - spec.chaos.mode
    string
    The chaos monkey mode:
    
    DrainAndDelete — drains and deletes a node when triggered;
    
    Disabled — leaves this NodeGroup intact.
    Default: Disabled
    Allowed values: Disabled, DrainAndDelete
  - spec.chaos.period
    string
    The time interval to use for the chaos monkey (can be specified in the Go format).
    
    Default: 6h
    Pattern: ^[0-9]+[mh]{1}$
- spec.cloudInstances
  object
  Parameter for provisioning the cloud-based VMs.
  
  Caution. Can only be used together with nodeType: CloudEphemeral.
  - spec.cloudInstances.classReference
    object
    Required value
    The reference to the InstanceClass object. It is unique for each cloud-provider-* module.
    - spec.cloudInstances.classReference.kind
      string
      The kind of the InstanceClass object (e.g., OpenStackInstanceClass). Supported kinds are defined by the installed cloud provider modules and specified in their documentation.
    - spec.cloudInstances.classReference.name
      string
      The name of the required InstanceClass object (e.g., finland-medium).
  - spec.cloudInstances.maxPerZone
    integer
    Required value
    The maximum number of instances for the group in each zone.
    
    This value is used as the upper bound in cluster-autoscaler.
    
    Allowed values: 0 <= X
  - spec.cloudInstances.maxSurgePerZone
    integer
    The maximum number of instances the platform creates in each zone above the desired number while it replaces the instances of the group.
    
    Replacement happens when parameters that require recreating a virtual machine change, for example in the InstanceClass. The parameter does not limit the number of instances that the cluster autoscaler orders when it scales the group up. For details on what NodeGroup parameters affect, see NodeGroup parameters and their result.
    
    The value 0 is treated as if the parameter is not set, so the default value applies.
    
    Default: 1
    Allowed values: 0 <= X
  - spec.cloudInstances.maxUnavailablePerZone
    integer
    The maximum number of unavailable instances (during rollout) in the group in each zone.
    
    Default: 0
    Allowed values: 0 <= X
  - spec.cloudInstances.minPerZone
    integer
    Required value
    The minimum number of instances for the group in each zone.
    
    This value is used in the MachineDeployment object and as a lower bound in cluster-autoscaler.
    
    Allowed values: 0 <= X
  - spec.cloudInstances.standby
    integer or string
    The summary number of overprovisioned nodes for this NodeGroup in all zones.
    
    An overprovisioned node is a cluster node on which resources are reserved that are available at any time for scaling. The presence of such a node allows the cluster autoscaler not to wait for node initialization (which may take several minutes), but to immediately place a load on it.
    
    The value can be an absolute number (for example, 2) or a percentage of desired nodes (for example, 10%). If a percentage is specified, the absolute number is calculated based on the percentage of the maximum number of nodes (the maxPerZone parameter) rounded down, but not less than one.
    
    Pattern: ^[0-9]+%?$
  - spec.cloudInstances.standbyHolder
    object
    Amount of reserved resources.
    
    Used to determine whether to order overprovisioned nodes.
    - spec.cloudInstances.standbyHolder.notHeldResources
      object
      Describes the resources that will not be held (consumed) by the standby holder.
      - spec.cloudInstances.standbyHolder.notHeldResources.cpu
        integer or string
        Describes the amount of CPU that will not be held by standby holder on Nodes from this NodeGroup.
        
        The value can be an absolute number of cpus (for example, 2) as well as a milli representation (for example, 1500m).
        
        Pattern: ^[0-9]+m?$
      - spec.cloudInstances.standbyHolder.notHeldResources.memory
        integer or string
        Describes the amount of memory that will not be held by standby holder on Nodes from this NodeGroup.
        
        The value can be an absolute number of bytes (for example, 128974848) as well as a fixed-point number using one of memory suffixes: G, Gi, M, Mi.
        
        Pattern: ^[0-9]+(\.[0-9]+)?(E|P|T|G|M|K|Ei|Pi|Ti|Gi|Mi|Ki)?$
  - spec.cloudInstances.zones
    array of strings
    List of availability zones to create instances in.
    
    The default value depends on the cloud provider selected and usually corresponds to all zones of the region being used.
    
    Example:
    zones: - Helsinki - Espoo - Tampere
- spec.cri
  object
  Container runtime parameters.
  - spec.cri.containerd
    object
    Containerd runtime parameters.
    
    If used, cri.type must be set to Containerd.
    - spec.cri.containerd.maxConcurrentDownloads
      integer
      Set the max concurrent downloads for each pull.
      
      Default: 3
  - spec.cri.docker
    object
    Docker settings for nodes.
    
    Note. the Docker is deprecated.
    - spec.cri.docker.manage
      boolean
      Enable Docker maintenance from bashible.
      
      Default: true
    - spec.cri.docker.maxConcurrentDownloads
      integer
      Set the max concurrent downloads for each pull.
      
      Default: 3
  - spec.cri.notManaged
    object
    Settings for not managed CRI for nodes.
    - spec.cri.notManaged.criSocketPath
      string
      Path to CRI socket.
  - spec.cri.type
    string
    Container runtime type.
    
    Value defaultCRI from the initial cluster configration (cluster-configuration.yaml parameter from the d8-cluster-configuration secret in the kube-system namespace) is used if not specified.
    
    Note. Docker support is deprecated.
    
    Allowed values: Docker, Containerd, NotManaged
- spec.disruptions
  object
  Disruptions settings for nodes.
  
  Example:
  disruptions: approvalMode: Automatic automatic: drainBeforeApproval: false windows: - from: '06:00' to: '08:00' days: - Tue - Sun
  - spec.disruptions.approvalMode
    string
    The approval mode for disruptive updates:
    
    Manual — disable automatic disruption approval; the alert will be displayed if disruption is needed. Caution. The master node group update mode must be Manual to avoid issues with draining.
    
    Automatic — automatically approve disruption-involving updates.
    
    RollingUpdate — in this mode, a new node with new settings will be created; then, the old node will be deleted. Available only for cloud nodes.
    
    If the RollingUpdate mode is not used, when updating, the node is first drained and then updated (rebooted) and put back into operation (uncordoned). Note that in this case, the cluster must have sufficient resources to accommodate the load while the node being updated is unavailable. In the RollingUpdate mode, the node is replaced by the updated node, i.e., an extra node appears in the cluster for the duration of the update. In cloud infrastructures, the RollingUpdate mode is convenient, for example, if there are no resources in the cluster to temporarily host the load from the node being updated.
    Default: Automatic
    Allowed values: Manual, Automatic, RollingUpdate
  - spec.disruptions.automatic
    object
    Additional parameters for the Automatic mode.
    - spec.disruptions.automatic.drainBeforeApproval
      boolean
      Drain Pods from the nodes before approving disruption.
      
      Caution. This setting ignores (nodes will be approved without draining Pods):
      
      for the nodeGroup master with a single node;
      
      for a single ready node in a nodeGroup picked out for Deckhouse placement.
      
      Default: true
    - spec.disruptions.automatic.windows
      array of objects
      Time windows for node disruptive updates.
      - spec.disruptions.automatic.windows.days
        array of strings
        Days of the week when node could be updated.
        
        Examples:
        days: Mon
        
        days: Wed
        
        Element of the array
        string
        Day of the week.
        
        Allowed values: Mon, Tue, Wed, Thu, Fri, Sat, Sun
      - spec.disruptions.automatic.windows.from
        string
        Required value
        Start time of disruptive update window (UTC timezone).
        
        Pattern: ^(?:\d|[01]\d|2[0-3]):[0-5]\d$
        Example:
        from: '13:00'
      - spec.disruptions.automatic.windows.to
        string
        Required value
        End time of disruptive update window (UTC timezone).
        
        Pattern: ^(?:\d|[01]\d|2[0-3]):[0-5]\d$
        Example:
        to: '18:30'
  - spec.disruptions.rollingUpdate
    object
    Additional parameters for the RollingUpdate mode.
    - spec.disruptions.rollingUpdate.windows
      array of objects
      Time windows for node disruptive updates.
      - spec.disruptions.rollingUpdate.windows.days
        array of strings
        Days of the week when node could be updated.
        
        Examples:
        days: Mon
        
        days: Wed
        
        Element of the array
        string
        Day of the week.
        
        Allowed values: Mon, Tue, Wed, Thu, Fri, Sat, Sun
      - spec.disruptions.rollingUpdate.windows.from
        string
        Required value
        Start time of disruptive update window (UTC timezone).
        
        Pattern: ^(?:\d|[01]\d|2[0-3]):[0-5]\d$
        Example:
        from: '13:00'
      - spec.disruptions.rollingUpdate.windows.to
        string
        Required value
        End time of disruptive update window (UTC timezone).
        
        Pattern: ^(?:\d|[01]\d|2[0-3]):[0-5]\d$
        Example:
        to: '18:30'
- spec.kubelet
  object
  Kubelet settings for nodes.
  - spec.kubelet.containerLogMaxFiles
    integer
    How many rotated log files to store before deleting them.
    
    Warning. This parameter does nothing if CRI type is Docker.
    
    Default: 4
    Allowed values: 1 <= X <= 20
  - spec.kubelet.containerLogMaxSize
    string
    Maximum log file size before it is rotated.
    
    Warning. This parameter does nothing if CRI type is Docker.
    
    Default: 50Mi
    Pattern: \d+[Ei|Pi|Ti|Gi|Mi|Ki|E|P|T|G|M|k|m]
  - spec.kubelet.maxPods
    integer
    Set the max count of pods per node for given NodeGroup. When left unspecified, the system automatically determines this limit according to the value of the podSubnetNodeCIDRPrefix ClusterConfiguration parameter:
    
    120 pods for podSubnetNodeCIDRPrefix ≥ 24;
    
    250 pods for podSubnetNodeCIDRPrefix = 23;
    
    500 pods for podSubnetNodeCIDRPrefix = 22;
    
    1000 pods for podSubnetNodeCIDRPrefix ≤ 21.
  - spec.kubelet.rootDir
    string
    Directory path for managing kubelet files (volume mounts,etc).
    
    Default: /var/lib/kubelet
  - spec.kubelet.seccompDefault
    boolean
    Use RuntimeDefault seccomp profile for workloads that don’t specify seccomp profile explicitly.
- spec.nodeTemplate
  object
  Specification of some of the fields that will be maintained in all nodes of the group.
  
  Example:
  nodeTemplate: labels: environment: production app: warp-drive-ai annotations: ai.fleet.com/discombobulate: 'true' taints: - effect: NoExecute key: ship-class value: frigate
  - spec.nodeTemplate.annotations
    object
    Similar to the standard metadata.annotations field.
    
    Example:
    annotations: ai.fleet.com/discombobulate: 'true'
  - spec.nodeTemplate.labels
    object
    Similar to the standard metadata.labels field.
    
    Example:
    labels: environment: production app: warp-drive-ai
  - spec.nodeTemplate.taints
    array of objects
    Similar to the .spec.taints field of the Node object.
    
    Available fields: effect, key, and values.
    
    Example:
    taints: - effect: NoExecute key: ship-class value: frigate
    - spec.nodeTemplate.taints.effect
      string
      Allowed values: NoSchedule, PreferNoSchedule, NoExecute
    - spec.nodeTemplate.taints.key
      string
    - spec.nodeTemplate.taints.value
      string
- spec.nodeType
  string
  Required value
  The type of nodes this group provides.
  - Cloud — nodes for this group will be automatically created (and deleted) in the cloud of the specified cloud provider;
  - Static — a static node hosted on a bare metal or virtual machine. The cloud-controller-manager does not manage the node even of one of the cloud providers is enabled;
  - Hybrid — a static node (created manually or using any external tools) hosted in the cloud integrated with one of the cloud provider. This node has the CSI running, and it is managed by the cloud-controller-manager: the Node object automatically gets the information about the zone and region based on the cloud data; if a node gets deleted from the cloud, its corresponding Node object will be deleted in Kubernetes.
  Allowed values: Cloud, Static, Hybrid
- spec.operatingSystem
  object
  Operating System settings for nodes.
  - spec.operatingSystem.manageKernel
    boolean
    Enable kernel maintenance from bashible.
    
    Default: true
- spec.providerType
  string
  Cloud provider type of the node group. None or an empty value — for a static node group.
- spec.systemType
  string
  How the nodes of the group are managed.
  - Mutable: bashible configures the node on the fly. An unset field means the same.
  - Immutable: The node agent brings the node to the state described in a NodeConfig object, without bashible. The machine must boot from a Deckhouse Engine image.
  Immutable is supported for:
  - Static: Machines must be added with d8 node add.
  - Cloud: Machines must be created by the cloud provider through Cluster API (CAPI).
  - Hybrid: Machines must be created by dhctl (CloudPermanent in deckhouse.io/v1).
  Immutable is not supported for other Hybrid groups (CloudStatic in deckhouse.io/v1).
  
  A group can become Immutable only while bashible has configured none of its nodes. Immutable cannot be changed or removed.
  Allowed values: Mutable, Immutable
status
object
- status.conditionSummary
  object
  - status.conditionSummary.ready
    string
    Status of the condition summary.
    
    Allowed values: True, False
  - status.conditionSummary.statusMessage
    string
    Status message about group handling.
- status.desired
  integer
  Number of desired machines in the group.
- status.engine
  string
  Machinery used to manage nodes in this group.
  
  The field value indicates which component manages the nodes:
  - None — nodes in the group are not managed by MCM or CAPI.
  - MCM — nodes in the group are managed by machine-controller-manager.
  - CAPI — nodes in the group are managed by Cluster API.
  Allowed values: None, MCM, CAPI
- status.error
  string
  Error message about possible problems with the group handling.
- status.instances
  integer
  Number of instances (in any state) in the group.
- status.kubernetesVersion
  string
  The current version of kubernetes on the nodes, or the version to which the nodes will be upgraded.
- status.lastMachineFailures
  array of objects
  - status.lastMachineFailures.lastOperation
    object
    - status.lastMachineFailures.lastOperation.description
      string
      Last operation’s description.
    - status.lastMachineFailures.lastOperation.lastUpdateTime
      string
      Timestamp of last status update for operation.
    - status.lastMachineFailures.lastOperation.state
      string
      Machine’s operation state.
    - status.lastMachineFailures.lastOperation.type
      string
      Type of operation.
  - status.lastMachineFailures.name
    string
    Machine’s name.
  - status.lastMachineFailures.ownerRef
    string
    Machine owner’s name.
  - status.lastMachineFailures.providerID
    string
    Machine’s ProviderID.
- status.max
  integer
  Maximum amount of instances in the group.
- status.min
  integer
  Minimal amount of instances in the group.
- status.nodes
  integer
  Number of Kubernetes nodes (in any state) in the group.
- status.ready
  integer
  Number of ready Kubernetes nodes in the group.
- status.standby
  integer
  Number of overprovisioned instances in the group.
- status.upToDate
  integer
  Number of up-to-date nodes in the group.

Defines the runtime parameters of a node group.

metadata
object
- metadata.name
  string
  Pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$
  Maximum length: 42
spec
object
Required value
- spec.chaos
  object
  Chaos monkey settings.
  
  Example:
  chaos: mode: DrainAndDelete period: 24h
  - spec.chaos.mode
    string
    The chaos monkey mode:
    
    DrainAndDelete — drains and deletes a node when triggered;
    
    Disabled — leaves this NodeGroup intact.
    Default: Disabled
    Allowed values: Disabled, DrainAndDelete
  - spec.chaos.period
    string
    The time interval to use for the chaos monkey (can be specified in the Go format).
    
    Default: 6h
    Pattern: ^[0-9]+[mh]{1}$
- spec.cloudInstances
  object
  Parameter for provisioning the cloud-based VMs.
  
  Caution. Can only be used together with nodeType: CloudEphemeral.
  - spec.cloudInstances.classReference
    object
    Required value
    The reference to the InstanceClass object. It is unique for each cloud-provider-* module.
    - spec.cloudInstances.classReference.kind
      string
      The kind of the InstanceClass object (e.g., OpenStackInstanceClass). Supported kinds are defined by the installed cloud provider modules and specified in their documentation.
    - spec.cloudInstances.classReference.name
      string
      The name of the required InstanceClass object (e.g., finland-medium).
  - spec.cloudInstances.maxPerZone
    integer
    Required value
    The maximum number of instances for the group in each zone.
    
    This value is used as the upper bound in cluster-autoscaler.
    
    With a value of 0, you need to set capacity for some InstanceClass. Get more details in the description of the necessary InstanceClass.
    
    Allowed values: 0 <= X
  - spec.cloudInstances.maxSurgePerZone
    integer
    The maximum number of instances the platform creates in each zone above the desired number while it replaces the instances of the group.
    
    Replacement happens when parameters that require recreating a virtual machine change, for example in the InstanceClass. The parameter does not limit the number of instances that the cluster autoscaler orders when it scales the group up. For details on what NodeGroup parameters affect, see NodeGroup parameters and their result.
    
    The value 0 is treated as if the parameter is not set, so the default value applies.
    
    Default: 1
    Allowed values: 0 <= X
  - spec.cloudInstances.maxUnavailablePerZone
    integer
    The maximum number of unavailable instances (during rollout) in the group in each zone.
    
    Default: 0
    Allowed values: 0 <= X
  - spec.cloudInstances.minPerZone
    integer
    Required value
    The minimum number of instances for the group in each zone.
    
    This value is used in the MachineDeployment object and as a lower bound in cluster-autoscaler.
    
    Allowed values: 0 <= X
  - spec.cloudInstances.standby
    integer or string
    The summary number of overprovisioned nodes for this NodeGroup all zones.
    
    An overprovisioned node is a cluster node on which resources are reserved that are available at any time for scaling. The presence of such a node allows the cluster autoscaler not to wait for node initialization (which may take several minutes), but to immediately place a load on it.
    
    The value can be an absolute number (for example, 2) or a percentage of desired nodes (for example, 10%). If a percentage is specified, the absolute number is calculated based on the percentage of the maximum number of nodes (the maxPerZone parameter) rounded down, but not less than one.
    
    Pattern: ^[0-9]+%?$
  - spec.cloudInstances.standbyHolder
    object
    Amount of reserved resources.
    
    Used to determine whether to order overprovisioned nodes.
    - spec.cloudInstances.standbyHolder.notHeldResources
      object
      Describes the resources that will not be held (consumed) by the standby holder.
      - spec.cloudInstances.standbyHolder.notHeldResources.cpu
        integer or string
        Describes the amount of CPU that will not be held by standby holder on Nodes from this NodeGroup.
        
        The value can be an absolute number of cpus (for example, 2) as well as a milli representation (for example, 1500m).
        
        Pattern: ^[0-9]+m?$
      - spec.cloudInstances.standbyHolder.notHeldResources.memory
        integer or string
        Describes the amount of memory that will not be held by standby holder on Nodes from this NodeGroup.
        
        The value can be an absolute number of bytes (for example, 128974848) as well as a fixed-point number using one of memory suffixes: G, Gi, M, Mi.
        
        Pattern: ^[0-9]+(\.[0-9]+)?(E|P|T|G|M|K|Ei|Pi|Ti|Gi|Mi|Ki)?$
  - spec.cloudInstances.zones
    array of strings
    List of availability zones to create instances in.
    
    The default value depends on the cloud provider selected and usually corresponds to all zones of the region being used.
    
    Example:
    zones: - Helsinki - Espoo - Tampere
- spec.cri
  object
  Container runtime parameters.
  - spec.cri.containerd
    object
    Containerd runtime parameters.
    
    If used, cri.type must be set to Containerd.
    - spec.cri.containerd.maxConcurrentDownloads
      integer
      Set the max concurrent downloads for each pull.
      
      Default: 3
  - spec.cri.type
    string
    Container runtime type.
    
    If not specified, the defaultCRI parameter value of ClusterConfiguration is used.
    
    Note. Docker support is deprecated.
    
    Allowed values: Docker, Containerd, NotManaged
- spec.disruptions
  object
  Disruptions settings for nodes.
  
  Example:
  disruptions: approvalMode: Automatic automatic: drainBeforeApproval: false windows: - from: '06:00' to: '08:00' days: - Tue - Sun
  - spec.disruptions.approvalMode
    string
    The approval mode for disruptive updates:
    
    Manual — disable automatic disruption approval; the alert will be displayed if disruption is needed. Caution. The master node group update mode must be Manual to avoid issues with draining.
    
    Automatic — automatically approve disruption-involving updates.
    
    RollingUpdate — in this mode, a new node with new settings will be created; then, the old node will be deleted. Available only for cloud nodes.
    
    If the RollingUpdate mode is not used, when updating, the node is first drained and then updated (rebooted) and put back into operation (uncordoned). Note that in this case, the cluster must have sufficient resources to accommodate the load while the node being updated is unavailable. In the RollingUpdate mode, the node is replaced by the updated node, i.e., an extra node appears in the cluster for the duration of the update. In cloud infrastructures, the RollingUpdate mode is convenient, for example, if there are no resources in the cluster to temporarily host the load from the node being updated.
    Default: Automatic
    Allowed values: Manual, Automatic, RollingUpdate
  - spec.disruptions.automatic
    object
    Additional parameters for the Automatic mode.
    - spec.disruptions.automatic.drainBeforeApproval
      boolean
      Drain Pods from the nodes before approving disruption.
      
      Caution. This setting ignores (nodes will be approved without draining Pods):
      
      for the nodeGroup master with a single node;
      
      for a single ready node in a nodeGroup picked out for Deckhouse placement.
      
      Default: true
    - spec.disruptions.automatic.windows
      array of objects
      Time windows for node disruptive updates.
      - spec.disruptions.automatic.windows.days
        array of strings
        Days of the week when node could be updated.
        
        Examples:
        days: Mon
        
        days: Wed
        
        Element of the array
        string
        Day of the week.
        
        Allowed values: Mon, Tue, Wed, Thu, Fri, Sat, Sun
      - spec.disruptions.automatic.windows.from
        string
        Required value
        Start time of disruptive update window (UTC timezone).
        
        Pattern: ^(?:\d|[01]\d|2[0-3]):[0-5]\d$
        Example:
        from: '13:00'
      - spec.disruptions.automatic.windows.to
        string
        Required value
        End time of disruptive update window (UTC timezone).
        
        Pattern: ^(?:\d|[01]\d|2[0-3]):[0-5]\d$
        Example:
        to: '18:30'
  - spec.disruptions.rollingUpdate
    object
    Additional parameters for the RollingUpdate mode.
    - spec.disruptions.rollingUpdate.windows
      array of objects
      Time windows for node disruptive updates.
      - spec.disruptions.rollingUpdate.windows.days
        array of strings
        Days of the week when node could be updated.
        
        Examples:
        days: Mon
        
        days: Wed
        
        Element of the array
        string
        Day of the week.
        
        Allowed values: Mon, Tue, Wed, Thu, Fri, Sat, Sun
      - spec.disruptions.rollingUpdate.windows.from
        string
        Required value
        Start time of disruptive update window (UTC timezone).
        
        Pattern: ^(?:\d|[01]\d|2[0-3]):[0-5]\d$
        Example:
        from: '13:00'
      - spec.disruptions.rollingUpdate.windows.to
        string
        Required value
        End time of disruptive update window (UTC timezone).
        
        Pattern: ^(?:\d|[01]\d|2[0-3]):[0-5]\d$
        Example:
        to: '18:30'
- spec.docker
  object
  Docker settings for nodes.
  
  If used, cri.type must be set to Docker.
  
  Note. the Docker is deprecated.
  - spec.docker.manage
    boolean
    Enable Docker maintenance from bashible.
    
    Default: true
  - spec.docker.maxConcurrentDownloads
    integer
    Set the max concurrent downloads for each pull.
    
    Default: 3
- spec.kubelet
  object
  Kubelet settings for nodes.
  - spec.kubelet.containerLogMaxFiles
    integer
    How many rotated log files to store before deleting them.
    
    Warning. This parameter does nothing if CRI type is Docker.
    
    Default: 4
    Allowed values: 1 <= X <= 20
  - spec.kubelet.containerLogMaxSize
    string
    Maximum log file size before it is rotated.
    
    Warning. This parameter does nothing if CRI type is Docker.
    
    Default: 50Mi
    Pattern: \d+[Ei|Pi|Ti|Gi|Mi|Ki|E|P|T|G|M|k|m]
  - spec.kubelet.maxPods
    integer
    Set the max count of pods per node for given NodeGroup. When left unspecified, the system automatically determines this limit according to the value of the podSubnetNodeCIDRPrefix ClusterConfiguration parameter:
    
    120 pods for podSubnetNodeCIDRPrefix ≥ 24;
    
    250 pods for podSubnetNodeCIDRPrefix = 23;
    
    500 pods for podSubnetNodeCIDRPrefix = 22;
    
    1000 pods for podSubnetNodeCIDRPrefix ≤ 21.
  - spec.kubelet.rootDir
    string
    Directory path for managing kubelet files (volume mounts,etc).
    
    Default: /var/lib/kubelet
  - spec.kubelet.seccompDefault
    boolean
    Use RuntimeDefault seccomp profile for workloads that don’t specify seccomp profile explicitly.
- spec.kubernetesVersion
  string
  The desired minor version of Kubernetes.
  
  By default, it corresponds to the version selected for the cluster globally (see installation documentation) or to the current version of the control plane (if the global version is not defined).
  
  Allowed values: 1.33, 1.34, 1.35, 1.36, 1.37
  Example:
  kubernetesVersion: '1.33'
- spec.nodeTemplate
  object
  Specification of some of the fields that will be maintained in all nodes of the group.
  
  Example:
  nodeTemplate: labels: environment: production app: warp-drive-ai annotations: ai.fleet.com/discombobulate: 'true' taints: - effect: NoExecute key: ship-class value: frigate
  - spec.nodeTemplate.annotations
    object
    Similar to the standard metadata.annotations field.
    
    Example:
    annotations: ai.fleet.com/discombobulate: 'true'
  - spec.nodeTemplate.labels
    object
    Similar to the standard metadata.labels field.
    
    Example:
    labels: environment: production app: warp-drive-ai
  - spec.nodeTemplate.taints
    array of objects
    Similar to the .spec.taints field of the Node object.
    
    Available fields: effect, key, and values.
    
    Example:
    taints: - effect: NoExecute key: ship-class value: frigate
    - spec.nodeTemplate.taints.effect
      string
      Allowed values: NoSchedule, PreferNoSchedule, NoExecute
    - spec.nodeTemplate.taints.key
      string
    - spec.nodeTemplate.taints.value
      string
- spec.nodeType
  string
  Required value
  The type of nodes this group provides.
  - Cloud — nodes for this group will be automatically created (and deleted) in the cloud of the specified cloud provider;
  - Static — a static node hosted on a bare metal or virtual machine. The cloud-controller-manager does not manage the node even of one of the cloud providers is enabled;
  - Hybrid — a static node (created manually or using any external tools) hosted in the cloud integrated with one of the cloud provider. This node has the CSI running, and it is managed by the cloud-controller-manager: the Node object automatically gets the information about the zone and region based on the cloud data; if a node gets deleted from the cloud, its corresponding Node object will be deleted in Kubernetes.
  Allowed values: Cloud, Static, Hybrid
- spec.operatingSystem
  Deprecated
  object
  Operating System settings for nodes.
  - spec.operatingSystem.manageKernel
    Deprecated
    boolean
    Enable kernel maintenance from bashible.
    
    Default: true
- spec.providerType
  string
  Cloud provider type of the node group. None or an empty value — for a static node group.
- spec.static
  object
  Static node parameters
  - spec.static.internalNetworkCIDRs
    array of strings
    Subnet CIDR
- spec.systemType
  string
  How the nodes of the group are managed.
  - Mutable: bashible configures the node on the fly. An unset field means the same.
  - Immutable: The node agent brings the node to the state described in a NodeConfig object, without bashible. The machine must boot from a Deckhouse Engine image.
  Immutable is supported for:
  - Static: Machines must be added with d8 node add.
  - Cloud: Machines must be created by the cloud provider through Cluster API (CAPI).
  - Hybrid: Machines must be created by dhctl (CloudPermanent in deckhouse.io/v1).
  Immutable is not supported for other Hybrid groups (CloudStatic in deckhouse.io/v1).
  
  A group can become Immutable only while bashible has configured none of its nodes. Immutable cannot be changed or removed.
  Allowed values: Mutable, Immutable
status
object
- status.conditionSummary
  object
  - status.conditionSummary.ready
    string
    Status of the condition summary.
    
    Allowed values: True, False
  - status.conditionSummary.statusMessage
    string
    Status message about group handling.
- status.desired
  integer
  Number of desired machines in the group.
- status.engine
  string
  Machinery used to manage nodes in this group.
  
  The field value indicates which component manages the nodes:
  - None — nodes in the group are not managed by Machine Controller Manager or CAPI.
  - MCM — nodes in the group are managed by Machine Controller Manager.
  - CAPI — nodes in the group are managed by Cluster API.
  Allowed values: None, MCM, CAPI
- status.error
  string
  Error message about possible problems with the group handling.
- status.instances
  integer
  Number of instances (in any state) in the group.
- status.kubernetesVersion
  string
  The current version of kubernetes on the nodes, or the version to which the nodes will be upgraded.
- status.lastMachineFailures
  array of objects
  - status.lastMachineFailures.lastOperation
    object
    - status.lastMachineFailures.lastOperation.description
      string
      Last operation’s description.
    - status.lastMachineFailures.lastOperation.lastUpdateTime
      string
      Timestamp of last status update for operation.
    - status.lastMachineFailures.lastOperation.state
      string
      Machine’s operation state.
    - status.lastMachineFailures.lastOperation.type
      string
      Type of operation.
  - status.lastMachineFailures.name
    string
    Machine’s name.
  - status.lastMachineFailures.ownerRef
    string
    Machine owner’s name.
  - status.lastMachineFailures.providerID
    string
    Machine’s ProviderID.
- status.max
  integer
  Maximum amount of instances in the group.
- status.min
  integer
  Minimal amount of instances in the group.
- status.nodes
  integer
  Number of Kubernetes nodes (in any state) in the group.
- status.ready
  integer
  Number of ready Kubernetes nodes in the group.
- status.standby
  integer
  Number of overprovisioned instances in the group.
- status.upToDate
  integer
  Number of up-to-date nodes in the group.

NodeConfig

Short names: nc

Scope: Cluster
Version: v1alpha1

NodeConfig is the desired state of one node and the node agent’s report on it.

The node agent (nodelet) applies spec.

apiVersion
string
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
kind
string
Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata
object
spec
object
Required value
Desired state of the node.
- spec.apiServerEndpoints
  array of strings
  Kubernetes API server URLs with scheme and port, for example https://10.0.0.1:6443.
  - Element of the array
    string
    Pattern: ^https://(\[[0-9A-Fa-f:]+\]|[A-Za-z0-9]([-A-Za-z0-9]*[A-Za-z0-9])?([.][A-Za-z0-9]([-A-Za-z0-9]*[A-Za-z0-9])?)*):(6553[0-5]|655[0-2][0-9]|65[0-4][0-9]{2}|6[0-4][0-9]{3}|[1-5][0-9]{4}|[1-9][0-9]{0,3})/?$
    Maximum length: 253
- spec.containerRuntime
  object
  ContainerRuntime holds containerd configuration.
  - spec.containerRuntime.maxConcurrentDownloads
    integer
    Maximum number of image layers downloaded in parallel.
    
    0 means no limit. Defaults to 8 when unset.
    
    Allowed values: 0 <= X
  - spec.containerRuntime.registryConfigOwner
    string
    Component that writes the containerd registry.d directory.
    
    Nodelet: The node agent writes the configuration from spec.registry to the directory and removes everything else in it.
    
    RegistryAgent: The registry agent static pod owns the directory.
    
    Defaults to Nodelet when unset. Set by node-manager.
    Allowed values: Nodelet, RegistryAgent
  - spec.containerRuntime.sandboxImage
    string
    The pause image used for pod sandboxes.
    
    If not specified, the pause image imported by the containerd system extension is used.
    
    Pattern: ^[^[:space:]]+$
    Maximum length: 1024
- spec.extensions
  array of objects
  Extensions the node installs, each pinned by digest.
  
  On an Immutable node an entry is a signed dm-verity system extension image. On a Mutable node an entry is a registry package: the node agent runs the package’s own install script as root.
  
  An entry named nodelet updates the node agent itself.
  
  Extension is a signed verity sysext built from a release channel, fetched from the registry-packages-proxy by digest. Optional repository selects the proxy’s per-registry config; optional path is its “path” parameter.
  - spec.extensions.digest
    string
    Required value
    Digest is the sha256 digest of the sysext image.
    
    Pattern: ^sha256:[a-f0-9]{64}$
  - spec.extensions.name
    string
    Required value
    Name is the extension name (also the sysext image basename).
    
    Pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$
    Length: 1..63
  - spec.extensions.path
    string
    Path to the artifact inside the registry.
    
    Maximum length: 512
  - spec.extensions.repository
    string
    Repository optionally selects the proxy’s per-registry client config.
    
    Pattern: ^[^[:space:]]+$
    Maximum length: 1024
  - spec.extensions.requestedBy
    object
    Object that requested the extension.
    
    Unset for an extension of the platform itself.
    - spec.extensions.requestedBy.kind
      string
      Required value
      Kind of the object.
      
      Allowed values: NodeExtensionRequest, NodeStaticPodRequest
    - spec.extensions.requestedBy.name
      string
      Required value
      Name of the object.
      
      Pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$
      Length: 1..253
- spec.featureGates
  object
  Node agent features to switch on or off, keyed by feature name.
  
  A feature missing from this map is on for an Immutable node if the node type supports it, and off for a Mutable node. A Mutable node runs only the features switched on here.
  
  The node rejects a document that names an unknown feature or switches on a feature its node type does not support.
  
  Set by node-controller.
- spec.internalNetworkCIDRs
  array of strings
  InternalNetworkCIDRs are the subnets the cluster’s own traffic runs on. The node picks the interface holding an address inside one of them and registers with that address; a node with one NIC needs none of this. The pattern is StaticClusterConfiguration’s own (candi/openapi/static_cluster_configuration.yaml): IPv4 only.
  - Element of the array
    string
    Pattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(3[0-2]|[1-2][0-9]|[0-9]))$
    Maximum length: 43
- spec.kernel
  object
  Kernel holds sysctl settings and kernel modules to load.
  - spec.kernel.modules
    array of objects
    Kernel modules to load, in this order.
    
    Dependencies are not loaded automatically: list each module after the modules it depends on. Do not list a driver built into the kernel: KernelModulesLoaded becomes False.
    
    A module added to the list is loaded only after the disruption is approved.
    
    KernelModule is a module to load with optional parameters.
    - spec.kernel.modules.name
      string
      Required value
      Length: 1..64
    - spec.kernel.modules.params
      array of strings
      - Element of the array
        string
        Maximum length: 256
  - spec.kernel.sysctl
    object
- spec.kubelet
  object
  Kubelet holds kubelet configuration parameters.
  - spec.kubelet.bootstrapToken
    string
    BootstrapToken is the bootstrap token used by kubelet to obtain its client certificate on first boot.
    
    Maximum length: 4096
  - spec.kubelet.clusterCABundle
    string
    PEM bundle of the cluster CA.
    
    Maximum length: 65536
  - spec.kubelet.clusterDNS
    array of strings
    ClusterDNS is the list of DNS server IP addresses for the cluster.
    - Element of the array
      string
      Maximum length: 45
  - spec.kubelet.clusterDomain
    string
    ClusterDomain is the DNS domain for this cluster (e.g. “cluster.local”).
    
    Pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$
    Maximum length: 253
  - spec.kubelet.containerLogMaxFiles
    integer
    Number of rotated log files kept per container.
    
    Defaults to 4 when unset.
    
    Allowed values: 1 <= X <= 1000
  - spec.kubelet.containerLogMaxSize
    string
    Maximum size of a container log file before it is rotated.
    
    Defaults to 50Mi when unset.
    
    Maximum length: 64
  - spec.kubelet.externalCloudProvider
    boolean
    ExternalCloudProvider enables –cloud-provider=external so the cloud-controller-manager manages the node (zone, region, providerID).
  - spec.kubelet.kubernetesVersion
    string
    KubernetesVersion is the cluster’s minor version, e.g. “1.34”. It decides which feature gates kubelet is started with (bashible turns DRA gates on by version); without it DRA workloads cannot run on immutable nodes.
    
    Pattern: ^[0-9]+\.[0-9]+$
    Maximum length: 16
  - spec.kubelet.maxPods
    integer
    Maximum number of pods on the node.
    
    Defaults to 120 when unset.
    
    Allowed values: 1 <= X <= 1000
  - spec.kubelet.nodeLabels
    object
    NodeLabels are labels added to the node object when kubelet registers. A NodeGroup caps neither their number nor their shape (crds/node_group.yaml:1915), so neither does this.
  - spec.kubelet.registerWithTaints
    array of objects
    RegisterWithTaints is a list of taints to add to the node object when kubelet registers itself. Only takes effect on initial registration.
    
    Taint represents a Kubernetes taint applied to a node during registration. Key, value and effect take what a NodeGroup takes (crds/node_group.yaml:1944): bare strings and an optional effect, so a taint copied from one still fits.
    - spec.kubelet.registerWithTaints.effect
      string
      Allowed values: NoSchedule, PreferNoSchedule, NoExecute
    - spec.kubelet.registerWithTaints.key
      string
      Required value
    - spec.kubelet.registerWithTaints.value
      string
  - spec.kubelet.resourceReservation
    object
    ResourceReservation controls how much CPU, memory and disk are held back from pods for the system itself (kubeReserved).
    - spec.kubelet.resourceReservation.mode
      string
      Reservation policy.
      
      Auto: Computed from the node’s capacity.
      
      Off: Nothing is reserved.
      
      Defaults to Auto when unset.
      
      Allowed values: Auto, Off
  - spec.kubelet.serverTLSBootstrap
    boolean
    ServerTLSBootstrap makes kubelet request a serving certificate from the cluster instead of signing its own. Default true. The zero master sets it to false: nothing approves serving CSRs until Deckhouse is installed.
- spec.network
  object
  Network holds hostname, DNS, NTP, interfaces and routes.
  - spec.network.dns
    object
    DNS resolver configuration.
    - spec.network.dns.search
      array of strings
      - Element of the array
        string
        Maximum length: 253
    - spec.network.dns.servers
      array of strings
      Name server IP addresses.
      - Element of the array
        string
        Maximum length: 45
  - spec.network.hostname
    string
    Host name of the node.
    
    Maximum length: 253
  - spec.network.interfaces
    array of objects
    Network interfaces, keyed by name.
    
    NetworkInterface describes a single NIC.
    - spec.network.interfaces.addresses
      array of strings
      Static addresses in CIDR notation, used when dhcp is false.
      - Element of the array
        string
        Maximum length: 43
    - spec.network.interfaces.cluster
      boolean
      Cluster names this NIC the one the cluster’s traffic runs on, for a machine whose addressing spec.internalNetworkCIDRs cannot tell apart.
    - spec.network.interfaces.dhcp
      boolean
      Required value
      DHCP enables DHCPv4 on the interface.
    - spec.network.interfaces.gateway
      string
      Default gateway IP address.
      
      Maximum length: 45
    - spec.network.interfaces.name
      string
      Required value
      Length: 1..253
  - spec.network.ntp
    object
    NTP time-sync configuration.
    - spec.network.ntp.servers
      array of strings
      - Element of the array
        string
        Maximum length: 253
  - spec.network.routes
    array of objects
    - spec.network.routes.gateway
      string
      Gateway IP address.
      
      Maximum length: 45
    - spec.network.routes.name
      string
      Maximum length: 253
    - spec.network.routes.networks
      array of strings
      Destination networks in CIDR notation.
      - Element of the array
        string
        Maximum length: 43
- spec.nodeName
  string
  Required value
  Name of the Kubernetes node this configuration applies to.
  
  Equals metadata.name and cannot be changed.
  
  Pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$
  Length: 1..253
- spec.osImage
  object
  Operating system image to use on the node.
  
  To determine whether an update is required, the node compares the image digest with the value saved to /config/os-digest during installation. The digest identifies a specific image publication and may change when the same content is republished.
  
  Required when systemType is Immutable.
  - spec.osImage.digest
    string
    Required value
    Operating system image digest.
    
    Identifies a specific image and is used by the node to determine whether an update is required. During installation, the node saves the digest and compares it with the specified value during subsequent checks.
    
    Pattern: ^sha256:[a-f0-9]{64}$
  - spec.osImage.path
    string
    Path to the image inside the registry.
    
    Maximum length: 512
  - spec.osImage.repository
    string
    Address of the container registry from which the operating system image is downloaded.
    
    The value is passed to registry-packages-proxy as the repository parameter.
    
    If the parameter is not specified, repository is not passed and registry-packages-proxy uses its default container registry.
    
    Pattern: ^[^[:space:]]+$
    Maximum length: 1024
  - spec.osImage.rootHash
    string
    Dm-verity root hash of the root filesystem inside the image, in ASCII hexadecimal format.
    
    Used to determine whether the actual image content has changed. Unlike digest, which may change when an image is republished, rootHash remains the same if the root filesystem content has not changed.
    
    Deckhouse Platform obtains the value from an image label. If the value cannot be obtained from the label, the node determines the root hash after downloading the image.
    
    Pattern: ^[a-f0-9]{64}$
- spec.registry
  object
  Registry is the container registry the node talks to directly, without the registry-packages-proxy. A node bootstrapping a control plane has no proxy yet but must pull images and sysexts; workers leave this empty.
  - spec.registry.address
    string
    Required value
    Address is the registry host, optionally with a port, e.g. “registry.deckhouse.io” or “registry.example.com:5000”.
    
    Pattern: ^[^[:space:]]+$
    Length: 1..1024
  - spec.registry.auth
    string
    Auth is the base64-encoded “user:password” pair, as it appears in the “auth” field of a docker config.
    
    Maximum length: 4096
  - spec.registry.caBundle
    string
    PEM bundle of the CA certificates the registry is verified with.
    
    Maximum length: 65536
  - spec.registry.path
    string
    Path is the repository path within the registry, e.g. “/deckhouse/ce”.
    
    Maximum length: 1024
  - spec.registry.scheme
    string
    Protocol the node uses to connect to the registry.
    
    Defaults to HTTPS when unset.
    
    Allowed values: HTTPS, HTTP
- spec.registryPackagesProxy
  object
  How the node reaches registry-packages-proxy, which serves it extensions and operating system images.
  - spec.registryPackagesProxy.accessToken
    string
    Bearer token the node presents to the proxy.
    
    Base64-encoded in the object.
    
    Maximum length: 4096
  - spec.registryPackagesProxy.endpoints
    array of strings
    registry-packages-proxy URLs with scheme and port.
    
    When unset, the node uses port 4219 of the apiServerEndpoints hosts. Set only in the initial configuration of the first master node.
    - Element of the array
      string
      Pattern: ^https?://(\[[0-9A-Fa-f:]+\]|[A-Za-z0-9]([-A-Za-z0-9]*[A-Za-z0-9])?([.][A-Za-z0-9]([-A-Za-z0-9]*[A-Za-z0-9])?)*):(6553[0-5]|655[0-2][0-9]|65[0-4][0-9]{2}|6[0-4][0-9]{3}|[1-5][0-9]{4}|[1-9][0-9]{0,3})/?$
      Maximum length: 253
- spec.staticPods
  array of objects
  Manifests of static pods that kubelet runs without using the Kubernetes scheduler.
  
  The node writes the manifests to /etc/kubernetes/manifests. Static pods must obtain any other required resources on their own. For example, a pod can issue its own certificates, use the node’s kubeconfig, and mount directories for persistent state using hostPath with the DirectoryOrCreate type.
  
  Control plane component manifests created by the node agent are not included in spec.staticPods and are not modified when this parameter is processed.
  
  Static pod.
  - spec.staticPods.manifest
    string
    Required value
    Complete pod manifest.
    
    When the manifest is written to disk, the $MY_IP substring is replaced with the node’s IP address, similarly to control plane component manifests.
    
    Length: 1..32768
  - spec.staticPods.name
    string
    Required value
    Manifest file name without an extension.
    
    Used as the list item key and is independent of metadata.name of the pod in the manifest. Kubelet determines the mirror pod name from the manifest rather than the file name.
    
    Pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$
    Length: 1..63
  - spec.staticPods.requestedBy
    object
    Object that requested the static pod.
    
    Unset for a static pod of the platform itself.
    - spec.staticPods.requestedBy.kind
      string
      Required value
      Kind of the object.
      
      Allowed values: NodeExtensionRequest, NodeStaticPodRequest
    - spec.staticPods.requestedBy.name
      string
      Required value
      Name of the object.
      
      Pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$
      Length: 1..253
- spec.statusToken
  string
  StatusToken is the bearer presented to the node’s :50000 status port. Whoever writes the document mints it (dhctl, or the NodeConfigTemplate render for a hand-installed machine); the cluster carries it over untouched, having no way to tell the node a new one. Deliberately not marked sensitive: the apiserver would answer “" and the carry-over would write that back, destroying the token.
  
  Maximum length: 4096
- spec.storage
  object
  Disk the operating system is installed on, and additional filesystems.
  
  The installation disk cannot be changed once set.
  - spec.storage.device
    string
    Device is the whole-disk block device to install onto, e.g. “/dev/sda”, “/dev/nvme0n1”, or a stable “/dev/disk/by-id/…” path. Ignored when diskSelector is set.
    
    Pattern: ^/dev/[A-Za-z0-9._/-]+$
    Maximum length: 1024
  - spec.storage.diskSelector
    object
    Selects the installation disk by attributes instead of a fixed path.
    
    All specified fields must match.
    - spec.storage.diskSelector.busPath
      string
      BusPath matches the hardware bus path (glob).
      
      Maximum length: 253
    - spec.storage.diskSelector.model
      string
      Model matches the device model (glob), e.g. “Samsung*”.
      
      Maximum length: 253
    - spec.storage.diskSelector.name
      string
      Name matches the kernel device name (glob), e.g. “nvme0n1”.
      
      Maximum length: 253
    - spec.storage.diskSelector.rotational
      boolean
      Rotational matches spinning (true) vs solid-state (false) disks.
    - spec.storage.diskSelector.serial
      string
      Serial matches the disk serial number (glob).
      
      Maximum length: 253
    - spec.storage.diskSelector.size
      string
      Size matches the disk capacity, optionally with a comparison operator, e.g. “>=100Gi”, “>1Ti”, “512Gi”.
      
      Maximum length: 253
    - spec.storage.diskSelector.type
      string
      Type matches the disk kind.
      
      Allowed values: SSD, HDD, NVMe, SD
    - spec.storage.diskSelector.wwid
      string
      WWID matches the World Wide Identifier (glob).
      
      Maximum length: 253
  - spec.storage.mounts
    array of objects
    Mounts are additional filesystems the node makes available, at /mnt/ or wherever bindTo names. Nothing here is partitioned: only existing partitions and blank whole disks are formatted (when empty) and mounted.
    
    Mount is one additional filesystem, at /mnt/ unless bindTo says otherwise. Exactly one of device or partitionSelector names the partition; an empty one is formatted (label = name), an existing filesystem mounted as is.
    - spec.storage.mounts.bindTo
      string
      BindTo mounts the filesystem at this path instead of /mnt/, for a directory something else knows by name (/var/lib/etcd). The directory must be empty: mounting over files hides them, and hidden etcd data is lost data.
      
      Pattern: ^/[A-Za-z0-9._/-]+$
      Maximum length: 1024
    - spec.storage.mounts.device
      string
      Device is the partition to use, e.g. “/dev/sdb1” or a stable “/dev/disk/by-id/…-part1” path.
      
      Pattern: ^/dev/[A-Za-z0-9._/-]+$
      Maximum length: 1024
    - spec.storage.mounts.filesystem
      string
      Filesystem to create on an empty partition.
      
      Defaults to Ext4 when unset.
      
      Allowed values: Ext4
    - spec.storage.mounts.mode
      string
      Mode is the mode of the filesystem root after mounting, as an octal string, e.g. “0700”. Left alone when unset. A freshly made ext4 has its root at 0755, which is a mode etcd refuses to start on.
      
      Pattern: ^0[0-7]{3}$
    - spec.storage.mounts.name
      string
      Required value
      Name identifies the mount, and is both the mount point (/mnt/) and the filesystem label written when this node formats the partition. Capped at 16 characters because that is the size of the ext4 volume label field.
      
      Pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$
      Length: 1..16
    - spec.storage.mounts.partitionSelector
      object
      PartitionSelector picks the partition by attributes instead of a fixed path.
      - spec.storage.mounts.partitionSelector.blank
        boolean
        Blank makes whole disks selectable, and only ones that carry nothing: no partition table, no filesystem — a cloud disk never touched. Without it a selector sees partitions only; a whole disk is where somebody’s layout lives.
      - spec.storage.mounts.partitionSelector.fsType
        string
        FSType matches the type of the filesystem already present (glob).
        
        Maximum length: 253
      - spec.storage.mounts.partitionSelector.label
        string
        Label matches the filesystem label (glob).
        
        Maximum length: 253
      - spec.storage.mounts.partitionSelector.name
        string
        Name matches the kernel device name (glob), e.g. “sdb1” or “nvme0n1p*”.
        
        Maximum length: 253
      - spec.storage.mounts.partitionSelector.partLabel
        string
        PartLabel matches the GPT partition name (glob).
        
        Maximum length: 253
      - spec.storage.mounts.partitionSelector.partUUID
        string
        PartUUID matches the GPT partition UUID exactly, ignoring case.
        
        Maximum length: 253
      - spec.storage.mounts.partitionSelector.size
        string
        Size matches the size, optionally with a comparison operator, e.g. “>=100Gi”, “>1Ti”, “512Gi”. Without an operator the comparison is “>=”; “=” allows 1%, since a disk rarely reports an exact round size.
        
        Maximum length: 253
      - spec.storage.mounts.partitionSelector.uuid
        string
        UUID matches the filesystem UUID exactly, ignoring case.
        
        Maximum length: 253
  - spec.storage.wipe
    boolean
    Wipe controls whether an already-provisioned disk is wiped and re-partitioned. Default false: only an unprovisioned (or non-matching) disk is set up, so an existing layout stays and a reboot never destroys data.
- spec.systemType
  string
  Required value
  How the node is managed.
  - Immutable: a Deckhouse Engine node. The node agent configures the whole node.
  - Mutable: a node configured by bashible. The node agent manages only what featureGates turn on.
  Cannot be changed.
  Allowed values: Immutable, Mutable
- spec.updatePolicy
  object
  UpdatePolicy controls how and when the node is updated.
  - spec.updatePolicy.mode
    string
    Mode is the update mode.
    
    Allowed values: Manual, Automatic
  - spec.updatePolicy.windows
    array of objects
    Time windows in UTC when the node may download and apply updates.
    
    An update is allowed if any window matches. Without windows, updates are allowed at any time.
    
    One update window, in UTC.
    - spec.updatePolicy.windows.days
      array of strings
      Days of the week the window applies to.
      
      When empty, the window applies every day.
      - Element of the array
        string
        Allowed values: Mon, Tue, Wed, Thu, Fri, Sat, Sun
    - spec.updatePolicy.windows.from
      string
      Required value
      Start of the window, H:MM or HH:MM (24-hour, UTC).
      
      Pattern: ^(?:\d|[01]\d|2[0-3]):[0-5]\d$
    - spec.updatePolicy.windows.to
      string
      Required value
      End of the window, same format as from.
      
      A window whose end is before its start runs over midnight.
      
      Pattern: ^(?:\d|[01]\d|2[0-3]):[0-5]\d$
status
object
Status is reported by the on-node agent after each reconcile pass.
- status.agent
  object
  Node agent build that wrote this status.
  - status.agent.version
    string
    Build version of the node agent.
- status.appliedGeneration
  integer
  AppliedGeneration is the spec generation the node has applied.
  
  The node has applied the current spec when AppliedGeneration equals metadata.generation and ConfigurationApplied is True at that generation. The value stays behind while a disruptive change waits for approval, and while a new agent build installed for this generation has not applied it yet.
- status.conditions
  array of objects
  Conditions of the node.
  
  A False condition whose reason ends with Pending is a wait. Any other False reason and Unknown mean a failure. AddressResolved does not affect phase.
  
  In Ready, ConfigurationApplied and DisruptionApproved, observedGeneration is the generation the node has processed. In the other conditions, it is the generation the node runs.
  
  Condition types:
  - Ready: True with reason ReconcileSucceeded when phase is Ready. Otherwise False, with the phase as the reason. The message names the condition that determined the phase.
  - ConfigurationApplied: True when the node runs the published configuration.
    
    With True, the reason is ReconcileSucceeded, or UnknownFields when the object has fields unknown to this agent build. The message lists them.
    
    Reasons with False that mean a wait:
    
    DisruptionApprovalPending
    
    Pending
    
    RestartPending
    
    Reasons with False that mean a failure:
    
    Rejected
    
    RolledBackToLastKnownGood
    
    RollbackFailed
    
    PartiallyApplied
    
    FieldsNotApplicable
  - DisruptionApproved: Whether the interruption that the configuration of the node needs is approved.
    
    Reasons:
    
    ApprovalPending (False): A configuration that interrupts the node waits for approval.
    
    AttemptsExhausted (False): The interruptions node-manager attempted failed, and it makes no new attempt for now.
    
    Approved (True): The interruption is approved.
    
    NoDisruptionPending (True): Nothing waits for approval.
  - AddressResolved: Whether the node has determined the address it registers with.
    
    Reasons with False:
    
    NoneInCIDR, Ambiguous: The node cannot choose an address.
    
    AddressDrift: The node keeps the address it registered with, but that address is no longer on the machine or the configuration now selects another one.
  - Subsystem conditions, one per subsystem the pass ran or skipped:
    
    APIEndpointsReachable
    
    SysctlApplied
    
    KernelModulesLoaded
    
    MountsReady
    
    ExtensionsReady
    
    ContainerdReady
    
    KubeletReady
    
    StaticPodsReady
    
    UnitsRunning
    
    OSImageUpToDate
    
    ControlPlaneBootstrapped
    
    Values of a subsystem condition:
    
    True: The reason is ReconcileSucceeded.
    
    False: The message gives the cause. The reason is Pending while the subsystem waits for something outside the node, otherwise ReconcileFailed or a more specific reason of the subsystem.
    
    Unknown: The subsystem did not run in this pass. The reason is ReconcileSkipped.
  Condition contains details for one aspect of the current state of this API Resource.
  - status.conditions.lastTransitionTime
    string
    Required value
    lastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
  - status.conditions.message
    string
    Required value
    message is a human readable message indicating details about the transition. This may be an empty string.
    
    Maximum length: 32768
  - status.conditions.observedGeneration
    integer
    observedGeneration represents the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance.
    
    Allowed values: 0 <= X
  - status.conditions.reason
    string
    Required value
    reason contains a programmatic identifier indicating the reason for the condition’s last transition. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API. The value should be a CamelCase string. This field may not be empty.
    
    Pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
    Length: 1..1024
  - status.conditions.status
    string
    Required value
    status of the condition, one of True, False, Unknown.
    
    Allowed values: True, False, Unknown
  - status.conditions.type
    string
    Required value
    type of condition in CamelCase or in foo.example.com/CamelCase.
    
    Pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
    Maximum length: 316
- status.extensions
  array of objects
  Extensions is the outcome of each configured system extension, one entry per extension. Republished on every pass, empty when the pass checked nothing — never “the previous outcome still holds”.
  
  ExtensionStatus is the reconcile outcome of one system extension.
  - status.extensions.digest
    string
    Digest depends on State.
    
    Ready: the digest installed on the node.
    
    Pending: the digest being installed. For the agent’s own entry, the digest installed and waiting for the new build to start.
    
    Failed: the digest the node refused or could not install.
  - status.extensions.message
    string
    Message carries the cause when State is Failed.
  - status.extensions.name
    string
    Required value
    Name is the extension name: a spec.extensions[].name, or the name of an extension removed from spec whose uninstall failed on a Mutable node.
  - status.extensions.state
    string
    Required value
    State of the extension.
    
    Ready: the extension is installed (on an Immutable node, also merged).
    
    Pending: the extension is being fetched or waits for the update window, or the agent restarts with a new build of itself (reason RestartPending).
    
    Failed: the cause is in Message.
    Allowed values: Ready, Pending, Failed
- status.features
  array of strings
  Features that ran on the node during its last reconciliation, sorted by name: the gates switched on in spec.featureGates plus the defaults of the node type.
  
  On a Deckhouse Engine node spec.featureGates is usually empty because unset gates are on there. This list shows what actually runs on the node.
- status.lastReconcileTime
  string
  LastReconcileTime is when the node last finished a reconcile pass — the only thing in the status that ages, so it is what tells a dead agent from a healthy one. Republished coarsely: judge staleness in tens of minutes.
- status.maintenanceToken
  string
  MaintenanceToken is the bearer token an operator presents to the node’s :50000 config-push port once the node has lost the API. Root-equivalent, so the apiserver keeps it out of every answer without the sensitive subresource, which nodes are not granted.
- status.network
  object
  Network is what the node resolved for itself: the interface it took its cluster address from, and that address. Written by the node alone.
  - status.network.address
    string
    Address is the address kubelet registered the node with.
  - status.network.clusterInterface
    string
    ClusterInterface is the NIC the address was taken from.
- status.observedGeneration
  integer
  ObservedGeneration is the latest spec generation the node has processed: it reaches the newest generation as soon as the node has looked at it, even while it is still held for approval (see AppliedGeneration).
- status.osImage
  object
  Status of the operating system image on the node. Filled in by the node.
  - status.osImage.attemptsLeft
    integer
    Number of remaining boot attempts with the image being verified before initramfs automatically rolls back to the previous image.
  - status.osImage.digest
    string
    Digest of the operating system image currently used by the node. The value is read from the node configuration partition.
    
    May be absent on nodes installed before the digest persistence mechanism was introduced.
  - status.osImage.failedDigest
    string
    Digest of an image that caused the node to roll back after booting. No further attempts are made to boot this image.
    
    Because a digest uniquely identifies immutable image content, a fixed version of the image has a different digest.
  - status.osImage.rootHash
    string
    Dm-verity root hash of the image currently used by the node. The value is read from the active A/B slot.
    
    Used to identify the actual root filesystem content independently of the image digest. For example, the digests of the desired and currently used images may differ after republishing, while identical rootHash values indicate that the root filesystem content has not changed.
  - status.osImage.slot
    string
    A/B slot (a or b) containing the image currently used by the node.
  - status.osImage.trialDigest
    string
    Digest of the image prepared for the next boot or being verified during the current boot.
    
    Set only during an update, while digest still contains the digest of the previous image.
- status.phase
  string
  Summary of the node.
  - Ready: The node runs the published configuration and every subsystem is healthy.
  - Pending: The node is healthy but does not run the published configuration yet.
  - Degraded: At least one condition other than AddressResolved reports a failure.
  Allowed values: Ready, Pending, Degraded
- status.staticPods
  array of objects
  Results of processing static pods. A separate entry is created for each pod.
  
  Value is regenerated on each reconciliation and remains empty if no static pods were checked during the reconciliation.
  
  Processing result for a single static pod.
  - status.staticPods.manifestDigest
    string
    sha256 of the manifest as published in spec, in the form sha256:<hex>.
    
    Set when state is Written.
  - status.staticPods.message
    string
    Detailed information about the error when state is Failed.
  - status.staticPods.name
    string
    Required value
    Static pod name corresponding to spec.staticPods[].name.
  - status.staticPods.reason
    string
    Reason for the Failed state.
    
    Possible values:
    
    ManifestRejected: Node rejected the manifest. The manifest in the resource must be corrected.
    
    WriteFailed: Manifest file could not be written to the node. Check the node state.
    
    RemoveFailed: Static pod was removed from spec, but the corresponding manifest file could not be removed from the node. The pod continues to run.
    Allowed values: ManifestRejected, WriteFailed, RemoveFailed
  - status.staticPods.state
    string
    Required value
    Manifest processing state.
    
    Possible values:
    
    Written: File on disk matches the manifest from spec.
    
    Failed: File does not match the manifest from spec, or the required operation could not be completed. The reason is specified in reason.
    Allowed values: Written, Failed
- status.units
  array of objects
  Units is the outcome of each managed systemd unit (containerd, kubelet and every unit an extension ships), one entry per unit. Republished on every pass like Extensions, and empty when the pass checked nothing.
  
  UnitStatus is the reconcile outcome of one managed systemd unit.
  - status.units.message
    string
    Message carries the cause when State is Failed.
  - status.units.name
    string
    Required value
    Name is the systemd unit name (e.g. containerd.service).
  - status.units.state
    string
    Required value
    State is Active when the unit is running, Pending when it is queued to be started later this pass, or Failed with the cause in Message.
    
    Allowed values: Active, Pending, Failed

NodeExtensionRequest

Short names: ner

Scope: Cluster
Version: v1alpha1

Request to install an extension on the selected nodes: a system extension (a sysext image, optionally with kernel modules) or a registry package.

The requester sets the kind in spec.extension.kind. A registry package (Package) is installed on the selected nodes that bashible configures. A system extension (SystemExtension) is installed on the selected Deckhouse Engine nodes. The kind is never inferred from the image. If none of the selected groups installs extensions of this kind, the Accepted condition reports the NotDeliverable reason.

The image is addressed the same way as any package in the registry-packages-proxy: a name, a digest, and an optional repository and path. node-controller copies these fields unchanged into spec.extensions of the NodeConfig, and the node agent installs the extension. node-controller does not look up the image by module or ModuleSource.

Example:

# A system extension with kernel modules, fetched from the registry of a ModuleSource.
apiVersion: deckhouse.io/v1alpha1
kind: NodeExtensionRequest
metadata:
  name: drbd
spec:
  extension:
    name: drbd
    digest: sha256:4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945
    kind: SystemExtension
    repository: registry.example.com/deckhouse/modules
    path: drbd/extensions
    kernelModules:
      - name: drbd
        params:
          - usermode_helper=disabled
      - name: drbd_transport_tcp
  nodeGroupSelector:
    matchNames:
      - worker
---
# A registry package from the cluster's main registry, on the nodes of every
# group bashible configures that carry the label.
apiVersion: deckhouse.io/v1alpha1
kind: NodeExtensionRequest
metadata:
  name: nvme-cli
spec:
  extension:
    name: nvme-cli
    digest: sha256:9b2a6f2bb7e3c5f6d1a0e4c8b7d6e5f4a3b2c1d0e9f8a7b6c5d4e3f2a1b0c9d8
    kind: Package
  nodeSelector:
    matchLabels:
      storage.example.com/nvme: "true"

apiVersion
string
kind
string
metadata
object
spec
object
Required value
- spec.extension
  object
  Required value
  Image to install and its kind. The image is addressed the same way as any package in the registry-packages-proxy.
  - spec.extension.digest
    string
    Required value
    Image manifest digest.
    
    Pattern: ^sha256:[a-f0-9]{64}$
  - spec.extension.kernelModules
    array of objects
    Kernel modules the node agent loads after merging the system extension, in the listed order. Allowed only with the SystemExtension kind. Each module is listed once.
    
    Dependencies are not loaded automatically: list every module the extension needs, each one after the modules it depends on.
    
    Adding a module interrupts each selected node: the node waits for an approved Disruption NodeOperation.
    - spec.extension.kernelModules.name
      string
      Required value
      Kernel module name.
      
      Pattern: ^[a-zA-Z0-9][a-zA-Z0-9_-]*$
      Length: 1..64
    - spec.extension.kernelModules.params
      array of strings
      Module parameters, passed as key=value strings.
      - Element of the array
        string
        Maximum length: 256
  - spec.extension.kind
    string
    Required value
    Image kind.
    
    Package: a registry package. The node agent installs it on the selected nodes that bashible configures.
    
    SystemExtension: a system extension. The node agent merges it on the selected Deckhouse Engine nodes.
    
    Nodes of groups of the other type do not receive the image. If none of the selected groups installs extensions of this kind, the Accepted condition reports the NotDeliverable reason.
    
    Cannot be changed after creation. To change the kind, delete the request and create it again.
    Allowed values: Package, SystemExtension
  - spec.extension.name
    string
    Required value
    Extension name.
    
    The image of a system extension must contain the extension-release file for this name.
    
    The names containerd, kubelet, kubernetes-cni, nodelet and qemu-guest-agent belong to platform extensions and are not accepted.
    
    Pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$
    Length: 1..63
  - spec.extension.path
    string
    The image’s path within the repository (the proxy’s --rpp-path). Left empty, the repository root is used.
    
    Maximum length: 512
  - spec.extension.repository
    string
    The key the registry-packages-proxy resolves credentials by (its --rpp-repository), for example a ModuleSource’s spec.registry.repo. Left empty, the image is pulled from the cluster’s main registry.
    
    Pattern: ^[^[:space:]]+$
    Maximum length: 1024
- spec.nodeGroupSelector
  object
  Narrows the extension to nodes of the named NodeGroups.
  - spec.nodeGroupSelector.matchNames
    array of strings
    The set of NodeGroup names the extension applies to.
    - Element of the array
      string
      Pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$
      Maximum length: 63
- spec.nodeSelector
  object
  Narrows the extension to nodes carrying the given labels.
  - spec.nodeSelector.matchLabels
    object
    The set of node labels a node must carry to be selected.
status
object
- status.appliedNodes
  integer
  Number of selected nodes that report the extension installed at the digest of this request.
  
  Allowed values: 0 <= X
- status.conditions
  array of objects
  Conditions of the request: Accepted and Ready.
  
  Accepted shows whether node-manager accepted the request. It is True with the Accepted reason or False with one of these reasons:
  - InvalidExtension: The extension has no name or no digest.
  - ReservedName: The name belongs to a platform extension.
  - Conflict: An older request of the same kind already uses this name or digest, or the platform installs a registry package with this name or digest on the same nodes.
  - KernelModuleConflict: An older request already loads one of the kernel modules with different parameters.
  - NotDeliverable: None of the selected NodeGroups installs extensions of this kind.
  Ready shows whether the selected nodes run the request’s digest:
  - True with the AllNodesApplied reason: Every selected node runs it.
  - True with the NoNodesSelected reason: The request selects no node. The selected NodeGroups do not exist, have no nodes, or have no nodes that match nodeSelector.
  - False with the RolloutInProgress reason: Some selected nodes do not run it yet.
  - False with the RefusedByNodes reason: At least one node refused the extension.
  - False with the NotAccepted reason: The Accepted condition is False.
  - status.conditions.lastTransitionTime
    string
    Required value
  - status.conditions.message
    string
    Required value
    Maximum length: 32768
  - status.conditions.observedGeneration
    integer
    Allowed values: 0 <= X
  - status.conditions.reason
    string
    Required value
    Pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
    Length: 1..1024
  - status.conditions.status
    string
    Required value
    Allowed values: True, False, Unknown
  - status.conditions.type
    string
    Required value
    Pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
    Maximum length: 316
- status.failedNodes
  integer
  Number of selected nodes that refused the extension at the request’s digest.
  
  Allowed values: 0 <= X
- status.matchedNodeGroups
  array of strings
  The selected NodeGroups whose nodes install this kind of image: Deckhouse Engine groups for a system extension, groups bashible manages for a registry package.
- status.matchedNodes
  integer
  Number of nodes in matchedNodeGroups that the selectors pick.
  
  Allowed values: 0 <= X
- status.nodes
  array of objects
  Selected nodes that have not applied the request, at most 32.
  - status.nodes.message
    string
    Why the node refused the request or what it waits for.
    
    Maximum length: 4096
  - status.nodes.name
    string
    Required value
    Node name.
  - status.nodes.state
    string
    Required value
    Failed if the node refused the request or Pending if it has not answered yet.
    
    Allowed values: Pending, Failed
- status.observedGeneration
  integer
  The generation of the spec this status reflects.
  
  Allowed values: 0 <= X
- status.pendingNodes
  integer
  Number of selected nodes that have reported neither the extension installed at the request’s digest nor a refusal yet.
  
  Allowed values: 0 <= X
- status.phase
  string
  Summary of the conditions.
  - Pending: The selected nodes are still installing the extension (the Ready condition has the RolloutInProgress reason).
  - Ready: The Ready condition is True.
  - Degraded: node-manager did not accept the request (the Accepted condition is False), or nodes refused the extension (the Ready condition has the RefusedByNodes reason).
  Allowed values: Pending, Ready, Degraded

NodeGroupConfiguration

Short names: ngc

Scope: Cluster
Version: v1alpha1

Executes bash scripts on nodes.

NodeOperation

Short names: nop

Scope: Cluster
Version: v1alpha1

Describes an operation on a node that interrupts its work.

The spec of an existing operation cannot be changed, except .spec.approved. To approve a Disruption operation, run:

d8 k patch nodeoperation <OPERATION_NAME> --type=merge -p '{"spec":{"approved":true}}'

In the command, replace <OPERATION_NAME> with the name of the operation.

Operations for a deleted Cluster API Machine carry the label node-manager.deckhouse.io/machine-uid. Only node-manager can set it.

Example:

# Reboot a node.
apiVersion: deckhouse.io/v1alpha1
kind: NodeOperation
metadata:
  name: reboot-worker-1
spec:
  type: Reboot
  nodeName: worker-1

apiVersion
string
kind
string
metadata
object
spec
object
Required value
- spec.approved
  boolean
  Whether the interruption may go ahead. If the field is not set, the operation is not approved.
  
  Only meaningful for Disruption. Reboot and Drain ignore it.
  
  In the Manual approval mode of the NodeGroup (spec.disruptions.approvalMode), node-manager sets it to false, and an operator sets it to true. In the other modes, node-manager sets it to true on creation. An approval cannot be withdrawn.
- spec.cause
  string
  Why the node is interrupted.
  
  For a Disruption, node-manager sets it on creation. For Reboot and Drain, an operator may set it.
  
  Maximum length: 1024
- spec.configGeneration
  integer
  Revision of the node’s NodeConfig this disruption is for (.metadata.generation of the NodeConfig object).
  
  Allowed only for Disruption. The API server rejects the field for other types. If the NodeConfig moves to a newer generation before the operation is approved, node-manager deletes the operation.
  
  Allowed values: 1 <= X
- spec.drain
  object
  How to evict the workload before the node is interrupted.
  
  Allowed for Reboot and Disruption. The API server rejects the field for Drain.
  - spec.drain.skip
    boolean
    Interrupt the node without evicting its workload first.
    
    When node-manager creates a Disruption, it sets the field to true for these nodes:
    
    The single node of the master NodeGroup
    
    The node of the Deckhouse pod, while fewer than two nodes of its group are Ready
    
    Every node of a NodeGroup with spec.disruptions.automatic.drainBeforeApproval: false
    Default: false
- spec.nodeConfigUID
  string
  NodeConfig object this disruption is for (.metadata.uid of the node’s NodeConfig).
  
  Only meaningful for Disruption. Other types ignore it.
  
  Maximum length: 36
- spec.nodeName
  string
  Required value
  The name of the node the operation applies to.
  
  Pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$
  Length: 1..253
- spec.type
  string
  Required value
  Operation to perform on the node.
  - Reboot: Reboot the node. Only the nodes of a NodeGroup with systemType: Immutable perform this operation. A node that bashible manages refuses it, and the operation fails with the reason NodeRefused.
  - Drain: Evict the workload from the node. The node stays unschedulable until an operator makes it schedulable again. node-manager also creates a Drain for the node of a deleted Cluster API Machine. .spec.cause names the machine.
    
    To cancel a running Drain, delete it. node-manager withdraws its eviction request from the node before the object is gone. The cordon the eviction placed stays: make the node schedulable again yourself.
  - Disruption: Interrupt the node’s work so that the node can apply a new configuration.
  node-manager creates Disruption operations. Creating one by hand is refused. Reboot and Drain are meant for an operator.
  Allowed values: Reboot, Drain, Disruption
status
object
- status.bootID
  string
  Boot ID of the node (status.nodeInfo.bootID of the Node object) when the reboot started.
  
  Maximum length: 64
- status.conditions
  array of objects
  Latest details of the operation’s progress:
  - Drained: Written by node-manager. While the workload is being evicted, the condition is False with the reason Draining. Once the workload has left, it is True with the reason Drained. When the operation skips the eviction, it is True with the reason DrainSkipped.
  - EvictionBlocked: Written by node-manager while evictions of the workload are refused, for example by a PodDisruptionBudget. The condition is True with the reason EvictionRefused, and its message lists the refusals as the eviction reports them, one per pod. The same text is in the operation’s message. The condition is removed once the refusals stop or the eviction ends. A finished operation keeps the refusals it ended on.
  - NodeReported: Written by the node. The condition is True with the reason Succeeded, Released, Failed, Refused or RebootStarted.
  - Retry: Written by node-manager on a failed Disruption. The condition says whether node-manager makes another attempt for the same NodeConfig revision.
  Reasons of the Retry condition:
  - Created (True): The next attempt exists, and message names it.
  - Scheduled (True): node-manager creates the next attempt at the time given in message.
  - AttemptsExhausted (False): 3 attempts for the same revision have failed. node-manager makes no new attempt until the NodeConfig changes or the oldest failed attempt is deleted.
  - Superseded (False): The NodeConfig has moved to another generation or has been recreated.
  - NotRequested (False): The node no longer asks to be interrupted for this revision.
  - status.conditions.lastTransitionTime
    string
    Required value
  - status.conditions.message
    string
    Required value
    Maximum length: 32768
  - status.conditions.observedGeneration
    integer
    Allowed values: 0 <= X
  - status.conditions.reason
    string
    Required value
    Pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
    Length: 1..1024
  - status.conditions.status
    string
    Required value
    Allowed values: True, False, Unknown
  - status.conditions.type
    string
    Required value
    Pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
    Maximum length: 316
- status.deadline
  string
  Time when the current stage of the operation times out and the operation fails.
  
  The node gets 30 minutes from startedAt to report the result or to come back after a reboot. The eviction gets the drain timeout of the NodeGroup (nodeDrainTimeoutSecond) plus 30 minutes.
- status.drainDeadline
  string
  When the eviction this operation asked for runs out of the time it was given.
  
  On the Drain of a deleted machine, the operation fails with PreparationTimedOut at this moment, and the machine is deleted with the pods that did not leave. The field is not set for a Static or CloudStatic NodeGroup without nodeDrainTimeoutSecond, and such a Drain has no time limit.
  
  It is pinned at the moment the eviction is requested, from the NodeGroup’s nodeDrainTimeoutSecond as it stood then, because that is the bound the draining controller itself started running to. Deriving it again later would let a change to the NodeGroup cut short a drain that is still running.
- status.finishedAt
  string
  When the operation reached a terminal phase. A finished operation is kept as the record of what was done to the node and is collected once it is a day old.
- status.message
  string
  Human-readable explanation of reason.
  
  Maximum length: 32768
- status.nodeWasUnschedulable
  boolean
  Whether the node was already out of the scheduler when the operation reached it.
  
  Releasing the node at the end of the operation restores this value instead of making the node schedulable, so an operation does not quietly undo a cordon an operator put there by hand.
- status.phase
  string
  Current phase of the operation, set by node-manager.
  - Pending: The operation is queued or waits for approval.
  - Approved: The operation may go ahead. The workload is being evicted from the node before the node receives the operation.
  - InProgress: For Drain, the workload is being evicted. For the other types, the node has been prepared and can perform the operation.
  - Completed: The operation finished successfully.
  - Failed: The operation failed.
  Allowed values: Pending, Approved, InProgress, Completed, Failed
- status.reason
  string
  Reason for the current phase of the operation.
  - Pending: Queued (nothing has been done yet), WaitingForApproval.
  - Approved: Draining (the workload is being evicted from the node), WaitingForOperation (another eviction of the node is running, and message names who requested it).
  - InProgress: Draining and WaitingForOperation (for Drain), WaitingForNode (the node has received the operation and has not reported yet), Rebooting (the node is rebooting and has not come back yet).
  - Completed: Succeeded, Released (the node gave up on the NodeConfig revision the Disruption is for), Drained (for Drain).
  - Failed: NodeFailed and NodeRefused (reported by the node), PreparationTimedOut (the workload did not leave the node in time), NodeTimedOut (the node did not report back or did not come back after a reboot in time), NodeNotFound, NodeGroupMissing, MachineDeleted (the machine of the node is being deleted).
  Pattern: ^[A-Za-z][A-Za-z0-9]*$
  Maximum length: 128
- status.startedAt
  string
  When the node was handed the operation. The wait for the node is measured from here.

NodeStaticPodRequest

Short names: nspr

Scope: Cluster
Version: v1alpha1

Request to run a static pod on the selected nodes without using the Kubernetes scheduler.

node-manager adds the manifest to spec.staticPods of the NodeConfig of each selected node. The node writes the manifest to disk, and kubelet runs the pod from it. The request reports whether the nodes wrote the manifest. It does not report whether the pod runs.

A user with permission to create NodeStaticPodRequests can run arbitrary manifests with root privileges on all selected nodes.

Example:

# A node-local agent on the nodes of the worker NodeGroup.
apiVersion: deckhouse.io/v1alpha1
kind: NodeStaticPodRequest
metadata:
  name: node-local-agent
spec:
  nodeGroupSelector:
    matchNames:
      - worker
  manifest: |
    apiVersion: v1
    kind: Pod
    metadata:
      name: node-local-agent
      namespace: kube-system
    spec:
      hostNetwork: true
      priorityClassName: system-node-critical
      containers:
        - name: agent
          image: deckhouse.local/images:node-local-agent
          args:
            - --listen-address=$MY_IP:5002
          env:
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          volumeMounts:
            - name: state
              mountPath: /var/lib/node-local-agent
      volumes:
        - name: state
          hostPath:
            path: /var/lib/node-local-agent
            type: DirectoryOrCreate

apiVersion
string
kind
string
metadata
object
spec
object
Required value
- spec.manifest
  string
  Required value
  Complete pod manifest.
  
  Requirements for the manifest:
  - metadata.name is a DNS subdomain (RFC 1123), metadata.namespace is a DNS label (RFC 1123).
  - The namespace is kube-system or starts with d8-.
  - spec.nodeName is not set.
  - The pod does not reference API objects. Not allowed: spec.serviceAccountName, volumes of the secret, configMap, persistentVolumeClaim and projected types, envFrom and env[].valueFrom with secretKeyRef or configMapKeyRef in containers and init containers.
  A manifest that does not meet these requirements is rejected on creation and update.
  
  When the manifest is written to disk, the $MY_IP substring is replaced with the node’s IP address.
  
  Maximum manifest size is 32 KiB. A node can run no more than 16 static pods defined by NodeStaticPodRequests.
  Length: 1..32768
- spec.nodeGroupSelector
  object
  Restricts the pod to nodes in the specified NodeGroups. If no groups are specified, all NodeGroups are selected.
  
  The node agent writes the manifest from the NodeConfig of each selected node. On nodes managed by bashible, a resource named registry-agent or describing the kube-system/registry-agent pod is not applied, because bashible writes that manifest itself. If the resource reaches no other node, the Ready condition reports the ServedByBashible reason.
  - spec.nodeGroupSelector.matchNames
    array of strings
    Names of NodeGroups on whose nodes the pod must run.
    - Element of the array
      string
      Pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?$
      Maximum length: 63
status
object
- status.appliedNodes
  integer
  Number of selected nodes that reported the current manifest written.
  
  Allowed values: 0 <= X
- status.conditions
  array of objects
  Conditions of the request: Accepted and Ready.
  
  Accepted shows whether node-manager accepted the request. It is True with the Accepted reason or False with one of these reasons:
  - InvalidName: The request name is not a DNS label (RFC 1123) of at most 63 characters.
  - ReservedName: The request name, or the metadata.namespace and metadata.name of the pod, is already used by a static pod of the platform.
  - InvalidManifest: The manifest does not meet the requirements of spec.manifest.
  - Conflict: An older request already defines a pod with the same metadata.namespace and metadata.name.
  - LimitExceeded: At least one selected NodeGroup already has 16 static pods from older requests. Only the nodes of the other selected NodeGroups get the manifest.
  Ready shows whether the selected nodes wrote the current manifest:
  - True with the AllNodesApplied reason: Every selected node wrote it.
  - True with the NoNodesSelected reason: The selector matches no existing NodeGroup, or the selected NodeGroups have no nodes.
  - True with the ServedByBashible reason: The selected NodeGroups are managed by bashible, which writes the registry-agent manifest on their nodes itself.
  - False with the RolloutInProgress reason: Some selected nodes have not written the current manifest yet.
  - False with the RefusedByNodes reason: At least one node refused the manifest.
  - False with the NotAccepted reason: The Accepted condition is False.
  - status.conditions.lastTransitionTime
    string
    Required value
  - status.conditions.message
    string
    Required value
    Maximum length: 32768
  - status.conditions.observedGeneration
    integer
    Allowed values: 0 <= X
  - status.conditions.reason
    string
    Required value
    Pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
    Length: 1..1024
  - status.conditions.status
    string
    Required value
    Allowed values: True, False, Unknown
  - status.conditions.type
    string
    Required value
    Pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
    Maximum length: 316
- status.failedNodes
  integer
  Number of selected nodes that refused the manifest.
  
  Allowed values: 0 <= X
- status.matchedNodeGroups
  array of strings
  NodeGroups that match the current selector.
- status.matchedNodes
  integer
  Number of nodes in the selected NodeGroups to which the node agent writes the manifest. Nodes of NodeGroups on which bashible writes the manifest itself are not counted.
  
  Allowed values: 0 <= X
- status.nodes
  array of objects
  Selected nodes that have not written the current manifest, at most 32.
  - status.nodes.message
    string
    Why the node refused the manifest or what it waits for.
    
    Maximum length: 4096
  - status.nodes.name
    string
    Required value
    Node name.
  - status.nodes.state
    string
    Required value
    Failed if the node refused the manifest or Pending if it has not answered yet.
    
    Allowed values: Pending, Failed
- status.observedGeneration
  integer
  Generation of spec that corresponds to the current status.
  
  Allowed values: 0 <= X
- status.pendingNodes
  integer
  Number of selected nodes that have not yet reported either the current manifest written or a failure.
  
  Allowed values: 0 <= X
- status.phase
  string
  Summary of the conditions.
  - Pending: The selected nodes are still writing the manifest (the Ready condition has the RolloutInProgress reason).
  - Ready: The Ready condition is True.
  - Degraded: node-manager did not accept the request (the Accepted condition is False), or nodes refused the manifest (the Ready condition has the RefusedByNodes reason).
  Allowed values: Pending, Ready, Degraded

NodeUser

Scope: Cluster

v1
v1alpha1

Defines the linux users to create on all nodes.

The user’s home directory is created in the /home/deckhouse/ directory.

spec
object
Required value
- spec.extraGroups
  array of strings
  Node user additional system groups.
  
  Examples:
  extraGroups: - docker
  extraGroups: - docker - ftp
- spec.isSudoer
  boolean
  Persistence of node user in sudo group.
  
  Default: false
  Example:
  isSudoer: true
- spec.nodeGroups
  array of strings
  List of NodeGroups to apply the user for.
  
  Default: ["*"]
  Examples:
  nodeGroups: - master - worker
  nodeGroups: - worker
  nodeGroups: - "*"
- spec.passwordHash
  string
  Hashed user password.
  
  The format corresponds to the password hashes in /etc/shadow. You can get it using the following command: openssl passwd -6.
  
  Example:
  passwordHash: "$2a$10$F9ey7zW.sVliT224RFxpWeMsgzO.D9YRG54a8T36/K2MCiT41nzmC"
- spec.sshPublicKey
  Deprecated
  string
  Node user SSH public key.
  
  Either sshPublicKey or sshPublicKeys must be specified.
  
  Example:
  sshPublicKey: ssh-rsa AAABBB
- spec.sshPublicKeys
  array of strings
  Node user SSH public keys.
  
  Either sshPublicKey or sshPublicKeys must be specified.
  
  Example:
  sshPublicKeys: - ssh-rsa AAABBB - cert-authority,principals="name" ssh-rsa BBBCCC
- spec.uid
  number
  Required value
  Node user ID.
  
  We recommend using the values >= 1100 to avoid conflicts with manually created users.
  
  This parameter does not change during the entire resource life.
  
  Allowed values: 1001 <= X
  
  Example:
  uid: 1100
status
object
User creation status information
- status.errors
  object
  List of user creation errors on each node

Deprecated resource. Support for the resource might be removed in a later release.

Defines the linux users to create on all nodes.

The user’s home directory is created in the /home/deckhouse/ directory.

spec
object
Required value
- spec.extraGroups
  array of strings
  Node user additional system groups.
  
  Examples:
  extraGroups: - docker
  extraGroups: - docker - ftp
- spec.isSudoer
  boolean
  Persistence of node user in sudo group.
  
  Default: false
  Example:
  isSudoer: true
- spec.nodeGroups
  array of strings
  List of NodeGroups to apply the user for.
  
  Default: ["*"]
  Examples:
  nodeGroups: - master - worker
  nodeGroups: - worker
  nodeGroups: - "*"
- spec.passwordHash
  string
  Hashed user password.
  
  The format corresponds to the password hashes in /etc/shadow. You can get it using the following command: openssl passwd -6.
  
  Example:
  passwordHash: "$2a$10$F9ey7zW.sVliT224RFxpWeMsgzO.D9YRG54a8T36/K2MCiT41nzmC"
- spec.sshPublicKey
  Deprecated
  string
  Node user SSH public key.
  
  Either sshPublicKey or sshPublicKeys must be specified.
  
  Example:
  sshPublicKey: ssh-rsa AAABBB
- spec.sshPublicKeys
  array of strings
  Node user SSH public keys.
  
  Either sshPublicKey or sshPublicKeys must be specified.
  
  Example:
  sshPublicKeys: - ssh-rsa AAABBB - cert-authority,principals="name" ssh-rsa BBBCCC
- spec.uid
  number
  Required value
  Node user ID.
  
  We recommend using the values >= 1100 to avoid conflicts with manually created users.
  
  This parameter does not change during the entire resource life.
  
  Allowed values: 1001 <= X
  
  Example:
  uid: 1100

SSHCredentials

Scope: Cluster

v1alpha2
v1alpha1

Contains credentials required by Cluster API Provider Static (CAPS) to connect over SSH. CAPS connects to the server (virtual machine) defined in the StaticInstance custom resource to manage its state.

A reference to this resource is specified in the credentialsRef parameter of the StaticInstance resource.

apiVersion
string
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
kind
string
Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata
object
spec
object
SSHCredentialsSpec defines the desired state of SSHCredentials.
- spec.privateSSHKey
  string
  Private SSH key in PEM format encoded as base64 string.
  
  If the user does not have permission to view sensitive SSHCredentials data, the API returns instead of the actual value.
- spec.sshExtraArgs
  string
  A list of additional arguments to pass to the openssh command.
  
  Examples:
  sshExtraArgs: "-vvv"
  sshExtraArgs: "-c chacha20-poly1305@openssh.com"
  sshExtraArgs: "-c aes256-gcm@openssh.com"
  sshExtraArgs: "-m umac-64-etm@openssh.com"
  sshExtraArgs: "-m hmac-sha2-512-etm@openssh.com"
- spec.sshPort
  integer
  A port to connect to the host via SSH.
  
  Default: 22
  Allowed values: 1 <= X <= 65535
- spec.sudoPasswordEncoded
  string
  Base64 encoded sudo password for the user.
  
  If the user does not have permission to view sensitive SSHCredentials data, the API returns instead of the actual value.
- spec.user
  string
  Required value
  A username to connect to the host via SSH.

A reference to this resource is specified in the credentialsRef parameter of the StaticInstance resource.

apiVersion
string
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
kind
string
Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata
object
spec
object
SSHCredentialsSpec defines the desired state of SSHCredentials.
- spec.privateSSHKey
  string
  Private SSH key in PEM format encoded as base64 string.
  
  If the user does not have permission to view sensitive SSHCredentials data, the API returns instead of the actual value.
- spec.sshExtraArgs
  string
  A list of additional arguments to pass to the openssh command.
  
  Examples:
  sshExtraArgs: "-vvv"
  sshExtraArgs: "-c chacha20-poly1305@openssh.com"
  sshExtraArgs: "-c aes256-gcm@openssh.com"
  sshExtraArgs: "-m umac-64-etm@openssh.com"
  sshExtraArgs: "-m hmac-sha2-512-etm@openssh.com"
- spec.sshPort
  integer
  A port to connect to the host via SSH.
  
  Default: 22
  Allowed values: 1 <= X <= 65535
- spec.sudoPassword
  string
  A sudo password for the user.
  
  If the user does not have permission to view sensitive SSHCredentials data, the API returns instead of the actual value.
- spec.user
  string
  Required value
  A username to connect to the host via SSH.

StaticInstance

Scope: Cluster

v1alpha2
v1alpha1

StaticInstance describes a machine for the Cluster API Provider Static.

apiVersion
string
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
kind
string
Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata
object
spec
object
StaticInstanceSpec defines the desired state of StaticInstance.
- spec.address
  string
  Required value
  The IP address of the host.
  
  Pattern: ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$
- spec.credentialsRef
  object
  Required value
  The reference to the SSHCredentials object.
  - spec.credentialsRef.apiVersion
    string
    API version of the referent.
  - spec.credentialsRef.fieldPath
    string
    If referring to a piece of an object instead of an entire object, this string should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2]. For example, if the object reference is to a container within a pod, this would take on a value like: “spec.containers{name}” (where “name” refers to the name of the container that triggered the event) or if no container name is specified “spec.containers[2]” (container with index 2 in this pod). This syntax is chosen only to have some well-defined way of referencing a part of an object.
  - spec.credentialsRef.kind
    string
    Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
  - spec.credentialsRef.name
    string
    Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
  - spec.credentialsRef.namespace
    string
    Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
  - spec.credentialsRef.resourceVersion
    string
    Specific resourceVersion to which this reference is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency
  - spec.credentialsRef.uid
    string
    UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids
status
object
StaticInstanceStatus defines the observed state of StaticInstance.
- status.conditions
  array of objects
  Conditions defines current service state of the StaticInstance.
  
  Condition contains details for one aspect of the current state of this API Resource.
  - status.conditions.lastTransitionTime
    string
    Required value
    lastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
  - status.conditions.message
    string
    Required value
    message is a human readable message indicating details about the transition. This may be an empty string.
    
    Maximum length: 32768
  - status.conditions.observedGeneration
    integer
    observedGeneration represents the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance.
    
    Allowed values: 0 <= X
  - status.conditions.reason
    string
    Required value
    reason contains a programmatic identifier indicating the reason for the condition’s last transition. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API. The value should be a CamelCase string. This field may not be empty.
    
    Pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
    Length: 1..1024
  - status.conditions.status
    string
    Required value
    status of the condition, one of True, False, Unknown.
    
    Allowed values: True, False, Unknown
  - status.conditions.type
    string
    Required value
    type of condition in CamelCase or in foo.example.com/CamelCase.
    
    Pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
    Maximum length: 316
- status.currentStatus
  object
  - status.currentStatus.lastUpdateTime
    string
  - status.currentStatus.phase
    string
    Allowed values: Error, Pending, Bootstrapping, Running, Cleaning
- status.machineRef
  object
  The reference to the StaticMachine object.
  - status.machineRef.apiVersion
    string
    API version of the referent.
  - status.machineRef.fieldPath
    string
    If referring to a piece of an object instead of an entire object, this string should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2]. For example, if the object reference is to a container within a pod, this would take on a value like: “spec.containers{name}” (where “name” refers to the name of the container that triggered the event) or if no container name is specified “spec.containers[2]” (container with index 2 in this pod). This syntax is chosen only to have some well-defined way of referencing a part of an object.
  - status.machineRef.kind
    string
    Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
  - status.machineRef.name
    string
    Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
  - status.machineRef.namespace
    string
    Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
  - status.machineRef.resourceVersion
    string
    Specific resourceVersion to which this reference is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency
  - status.machineRef.uid
    string
    UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids
- status.nodeRef
  object
  The reference to the Node object.
  - status.nodeRef.apiVersion
    string
    API version of the referent.
  - status.nodeRef.fieldPath
    string
    If referring to a piece of an object instead of an entire object, this string should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2]. For example, if the object reference is to a container within a pod, this would take on a value like: “spec.containers{name}” (where “name” refers to the name of the container that triggered the event) or if no container name is specified “spec.containers[2]” (container with index 2 in this pod). This syntax is chosen only to have some well-defined way of referencing a part of an object.
  - status.nodeRef.kind
    string
    Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
  - status.nodeRef.name
    string
    Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
  - status.nodeRef.namespace
    string
    Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
  - status.nodeRef.resourceVersion
    string
    Specific resourceVersion to which this reference is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency
  - status.nodeRef.uid
    string
    UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids

StaticInstance describes a machine for the Cluster API Provider Static.

apiVersion
string
APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
kind
string
Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
metadata
object
spec
object
StaticInstanceSpec defines the desired state of StaticInstance.
- spec.address
  string
  Required value
  The IP address of the host.
  
  Pattern: ^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$
- spec.credentialsRef
  object
  Required value
  The reference to the SSHCredentials object.
  - spec.credentialsRef.apiVersion
    string
    API version of the referent.
  - spec.credentialsRef.fieldPath
    string
    If referring to a piece of an object instead of an entire object, this string should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2]. For example, if the object reference is to a container within a pod, this would take on a value like: “spec.containers{name}” (where “name” refers to the name of the container that triggered the event) or if no container name is specified “spec.containers[2]” (container with index 2 in this pod). This syntax is chosen only to have some well-defined way of referencing a part of an object.
  - spec.credentialsRef.kind
    string
    Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
  - spec.credentialsRef.name
    string
    Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
  - spec.credentialsRef.namespace
    string
    Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
  - spec.credentialsRef.resourceVersion
    string
    Specific resourceVersion to which this reference is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency
  - spec.credentialsRef.uid
    string
    UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids
status
object
StaticInstanceStatus defines the observed state of StaticInstance.
- status.conditions
  array of objects
  Conditions defines current service state of the StaticInstance.
  
  Condition contains details for one aspect of the current state of this API Resource.
  - status.conditions.lastTransitionTime
    string
    Required value
    lastTransitionTime is the last time the condition transitioned from one status to another. This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
  - status.conditions.message
    string
    Required value
    message is a human readable message indicating details about the transition. This may be an empty string.
    
    Maximum length: 32768
  - status.conditions.observedGeneration
    integer
    observedGeneration represents the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date with respect to the current state of the instance.
    
    Allowed values: 0 <= X
  - status.conditions.reason
    string
    Required value
    reason contains a programmatic identifier indicating the reason for the condition’s last transition. Producers of specific condition types may define expected values and meanings for this field, and whether the values are considered a guaranteed API. The value should be a CamelCase string. This field may not be empty.
    
    Pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
    Length: 1..1024
  - status.conditions.status
    string
    Required value
    status of the condition, one of True, False, Unknown.
    
    Allowed values: True, False, Unknown
  - status.conditions.type
    string
    Required value
    type of condition in CamelCase or in foo.example.com/CamelCase.
    
    Pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
    Maximum length: 316
- status.currentStatus
  object
  - status.currentStatus.lastUpdateTime
    string
  - status.currentStatus.phase
    string
    Allowed values: Error, Pending, Bootstrapping, Running, Cleaning
- status.machineRef
  object
  The reference to the StaticMachine object.
  - status.machineRef.apiVersion
    string
    API version of the referent.
  - status.machineRef.fieldPath
    string
    If referring to a piece of an object instead of an entire object, this string should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2]. For example, if the object reference is to a container within a pod, this would take on a value like: “spec.containers{name}” (where “name” refers to the name of the container that triggered the event) or if no container name is specified “spec.containers[2]” (container with index 2 in this pod). This syntax is chosen only to have some well-defined way of referencing a part of an object.
  - status.machineRef.kind
    string
    Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
  - status.machineRef.name
    string
    Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
  - status.machineRef.namespace
    string
    Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
  - status.machineRef.resourceVersion
    string
    Specific resourceVersion to which this reference is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency
  - status.machineRef.uid
    string
    UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids
- status.nodeRef
  object
  The reference to the Node object.
  - status.nodeRef.apiVersion
    string
    API version of the referent.
  - status.nodeRef.fieldPath
    string
    If referring to a piece of an object instead of an entire object, this string should contain a valid JSON/Go field access statement, such as desiredState.manifest.containers[2]. For example, if the object reference is to a container within a pod, this would take on a value like: “spec.containers{name}” (where “name” refers to the name of the container that triggered the event) or if no container name is specified “spec.containers[2]” (container with index 2 in this pod). This syntax is chosen only to have some well-defined way of referencing a part of an object.
  - status.nodeRef.kind
    string
    Kind of the referent. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
  - status.nodeRef.name
    string
    Name of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#names
  - status.nodeRef.namespace
    string
    Namespace of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/
  - status.nodeRef.resourceVersion
    string
    Specific resourceVersion to which this reference is made, if any. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#concurrency-control-and-consistency
  - status.nodeRef.uid
    string
    UID of the referent. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#uids

Managing nodes: custom resources

Instance

InstanceTypesCatalog

NodeGroup

NodeConfig

NodeExtensionRequest

NodeGroupConfiguration

NodeOperation

NodeStaticPodRequest

NodeUser

SSHCredentials

StaticInstance

An error has occurred

Tell us what you didn’t like.

Managing nodes: custom resources

Instance

InstanceTypesCatalog

NodeGroup

NodeConfig

NodeExtensionRequest

NodeGroupConfiguration

NodeOperation

NodeStaticPodRequest

NodeUser

SSHCredentials

StaticInstance

An error has occurred

Tell us what you didn’t like.

Request trial access

Thank you

Error

Request callback

Thank you

Something went wrong

Book your sessions

Thank you

Error

Request demo

Thank you

Error

Get the PCI SSC compliance report

Thank you

Error

Request partner program details

Request received

Error