Managing control plane: configuration

• Updates from version 1 to 2:

  Delete etcd.externalMembersNames

• Updates from version 2 to 3:

  Replace .apiserver.loadBalancer with .apiserver.publishAPI.loadBalancer.

Schema version: 3

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: control-plane-manager
spec:
  version: 3
  enabled: true
  settings:
    apiserver:
      bindToWildcard: true
      certSANs:
      - bakery.infra
      - devs.infra
      publishAPI: {}

• settings

  object
  • settings.apiserver

    object

    kube-apiserver parameters.

    • settings.apiserver.admissionPlugins

      array of strings

      List of enabled additional admission plugins.

      Note, that in addition to the admission plugins enabled by default in Kubernetes, the following admission plugins are also always enabled:

      • ExtendedResourceToleration
      • PodNodeSelector
      • PodTolerationRestriction

      • EventRateLimit with the following config:

        apiVersion: eventratelimit.admission.k8s.io/v1alpha1
        kind: Configuration
        limits:
        - type: Namespace
          qps: 50
          burst: 100
          cacheSize: 2000
        

      Note that th PodNodeSelector admission plugin does not require specifiying a global configuration, it relies on annotated Namespaces.

      Example:

      admissionPlugins:
      - AlwaysPullImages
      - NamespaceAutoProvision
      

      • Element of the array

        string

        Allowed values: AlwaysPullImages, NamespaceAutoProvision, OwnerReferencesPermissionEnforcement, PodNodeSelector, PodTolerationRestriction

    • settings.apiserver.auditLog

      object

      Audit policy settings

      Default: {"output":"File"}

      • settings.apiserver.auditLog.output

        string

        Required value

        Audit logs target stream.

        Default: File

        Allowed values: File, Stdout

        Example:

        output: Stdout
        

      • settings.apiserver.auditLog.path

        string

        Directory path for logs if the output is “File”, otherwise ignored.

        Default: /var/log/kube-audit

        Pattern: ^[a-zA-Z0-9_/.-]+[a-zA-Z0-9_.-]$

    • settings.apiserver.auditPolicyEnabled

      boolean

      Set the audit policies using the configuration from the kube-system/audit-policy Secret.

      Default: false

    • settings.apiserver.authn

      object

      Optional authentication parameters for Kubernetes API clients.

      By default, they are taken from user-authn module ConfigMap.

      • settings.apiserver.authn.oidcCA

        string

        OIDC provider CA.

      • settings.apiserver.authn.oidcIssuerAddress

        string

        OIDC provider network address alias.

        Examples:

        oidcIssuerAddress: 1.2.3.4
        

        oidcIssuerAddress: ''
        

      • settings.apiserver.authn.oidcIssuerURL

        string

        OIDC provider URL.

        Example:

        oidcIssuerURL: https://my-super-site.tech/
        

      • settings.apiserver.authn.webhookCA

        string

        Authorization webhook CA.

      • settings.apiserver.authn.webhookCacheTTL

        string

        The duration to cache responses from the webhook token authenticator.

        It is specified as a string containing the time unit in hours and minutes: 30m, 1h, 2h30m, 24h.

        Pattern: ^([0-9]+h)?([0-9]+m)?([0-9]+s)?$

        Example:

        webhookCacheTTL: 5m
        

      • settings.apiserver.authn.webhookURL

        string

        Authentication webhook URL.

        Example:

        webhookURL: https://127.0.0.1:40443/
        

    • settings.apiserver.authz

      object

      Optional authorization parameters for Kubernetes API clients.

      By default, they are taken from user-authz module ConfigMap.

      • settings.apiserver.authz.webhookCA

        string

        Authorization webhook CA.

      • settings.apiserver.authz.webhookURL

        string

        Authorization webhook URL.

        Example:

        webhookURL: https://127.0.0.1:40443/
        

    • settings.apiserver.basicAuditPolicyEnabled

      boolean

      Enforce basic Deckhouse audit policies.

      Default: true

    • settings.apiserver.bindToWildcard

      boolean

      Specifies whether to listen on 0.0.0.0.

      By default, the API server listens on the hostIP. The latter usually corresponds to the Internal node address; however, the actual IP depends on the cluster type (Static or Cloud) and the layout selected.

      Default: false

    • settings.apiserver.certSANs

      array of strings

      Array of SANs, with which the API server certificate will be generated.

      In addition to the passed list, the following list is always used:

      • kubernetes;
      • kubernetes.default;
      • kubernetes.default.svc;
      • kubernetes.default.svc.cluster.local;
      • Kubernetes API server address;
      • 127.0.0.1;
      • host name;
      • host IP.

      Example:

      certSANs:
      - my-site.com
      - 192.168.67.76
      

      • Element of the array

        string

        Pattern: ^[0-9a-zA-Z\.-]+$

    • settings.apiserver.encryptionEnabled

      boolean

      Enables data encryption in etcd (see Encrypting secret data at rest).

      When enabled, a Secret kube-system/d8-secret-encryption-key is created to store the encryption key.

      Enabling this parameter automatically enables the CRDSensitiveData feature gate for kube-apiserver, which protects custom resource fields marked with x-kubernetes-sensitive-data: true via encryption in etcd, RBAC-based field filtering with the <resource>/sensitive subresource and hiding values from the audit log.

      Warning. Once enabled, this parameter can’t be disabled.

      Default: false

    • settings.apiserver.publishAPI

      object

      Publishing access to Kubernetes API.

      • settings.apiserver.publishAPI.ingress

        object

        Settings for PublishAPI type Ingress.

        • settings.apiserver.publishAPI.ingress.addKubeconfigGeneratorEntry

          boolean

          Setting it to false will remove an entry in kubeconfig-generator.

          Default: true

        • settings.apiserver.publishAPI.ingress.enabled

          boolean

          Setting it to true will create an Ingress resource kubernetes-api in the kube-system namespace in the cluster (it exposes the Kubernetes API).

          Default: false

        • settings.apiserver.publishAPI.ingress.https

          object

          The HTTPS mode for the API server Ingress.

          Examples:

          https:
            mode: SelfSigned
          

          https:
            mode: Global
            global:
              kubeconfigGeneratorMasterCA: plainstring
          

          • settings.apiserver.publishAPI.ingress.https.global

            object

            An additional parameter for the Global mode.

            • settings.apiserver.publishAPI.ingress.https.global.kubeconfigGeneratorMasterCA

              string

              If there is an external load balancer in front of the Ingress that terminates HTTPS traffic using non-public CA, then you need to specify the CA so it will be included in kubectl-config.

              If you are using certificates issued by the cert-manager module and Let’s Encrypt in your cluster, you should set an empty string "" as the value.

              Also, you can set the external LB’s certificate itself as a CA if you can’t get the CA that signed it for some reason. Note that after the certificate is updated on the LB, all the previously generated kubeconfigs will stop working.

          • settings.apiserver.publishAPI.ingress.https.mode

            string

            The mode of issuing certificates for the Ingress resource.

            In the SelfSigned mode, a CA-signed certificate will be issued for the Ingress resource.

            Use the following command to get the certificate: d8 k -n kube-system get secrets kubernetes-api-ca-key-pair -oyaml.

            In the Global mode, the policies specified in the global.modules.https.mode global parameter will be applied. Thus, if the global parameter has the CertManager mode set (with letsencrypt as the ClusterIssuer), then the Let’s Encrypt certificate will be issued for the Ingress resource.

            Default: SelfSigned

            Allowed values: SelfSigned, Global

        • settings.apiserver.publishAPI.ingress.ingressClass

          string

          The Ingress class that will be used to expose the Kubernetes API via Ingress.

          Pattern: ^[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*$

        • settings.apiserver.publishAPI.ingress.whitelistSourceRanges

          array of strings

          An array of CIDRs that are allowed to connect to the API server.

          • Element of the array

            string

            Pattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/(3[0-2]|[1-2][0-9]|[0-9]))?$

      • settings.apiserver.publishAPI.loadBalancer

        object

        Settings type LoadBalancer

        • settings.apiserver.publishAPI.loadBalancer.annotations

          object

          Annotations to attach to a service to fine-tune the load balancer.

          Caution! The module does not take into account the specifics of setting annotations in various cloud environments. If the annotations for load balancer provisioning are only applied when creating a service, you will need to delete and add the apiserver.loadBalancer parameter to update such parameters.

        • settings.apiserver.publishAPI.loadBalancer.enabled

          boolean

          If set to true, service kube-system/d8-control-plane-apiserver of the LoadBalancer type will be created.

          Default: false

        • settings.apiserver.publishAPI.loadBalancer.port

          integer

          External LoadBalancer TCP port.

          Default: 443

          Allowed values: 1 <= X <= 65534

        • settings.apiserver.publishAPI.loadBalancer.sourceRanges

          array of strings

          A list of CIDRs that are allowed to connect to the API.

          The cloud provider may not support this option or ignore it.

          • Element of the array

            string

            Pattern: ^[0-9]{1,}\.[0-9]{1,}\.[0-9]{1,}\.[0-9]{1,}\/[0-9]+$

    • settings.apiserver.serviceAccount

      object

      ServiceAccount issuing settings.

      Examples:

      serviceAccount: {}
      

      serviceAccount:
        additionalAPIAudiences:
        - istio-ca
      

      • settings.apiserver.serviceAccount.additionalAPIAudiences

        array of strings

        A list of API audiences to add when provisioning ServiceAccount tokens.

        The default audiences list includes:

        • serviceAccount.issuer, if specified.
        • serviceAccount.additionalAPIIssuers, if specified.
        • https://kubernetes.default.svc.${clusterDomain}, for example, https://kubernetes.default.svc.cluster.local – this audience is always added for backward compatibility.

        The service account token authenticator will validate that tokens used against the API are bound to at least one of these audiences.

        This option is beneficial when migrating from one API issuer to another.

      • settings.apiserver.serviceAccount.additionalAPIIssuers

        array of strings

        A list of additional issuers to include when provisioning ServiceAccount tokens.

        Issuers (iss) are used to verify the source of the tokens, ensuring they originate from trusted entities.

        The first issuer is automatically generated based on the template https://kubernetes.default.svc.${clusterDomain}, for example, https://kubernetes.default.svc.cluster.local. When multiple issuers are specified, the first issuer is used to generate tokens, and all provided issuers are accepted for token verification.

        This option is beneficial when migrating from one API issuer to another.

      • settings.apiserver.serviceAccount.issuer

        string

        ServiceAccount issuer. This is the URL of the API server. The values of this field are used as the iss claim of the token and to verify Service Account JWT tokens.

        Note, all pods in the cluster using ServiceAccount tokens must be restarted upon changing this option.

        Example:

        issuer: https://api.example.com
        

  • settings.enabledFeatureGates

    array of strings

    List of feature gates enabled in the control plane.

    More information about feature gates is available in the Kubernetes documentation. The list of feature gates supported in DKP can be found on the module overview page.

    Default: []

    • Element of the array

      string

      Pattern: ^[a-zA-Z]+$

  • settings.encryptionAlgorithm

    string

    Asymmetric encryption algorithm used when generating keys and certificates for the following control-plane components:

    • apiserver;
    • apiserver-kubelet-client;
    • apiserver-etcd-client;
    • front-proxy-client;
    • etcd-server;
    • etcd-peer;
    • etcd-healthcheck-client.

    Certificates for the components listed above will be reissued using the selected algorithm and key length.

    If parameter not set, the value from ClusterConfiguration.encryptionAlgorithm is used (defaults to RSA-2048).

    Warning. When reissuing certificates, the root certificate (CA) is not rotated. The root certificate is created with the selected algorithm only during the initial cluster bootstrap.

    Default: RSA-2048

    Allowed values: RSA-2048, RSA-3072, RSA-4096, ECDSA-P256

    Example:

    encryptionAlgorithm: ECDSA-P256
    

  • settings.etcd

    object

    etcd parameters.

    • settings.etcd.backup

      object

      etcd backup parameters.

      • settings.etcd.backup.cronSchedule

        string

        Backup schedule etcd in cron format. The local time zone of the kube-controller-manager is used.

        Default: 0 0 * * *

        Example:

        cronSchedule: 0 1 * * *
        

      • settings.etcd.backup.enabled

        boolean

        Enable etcd backup.

        Default: true

      • settings.etcd.backup.hostPath

        string

        Host path on masters nodes for etcd backup.

        Default: /var/lib/etcd

    • settings.etcd.defrag

      object

      etcd defragmentation parameters.

      Default: {"cronSchedule":"0 1 * * *"}

      • settings.etcd.defrag.cronSchedule

        string

        Defragmentation schedule in cron format (UTC).

        Default: 0 1 * * *

        Pattern: ^(\*(\/[0-9]+)?|[0-9]+(-[0-9]+)?(\/[0-9]+)?(,[0-9]+(-[0-9]+)?(\/[0-9]+)?)*)\s+(\*(\/[0-9]+)?|[0-9]+(-[0-9]+)?(\/[0-9]+)?(,[0-9]+(-[0-9]+)?(\/[0-9]+)?)*)\s+(\*(\/[0-9]+)?|[0-9]+(-[0-9]+)?(\/[0-9]+)?(,[0-9]+(-[0-9]+)?(\/[0-9]+)?)*)\s+(\*(\/[0-9]+)?|[0-9]+(-[0-9]+)?(\/[0-9]+)?(,[0-9]+(-[0-9]+)?(\/[0-9]+)?)*)\s+(\*(\/[0-9]+)?|[0-9]+(-[0-9]+)?(\/[0-9]+)?(,[0-9]+(-[0-9]+)?(\/[0-9]+)?)*)$

        Example:

        cronSchedule: 0 1 * * *
        

      • settings.etcd.defrag.enabled

        boolean

        Enable periodic etcd defragmentation.

        Defaults to true when the cluster has three or more etcd members (masters ≥ 3, or masters = 2 with an etcd-arbiter node); otherwise defaults to false. An explicit value always takes priority over the computed default.

        Warning. Enabling defragmentation on a single-master cluster will cause a brief control plane unavailability while etcd is being defragmented.

    • settings.etcd.maxDbSize

      number

      quota-backend-bytes parameter. Deckhouse automatically manages the quota-backend-bytes parameter. If the maxDbSize parameter is set, deckhouse will use this value for the quota-backend-bytes etcd parameter.

      Minimum: 512MB.

      Maximum: 8GB.

      Experimental. It can be removed in the future.

      Allowed values: 536870912 <= X <= 8589934592

  • settings.failedNodePodEvictionTimeoutSeconds

    integer

    The number of seconds after which pods will be deleted from the node with the Unreachable status.

    Note! If you change the parameter, the pods must be restarted.

    Default: 300

  • settings.nodeMonitorGracePeriodSeconds

    integer

    The number of seconds after which the node will enter the Unreachable status in case of lost connection.

    Default: 40

  • settings.rootKubeconfigSymlink

    boolean

    Whether to maintain a symlink from /root/.kube/config to /etc/kubernetes/admin.conf on control-plane nodes.

    When the user-authz module is enabled and this parameter is false, the symlink is removed and the NODE_ADMIN_KUBECONFIG environment variable is set on control-plane-manager pods accordingly. Use d8 k with --kubeconfig=/etc/kubernetes/admin.conf (or other credentials) when you need API access from the node.

    When the user-authz module is disabled, control-plane-manager ignores this parameter and keeps the default (symlink present).

    Setting the value back to true restores the symlink on the next reconciliation.

    Default: true

    Examples:

    rootKubeconfigSymlink: true
    

    rootKubeconfigSymlink: false