The list of alerts

The page displays a list of all alerts of monitoring in the Deckhouse Platform.

Alerts are grouped by modules. To the right of the alert name, there are icons indicating the minimum DP edition in which the alert is available, and the alert severity.

For each alert, a summary is provided, and if available, the detailed alert description can be viewed by expanding it.

Alert severity

Alert descriptions contain the Severity (S) parameter, which indicates the level of criticality. Its value varies from S1 to S9 and can be interpreted as follows:

• S1 — maximum level, critical failure/crash (immediate action required);
• S2 — high level, close to maximum, possible accident (rapid response required);
• S3 — medium level, potentially serious problem (verification required);
• S4-S9 — low level. There is a problem, but overall performance is not impaired.

Module admission-policy-engine

• ControllerPolicyViolation CE S4
  Pod-creating controllers in the XXX namespace violate the XXX policy, so their Pods are denied.

  The XXX policy (XXX) denies the Pods that violate it. For a transition period, it only warns about a violation in a pod-creating controller (Deployment, StatefulSet, DaemonSet, Job, CronJob, ReplicationController) instead of denying the controller. The controller is accepted, but it cannot create its Pods: the denial appears only in the events of the ReplicaSet or the controller.

  Controllers that violate the policy in the namespace: VALUE.

  The d8-pod-security-baseline and d8-pod-security-restricted policies are the Pod Security Standards.

  To list the violating controllers and the rules they break, run the following Prometheus query:

  count by (violating_kind, violating_name, kind, violation_msg) (
    d8_gatekeeper_exporter_constraint_violations{
      source_type="XXX",
      name=~"d8-controllers-XXXXXX-deny(-d8)?(-default)?XXX",
      violating_namespace="XXX"
    }
  )
  

  Bring the pod template of each controller in line with the policy. To deny such controllers instead of warning about them, set spec.controllerValidation.mode: Enforce in the policy, or the podSecurityStandards.controllerValidationEnforcement module parameter to Policy. You can also check the admission-policy-engine reports in the Deckhouse web interface.

• D8AdmissionPolicyEngineAuditUnavailable CE S4
  The audit controller of admission-policy-engine has no available replicas.

  The gatekeeper-audit Deployment in the d8-admission-policy-engine namespace has no available replicas.

  Admission of new objects is not affected, but the module stops reporting and stops picking up new policies:

  • Violations among the objects that already run are not collected, so the PodSecurityStandardsViolation, OperationPolicyViolation and SecurityPolicyViolation alerts go silent instead of reporting that all is well.
  • The list of resources the webhook intercepts is not updated, so a policy added while the audit is down is not enforced on admission.

  Find out why the replica is gone:

  d8 k -n d8-admission-policy-engine get pods -l app=gatekeeper,control-plane=audit-controller
  d8 k -n d8-admission-policy-engine describe pods -l app=gatekeeper,control-plane=audit-controller
  d8 k -n d8-admission-policy-engine logs deploy/gatekeeper-audit -c manager --tail=200
  d8 k -n d8-admission-policy-engine logs deploy/gatekeeper-audit -c constraint-exporter --tail=200
  

• D8AdmissionPolicyEngineNotBootstrapped CE S7
  Not every ConstraintTemplate of the admission-policy-engine module is ready for 10 minutes.

  Some ConstraintTemplates of the admission-policy-engine module are missing or not processed by Gatekeeper. The constraints of a template are applied only once Gatekeeper has established its CRD.

  On a new installation, the module also renders no webhooks and no policies until every template is ready. In a cluster where the module has already worked, the webhooks and the constraints of the other templates stay in effect.

  Steps to troubleshoot:

  1. Verify that the module’s components are up and running:

     d8 k get pods -n d8-admission-policy-engine
     

  2. Check logs for issues, such as missing constraint templates or incomplete CRD creation:

     d8 k logs -n d8-system -lapp=deckhouse --tail=1000 | grep admission-policy-engine
     

• D8AdmissionPolicyEngineWebhookDegraded CE S3
  The admission webhook of admission-policy-engine has unavailable replicas.

  The gatekeeper-controller-manager Deployment in the d8-admission-policy-engine namespace has unavailable replicas, and the remaining ones still serve admission requests.

  The webhook has no redundancy left. Losing the last replica makes the API server reject every request the webhook intercepts, because the webhook has failurePolicy: Fail.

  A rollout of the Deployment raises this alert for a short time and needs no action. If the alert holds, find out why the replica does not start:

  d8 k -n d8-admission-policy-engine get pods -l app=gatekeeper,control-plane=controller-manager -o wide
  d8 k -n d8-admission-policy-engine describe pods -l app=gatekeeper,control-plane=controller-manager
  d8 k -n d8-admission-policy-engine get events --sort-by=.lastTimestamp | tail -30
  

• D8AdmissionPolicyEngineWebhookNotServing CE S2
  The API server cannot call the admission webhook of admission-policy-engine.

  The API server is rejecting requests because it fails to call a webhook of the admission-policy-engine module. The webhook has failurePolicy: Fail, so the cluster is blocked for everything the webhook intercepts.

  The alert is independent of the replica counters: it fires while the pods are Ready as well. A replica that is running but unreachable to the API server produces the same outage, and only this alert reports it. Typical causes are a gatekeeper-webhook-server-cert that no longer matches the caBundle of the ValidatingWebhookConfiguration, an empty gatekeeper-webhook-service endpoint list, and a network policy that blocks the API server.

  The API server counts a webhook that exceeds its timeoutSeconds of 15 the same way it counts one it cannot reach, so a few slow evaluations of a large object raise the metric as well. They do not raise the alert: for: 15m requires the failures to continue, while a burst leaves the rate elevated for little more than its own duration plus the five-minute window.

  Read the reason the API server reports, then check the certificate and the endpoints:

  d8 k -n d8-admission-policy-engine logs deploy/gatekeeper-controller-manager -c manager --all-pods=true --tail=200
  d8 k -n d8-admission-policy-engine get endpoints gatekeeper-webhook-service
  d8 k get validatingwebhookconfiguration d8-admission-policy-engine-config -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d | openssl x509 -noout -subject -dates
  d8 k -n d8-admission-policy-engine get secret gatekeeper-webhook-server-cert -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -subject -dates
  

  The two certificates must match. Deckhouse Platform reissues them through the module’s hook, so deleting the secret makes it regenerate the pair.

• D8AdmissionPolicyEngineWebhookUnavailable CE S2
  The admission webhook of admission-policy-engine has no available replicas.

  The gatekeeper-controller-manager Deployment in the d8-admission-policy-engine namespace has no available replicas.

  The Deployment serves the admission webhook of the module, and the webhook has failurePolicy: Fail. While no replica is available, the API server rejects every request the webhook intercepts, cluster-wide:

  • Pods and the controllers that create them are not created or updated.
  • Deletion of those resources is rejected as well.
  • Roles, RoleBindings and Gatekeeper constraints are not applied.
  • The d8 k exec, d8 k attach and d8 k debug requests intercepted by the webhooks are rejected. This includes d8 k exec and d8 k attach in the d8-* and kube-* namespaces.

  Find out why the replicas are gone:

  d8 k -n d8-admission-policy-engine get deploy gatekeeper-controller-manager
  d8 k -n d8-admission-policy-engine get pods -l app=gatekeeper,control-plane=controller-manager -o wide
  d8 k -n d8-admission-policy-engine describe pods -l app=gatekeeper,control-plane=controller-manager
  d8 k -n d8-admission-policy-engine logs deploy/gatekeeper-controller-manager -c manager --all-pods=true --tail=200
  

  If a pod stays Pending, look at the events of the namespace and at the nodes the replicas can run on:

  d8 k -n d8-admission-policy-engine get events --sort-by=.lastTimestamp | tail -30
  d8 k get nodes -l node-role.deckhouse.io/system
  d8 k get nodes -l node-role.kubernetes.io/control-plane
  

• OperationPolicyViolation CE S3
  XXXObjects in the XXX namespaceXXXCluster-scoped objectsXXX violate the XXX OperationPolicy.

  Existing objects XXXin the XXX namespace XXXof cluster-scoped kinds XXXviolate the XXX OperationPolicy. An object is checked when it is created or updated, so these objects were created before the policy applied to them or before its rules changed.

  Objects that violate the policy: VALUE.

  To list the violating objects and the rules they break, run the following Prometheus query:

  count by (violating_kind, violating_name, kind, violation_msg) (
    d8_gatekeeper_exporter_constraint_violations{
      source_type="OperationPolicy",
      name="XXX",
      violation_enforcement="deny",
      violating_namespace="XXX"
    }
  )
  

  Bring the objects in line with the policy. You can also check the admission-policy-engine reports in the Deckhouse web interface.

• PodSecurityStandardsViolation CE S3
  Pods in the XXX namespace violate the XXX Pod Security Standard.

  Running Pods in the XXX namespace violate the Pod Security Standard published as the XXX SecurityPolicy. A Pod is checked when it is created, so these Pods were created before the standard applied to them or before its rules changed. They keep running until they are restarted, and are denied then.

  Pods that violate the standard in the namespace: VALUE.

  To list the violating Pods and the rules they break, run the following Prometheus query:

  count by (violating_name, kind, violation_msg) (
    d8_gatekeeper_exporter_constraint_violations{
      source_type="PSS",
      name=~"XXX-.+",
      violation_enforcement="deny",
      violating_kind="Pod",
      violating_namespace="XXX"
    }
  )
  

  Bring the workloads in line with the standard, or exempt them with a SecurityPolicyException. You can also check the admission-policy-engine reports in the Deckhouse web interface.

• SecurityPolicyViolation CE S3
  XXXObjects in the XXX namespaceXXXCluster-scoped objectsXXX violate the XXX SecurityPolicy.

  Existing objects XXXin the XXX namespace XXXof cluster-scoped kinds XXXviolate the XXX SecurityPolicy. An object is checked when it is created or updated, so these objects were created before the policy applied to them or before its rules changed.

  Objects that violate the policy: VALUE.

  To list the violating objects and the rules they break, run the following Prometheus query:

  count by (violating_kind, violating_name, kind, violation_msg) (
    d8_gatekeeper_exporter_constraint_violations{
      source_type="SecurityPolicy",
      name="XXX",
      violation_enforcement="deny",
      violating_namespace="XXX"
    }
  )
  

  Bring the objects in line with the policy, or exempt them with a SecurityPolicyException. You can also check the admission-policy-engine reports in the Deckhouse web interface.

Module cert-manager

• CertmanagerCertificateChallengePending CE S4
  ACME Challenge for domain "XXX" is pending for more than 1 hour.

  The ACME Challenge NAMESPACE/NAME for domain XXX has been in pending state for more than 1 hour and requires investigation.

  Common causes:

  • DNS misconfiguration (for DNS-01 challenges).
  • Ingress or firewall issues blocking HTTP-01 validation.
  • ACME provider rate limiting.
  • Network connectivity issues.

  Resolution steps:

  1. Check the Challenge status and reason:

     d8 k -n NAMESPACE describe challenge NAME
     

  2. Check cert-manager logs for errors:

     d8 k -n d8-cert-manager logs -l app=cert-manager -c cert-manager --tail=100
     

  3. If the challenge is stuck, delete the Order to trigger a retry:

     d8 k -n NAMESPACE get orders
     d8 k -n NAMESPACE delete order ORDER_NAME
     

• CertmanagerCertificateChallengePending CE S6
  ACME Challenge for domain "XXX" is pending for more than 10 minutes.

  The ACME Challenge NAMESPACE/NAME for domain XXX has been in pending state for more than 10 minutes.

  This may indicate issues with the ACME challenge validation process.

  Common causes:

  • DNS propagation delays (for DNS-01 challenges).
  • Ingress or firewall issues blocking HTTP-01 validation.
  • ACME provider rate limiting or temporary unavailability.

  Diagnostic steps:

  1. Check the Challenge status:

     d8 k -n NAMESPACE describe challenge NAME
     

  2. Check cert-manager logs for errors:

     d8 k -n d8-cert-manager logs -l app=cert-manager -c cert-manager --tail=100
     

• CertmanagerCertificateChallengeStuck CE S4
  ACME Challenge for domain "XXX" is stuck in "STATUS_REFERENCE" state.

  The ACME Challenge NAMESPACE/NAME for domain XXX is in STATUS_REFERENCE state and requires manual intervention.

  Reason: XXX

  The certificate will not be issued or renewed until the issue is resolved.

  Common causes:

  • DNS misconfiguration (for DNS-01 challenges).
  • Ingress or firewall issues (for HTTP-01 challenges).
  • Rate limiting by the ACME provider.
  • Authorization expired before challenge completion.

  Resolution steps:

  1. Check the Challenge status and reason:

     d8 k -n NAMESPACE describe challenge NAME
     

  2. Delete the failed Order to trigger a new certificate request:

     d8 k -n NAMESPACE get orders
     d8 k -n NAMESPACE delete order ORDER_NAME
     

• CertmanagerCertificateExpired CE S4
  Certificate NAMESPACE/NAME is not provided.

  Certificate is not provided.

  To check the certificate details, run the following command:

  d8 k -n NAMESPACE describe certificate NAME
  

• CertmanagerCertificateExpiredSoon CE S4
  Certificate will expire soon.

  The certificate NAMESPACE/NAME will expire in less than two weeks.

  To check the certificate details, run the following command:

  d8 k -n NAMESPACE describe certificate CERTIFICATE-NAME
  

• CertmanagerCertificateOrderErrors CE S5
  Cert-manager couldn't order a certificate.

  Cert-manager received responses with the status code STATUS_REFERENCE when requesting SCHEME://HOST_NAMEPATH.

  This can affect certificate ordering and prolongation in the future. For details, check the cert-manager logs using the following command:

  d8 k -n d8-cert-manager logs -l app=cert-manager -c cert-manager
  

Module chrony

• NodeTimeNotSynchronizedAtBoot CE S5
  Master node NODE_NAME started kubelet before its clock was synchronized.

  During the last boot of the master node NODE_NAME, the clock was not synchronized from NTP before kubelet started. Either no NTP server answered within the waiting time, the servers that answered disagreed with each other, nodelet could not run the synchronization, or the synchronization hung and systemd stopped it. kubelet and the control plane started with a clock taken from the RTC or the hypervisor. If that clock was wrong, the control plane could issue tokens and certificates with incorrect timestamps.

  To resolve the issue:

  1. Check the synchronization log on the node. On a node managed by bashible, run:

     journalctl -u nodelet.service -b
     

     A Deckhouse Engine node has no SSH access. Read the log from a debug pod, where IMAGE is any image available to the cluster that has the chroot utility:

     d8 k -n kube-system debug node/NODE_NAME -i --profile=sysadmin --image=IMAGE -- chroot /host journalctl --no-pager -u nodelet.service -b | grep -E 'clock|NTP'
     

  2. On a node managed by bashible, check that the installed nodelet.service has Type=notify and that nodelet started. If the package is missing or outdated, bashible installs it on its next run. To start that run sooner, run the following command on the node:

     rm -f /var/lib/bashible/configuration_checksum
     

  3. Otherwise, ensure the NTP servers defined in the chrony configuration are reachable through the node network and their names are resolvable by the node DNS settings, without Kubernetes DNS. If the log reports the NTP servers disagree or the addresses of SERVER disagree, use servers whose answers agree with each other.

  4. The alert stays active until the node boots again and the synchronization succeeds. On a node managed by bashible, disabling boot-time synchronization also clears it. After you have checked the clock on the node, you can clear the alert without a reboot. On a node managed by bashible, run on the node:

     systemctl reset-failed nodelet.service
     rm -f /var/run/node-exporter-textfile/d8_early_time_sync.prom
     

     If nodelet.service is in the failed state, systemctl reset-failed keeps bashible from writing the failure metric again.

     On a Deckhouse Engine node, remove the file from a debug pod, as in step 1:

     d8 k -n kube-system debug node/NODE_NAME -i --profile=sysadmin --image=IMAGE -- chroot /host rm -f /var/run/node-exporter-textfile/d8_early_time_sync.prom
     

     Each debug command leaves a node-debugger-* pod in the kube-system namespace. Delete it with d8 k -n kube-system delete pod POD_NAME, using the name the command printed.

     The alert resolves about 10 minutes after the file is removed.

• NodeTimeOutOfSync CE S5
  Clock on the node NODE_NAME is drifting.

  Time on the node NODE_NAME is out of sync and drifts apart from the NTP server clock by VALUE seconds.

  To resolve the time synchronization issues:

  • Fix network errors:
    • Ensure the upstream time synchronization servers defined in the chrony configuration are available.
    • Eliminate large packet loss and excessive latency to upstream time synchronization servers.
  • Modify the NTP servers list defined in the chrony configuration.
• NTPDaemonOnNodeDoesNotSynchronizeTime CE S5
  NTP daemon on the node NODE_NAME haven't synchronized time for too long.

  Steps to troubleshoot:

  1. Check if the chrony pod is running on the node by executing the following command:

     d8 k -n d8-chrony --field-selector spec.nodeName="NODE_NAME" get pods
     

  2. Verify the chrony daemon’s status by executing the following command:

     d8 k -n d8-chrony exec POD_NAME -- /opt/chrony-static/bin/chronyc sources
     

  3. Resolve the time synchronization issues:

     • Fix network errors:
       • Ensure the upstream time synchronization servers defined in the chrony configuration are available.
       • Eliminate large packet loss and excessive latency to upstream time synchronization servers.
     • Modify the NTP servers list defined in the chrony configuration.

Module cloud-provider-aws

• D8CloudProviderAWSIMDSv2NotConverged CE S6
  Some EC2 instances of the cluster still allow IMDSv1.

  imdsv2: true is set in AWSClusterConfiguration, but VALUE EC2 instances of the cluster still accept metadata requests without a session token, which leaves them more exposed to unauthorized metadata access, including through SSRF attacks.

  Existing instances are not reconfigured by the setting alone:

  1. Master, static and bastion instances are updated by a converge run. Start the DKP installer and run dhctl converge.
  2. Ephemeral instances are re-created when their nodes are rolled, which happens on its own. Check that the nodes of every CloudEphemeral NodeGroup are being replaced: d8 k get nodegroups.

  The instance state is re-read once per discovery period of the cloud data discoverer (one hour by default), so the alert can lag behind the last node being replaced.

Module cloud-provider-huaweicloud

• D8HuaweiCloudMachineStuckInDeleting EE S6
  HuaweiCloudMachine NAME is stuck in Deleting for more than 24h.

  The HuaweiCloudMachine NAME in namespace NAMESPACE has been stuck in Deleting state for more than 24 hours.

  This usually means the deletion job in HuaweiCloud is failing or the API is unavailable.

  To inspect the machine status, run:

  d8 k get huaweicloudmachine -n NAMESPACE NAME -o yaml
  

  Check the controller logs for errors:

  d8 k logs -n d8-cloud-provider-huaweicloud -l app=caphc-controller-manager
  

Module cloud-provider-openstack

• OpenStackPreemptibleUnsupported EE S8
  NodeGroup NODE_GROUP_NAME requests preemptible instances, but this cluster cannot deliver them (reason: XXX).

  The OpenStackInstanceClass referenced by NodeGroup NODE_GROUP_NAME has spec.preemptible: true, but the raw Nova preemptible tag will not reach the virtual machines: nodes are created as regular (non-preemptible) VMs, and Selectel-side preemption never kicks in.

  XXX Reason: the NodeGroup runs on the MCM engine, which has no way to emit raw Nova tags — machine-controller-manager’s OpenStackMachineClass CRD does not expose the field. Only the CAPI engine (capi-controller-manager + capo-controller-manager) renders raw Nova tags via OpenStackMachineTemplate.spec.template.spec.tags.

  How to fix:

  1. Confirm the current engine and offending InstanceClass:

     d8 k get ng NODE_GROUP_NAME -o jsonpath='{.status.engine}{"\n"}'
     d8 k get ng NODE_GROUP_NAME -o jsonpath='{.spec.cloudInstances.classReference}{"\n"}'
     

  2. Choose one of the following actions:

     • Recommended. Remove the node.deckhouse.io/use-mcm annotation from the NodeGroup so it migrates to CAPI. Nodes will be recreated on the CAPI engine, this time carrying the preemptible tag:

       d8 k annotate ng NODE_GROUP_NAME node.deckhouse.io/use-mcm-
       

     • Alternatively, if this NodeGroup must stay on MCM, set spec.preemptible: false on the OpenStackInstanceClass (nodes stay regular, the alert clears):

       d8 k patch openstackinstanceclass CLASS_NAME --type=merge -p '{"spec":{"preemptible":false}}'
       

       Note: a single OpenStackInstanceClass shared between a CAPI NodeGroup (where the field works) and an MCM one (where it doesn’t) will keep firing this alert on the MCM side. Use separate InstanceClasses per engine when the classes differ in preemptible.

  XXX Reason: the cluster’s connection.authURL does not point at Selectel (selcloud.ru / selectel). The raw Nova preemptible tag is a Selectel-specific mechanism; on other OpenStack providers Nova accepts the tag but ignores it, so instances are billed and behave as regular non-preemptible VMs.

  How to fix:

  1. Confirm the offending InstanceClass and the cluster’s authURL:

     d8 k get ng NODE_GROUP_NAME -o jsonpath='{.spec.cloudInstances.classReference}{"\n"}'
     d8 k -n kube-system get secret d8-node-manager-cloud-provider -o jsonpath='{.data.openstack}' | base64 -d | jq -r '.connection.authURL'
     

  2. Set spec.preemptible: false on the OpenStackInstanceClass — the field only makes sense on Selectel-hosted clusters:

     d8 k patch openstackinstanceclass CLASS_NAME --type=merge -p '{"spec":{"preemptible":false}}'
     

     The alert clears on the next scrape after the change; existing nodes stay running (they are already regular VMs — the tag was never emitted).

  XXX Reason: unknown (XXX). This alert should surface only two well-known cases (mcm, non-selectel). A new reason implies node-controller has been extended without this runbook — check the release notes for cloud-provider-openstack before deciding what to do. XXX

Module cloud-provider-yandex

• D8CloudProviderYandexCloudMigrationPending CE S6
  The cloud-provider-yandex module is waiting for migration resources to be applied.

  The cloud-provider-yandex module has detected the legacy YandexProviderClusterConfiguration secret in the cluster and is waiting for the new migration resources (ModuleConfig v2, CredentialSecret, NodeGroups, InstanceClasses) to be applied before completing the migration.

  To complete the migration, apply the resources from the d8-migration-resources Secret:

  d8 k -n d8-cloud-provider-yandex get secret d8-migration-resources -o jsonpath='{.data.resources\.yaml}' | base64 -d | d8 k apply -f -
  

  After applying the resources, the alert will disappear automatically once the module detects that all required resources are in place.

  After the migration is complete, the legacy YandexProviderClusterConfiguration secret is no longer needed and can be deleted manually:

  1. Make sure the cloud discovery data is stored outside the legacy secret. The module reads it from the d8-candi-cloud-provider-discovery-data Secret first and uses the legacy secret only as a fallback, so the following command must print a non-empty value:

     d8 k -n d8-cloud-provider-yandex get secret d8-candi-cloud-provider-discovery-data -o jsonpath='{.data.cloud-provider-discovery-data\.json}'
     

     If it prints nothing, the discovery data exists only in the legacy secret, and deleting that secret would lose it. Copy the payload over — the key holds exactly the same discovery data dhctl recorded from the infrastructure run:

     d8 k -n kube-system get secret d8-provider-cluster-configuration \
       -o jsonpath='{.data.cloud-provider-discovery-data\.json}' | base64 -d > /tmp/yandex-discovery.json
     
     d8 k -n d8-cloud-provider-yandex create secret generic d8-candi-cloud-provider-discovery-data \
       --from-file=cloud-provider-discovery-data.json=/tmp/yandex-discovery.json \
       --dry-run=client -o yaml | d8 k apply -f -
     

  2. Back up the secret. It contains cloud credentials and discovery data, so store the file securely:

     d8 k -n kube-system get secret d8-provider-cluster-configuration -o yaml > d8-provider-cluster-configuration-backup.yaml
     

  3. Allow the deletion. The secret is protected by an admission webhook and cannot be deleted without the annotation:

     d8 k -n kube-system annotate secret d8-provider-cluster-configuration deckhouse.io/allow-delete=true
     

  4. Delete the secret:

     d8 k -n kube-system delete secret d8-provider-cluster-configuration
     

• D8YandexNatInstanceConnectionsQuotaUtilization CE S4
  Connection quota utilization of the Yandex NAT instance exceeds 85% over the last 5 minutes.

  The connection quota for the Yandex NAT instance has exceeded 85% utilization over the past 5 minutes.

  To prevent potential issues, contact Yandex technical support and request an increase in the connection quota.

• NATInstanceWithDeprecatedAvailabilityZone CE S9
  NAT instance NAME is in a deprecated availability zone.

  The NAT instance NAME is located in the availability zone ru-central1-c, which has been deprecated by Yandex Cloud. To resolve this issue, migrate the NAT instance to either ru-central1-a or ru-central1-b by following the instructions below.

  The migration process involves irreversible changes and may result in a significant downtime. The duration typically depends on Yandex Cloud’s response time and can last several tens of minutes.

  1. Migrate the NAT instance. To get providerClusterConfiguration.withNATInstance, run the following command:

     d8 k -n d8-system exec -ti svc/deckhouse-leader -c deckhouse -- deckhouse-controller module values -g cloud-provider-yandex -o json | jq -c | jq '.cloudProviderYandex.internal.providerClusterConfiguration.withNATInstance'
     

     • If you specified withNATInstance.natInstanceInternalAddress and/or withNATInstance.internalSubnetID in providerClusterConfiguration, remove them using the following command:

       d8 system edit provider-cluster-configuration
       

     • If you specified withNATInstance.externalSubnetID and/or withNATInstance.natInstanceExternalAddress in providerClusterConfiguration, change them to the appropriate values.

       To get the address and subnet ID, use the Yandex Cloud Console or CLI.

       To change withNATInstance.externalSubnetID and withNATInstance.natInstanceExternalAddress, run the following command:

       d8 system edit provider-cluster-configuration
       

  2. Run the appropriate edition and version of the Deckhouse installer container on the local machine. You may have to change the container registry address to do that. After that, perform the converge.

     • Get the appropriate edition and version of Deckhouse:

       DH_VERSION=$(d8 k -n d8-system get deployment deckhouse -o jsonpath='{.metadata.annotations.core\.deckhouse\.io\/version}')
       DH_EDITION=$(d8 k -n d8-system get deployment deckhouse -o jsonpath='{.metadata.annotations.core\.deckhouse\.io\/edition}' | tr '[:upper:]' '[:lower:]')
       echo "DH_VERSION=$DH_VERSION DH_EDITION=$DH_EDITION"
       

     • Run the installer:

       docker run --pull=always -it -v "$HOME/.ssh/:/tmp/.ssh/" registry.deckhouse.io/deckhouse/${DH_EDITION}/install:${DH_VERSION} bash
       

     • Perform the converge:

       dhctl converge --ssh-agent-private-keys=/tmp/.ssh/SSH_KEY_FILENAME --ssh-user=USERNAME --ssh-host MASTER-NODE-0-HOST
       

  3. Update the route table.

     • Get the route table name:

       d8 k -n d8-system exec -ti svc/deckhouse-leader -c deckhouse -- deckhouse-controller module values -g cloud-provider-yandex -o json | jq -c | jq '.global.prefix // .global.clusterConfiguration.cloud.prefix'
       

     • Get the NAT instance name:

       d8 k -n d8-system exec -ti svc/deckhouse-leader -c deckhouse -- deckhouse-controller module values -g cloud-provider-yandex -o json | jq -c | jq '.cloudProviderYandex.internal.providerDiscoveryData.natInstanceName'
       

     • Get the NAT instance internal IP address:

       yc compute instance list | grep -e "INTERNAL IP" -e NAT_INSTANCE_NAME_FROM_PREVIOUS_STEP
       

     • Update the route:

       yc vpc route-table update --name ROUTE_TABLE_NAME_FROM_PREVIOUS_STEP --route "destination=0.0.0.0/0,next-hop=NAT_INSTANCE_INTERNAL_IP_FROM_PREVIOUS_STEP"
       

• NodeGroupNodeWithDeprecatedAvailabilityZone CE S9
  NodeGroup NODE_GROUP_NAME contains nodes in a deprecated availability zone.

  Certain nodes in the node group NODE_GROUP_NAME are located in the availability zone ru-central1-c, which has been deprecated by Yandex Cloud.

  Steps to troubleshoot:

  1. Identify the nodes that need to be migrated by running the following command:

     d8 k get node -l "topology.kubernetes.io/zone=ru-central1-c"
     

  2. Migrate your nodes, disks, and load balancers to one of the supported zones: ru-central1-a, ru-central1-b, or ru-central1-d. Refer to the Yandex migration guide for detailed instructions.

     You can’t migrate public IP addresses between zones. For details, refer to the migration guide.

Module cloud-provider-zvirt

• D8CloudProviderZvirtMigrationPending SE-PLUS S6
  The cloud-provider-zvirt module is waiting for migration resources to be applied.

  The cloud-provider-zvirt module has detected the legacy ZvirtClusterConfiguration secret in the cluster and is waiting for the new migration resources (ModuleConfig v2, CredentialSecret, NodeGroups, InstanceClasses) to be applied before completing the migration.

  To complete the migration, apply the resources from the d8-migration-resources Secret:

  d8 k -n d8-cloud-provider-zvirt get secret d8-migration-resources -o jsonpath='{.data.resources\.yaml}' | base64 -d | d8 k apply -f -
  

  After applying the resources, the alert will disappear automatically once the module detects that all required resources are in place.

  After the migration is complete, the legacy ZvirtProviderClusterConfiguration secret is no longer needed and can be deleted manually:

  1. Make sure the cloud discovery data is stored outside the legacy secret. The module reads it from the d8-candi-cloud-provider-discovery-data Secret first and uses the legacy secret only as a fallback, so the following command must print a non-empty value:

     d8 k -n d8-cloud-provider-zvirt get secret d8-candi-cloud-provider-discovery-data -o jsonpath='{.data.cloud-provider-discovery-data\.json}'
     

     If it prints nothing, the discovery data exists only in the legacy secret, and deleting that secret would lose it. Copy the payload over — the key holds exactly the same discovery data dhctl recorded from the infrastructure run:

     d8 k -n kube-system get secret d8-provider-cluster-configuration \
       -o jsonpath='{.data.cloud-provider-discovery-data\.json}' | base64 -d > /tmp/zvirt-discovery.json
     
     d8 k -n d8-cloud-provider-zvirt create secret generic d8-candi-cloud-provider-discovery-data \
       --from-file=cloud-provider-discovery-data.json=/tmp/zvirt-discovery.json \
       --dry-run=client -o yaml | d8 k apply -f -
     

  2. Back up the secret. It contains cloud credentials and discovery data, so store the file securely:

     d8 k -n kube-system get secret d8-provider-cluster-configuration -o yaml > d8-provider-cluster-configuration-backup.yaml
     

  3. Allow the deletion. The secret is protected by an admission webhook and cannot be deleted without the annotation:

     d8 k -n kube-system annotate secret d8-provider-cluster-configuration deckhouse.io/allow-delete=true
     

  4. Delete the secret:

     d8 k -n kube-system delete secret d8-provider-cluster-configuration
     

Module cni-cilium

• CiliumAgentEndpointsNotReady CE S4
  Over 50% of all known endpoints aren't ready in agent NAMESPACE/POD_NAME.

  For details, refer to the logs of the agent:

  d8 k -n NAMESPACE logs POD_NAME
  

• CiliumAgentMapPressureCritical CE S4
  eBPF map MAP_NAME exceeds 90% utilization in agent NAMESPACE/POD_NAME.

  The eBPF map resource utilization limit has almost been reached.

  Check with the vendor for potential remediation steps.

• CiliumAgentMetricNotFound CE S4
  Agent NAMESPACE/POD_NAME isn't sending some metrics.

  Steps to troubleshoot:

  1. Check the logs of the agent:

     d8 k -n NAMESPACE logs POD_NAME
     

  2. Verify the agent’s health status:

     d8 k -n NAMESPACE exec -ti POD_NAME cilium-health status
     

  3. Compare the metrics with those of a neighboring agent.

  Note that the absence of metrics can indirectly indicate that new pods can’t be created on the node due to connectivity issues with the agent.

• CiliumAgentPolicyImportErrors CE S4
  Agent NAMESPACE/POD_NAME fails to import policies.

  For details, refer to the logs of the agent:

  d8 k -n NAMESPACE logs POD_NAME
  

• CiliumAgentUnreachableHealthEndpoints CE S4
  Agent NAMESPACE/POD_NAME can't reach some of the node health endpoints.

  For details, query Cilium health status API:

  d8 k -n NAMESPACE exec POD_NAME -- cilium-health status
  

  You can also refer to the agent logs:

  d8 k -n NAMESPACE logs POD_NAME
  

• CniCiliumClusterMeshPeeringModuleDisabled EE S4
  ClusterMesh is enabled, but the peering module is disabled.

  ClusterMesh connects the clusters peered by the peering module. With the module disabled, no new mesh with a peer comes up, and the meshes already configured are not updated.

  To resolve this issue, enable the peering module, or set clusterMesh.enabled: false in the cni-cilium ModuleConfig.

• CniCiliumNonStandardVXLANPortFound CE S4
  Cilium configuration uses a non-standard VXLAN port.

  The Cilium configuration specifies a non-standard VXLAN port XXX. The recommended port number is XXX.

  To resolve this issue, update the tunnel-port parameter in the cilium-configmap ConfigMap located in the d8-cni-cilium namespace to match the recommended port.

  If you configured the non-standard port on purpose, ignore this alert.

• CniCiliumOrphanEgressGatewayPolicyFound SE-PLUS S4
  Orphaned EgressGatewayPolicy with an irrelevant EgressGateway name has been found.

  The cluster contains an orphaned EgressGatewayPolicy named NAME with an irrelevant EgressGateway name.

  To resolve this issue, verify the EgressGateway name specified in the EgressGatewayPolicy resource EGRESSGATEWAY and update it as needed.

• CNICiliumPolicyWithRequiresFieldsFound CE S4
  The XXX XXXNAMESPACE/XXXNAME uses the fromRequires or toRequires fields.

  The XXX NAMEXXX in the NAMESPACE namespaceXXX uses the fromRequires or toRequires fields.

  Cilium 1.19 and later doesn’t support these fields. The Cilium agent of these versions can’t read such policies and fails on all nodes. Deckhouse won’t deploy a release with such a Cilium version until all these policies are rewritten.

  To resolve this issue, rewrite each fromRequires constraint as an ingressDeny rule with an inverted selector. Don’t move the selector into the deny rule as is: this inverts the policy. A deny rule wins over allow rules of any policy, so it keeps the original scope: fromRequires applied to all policies selecting the endpoint.

  For example, fromRequires with matchLabels: {env: prod} becomes:

  ingressDeny:
  - fromEndpoints:
    - matchExpressions:
      - {key: env, operator: NotIn, values: [prod]}
  XXX
    - matchExpressions:
      - {key: k8s:io.kubernetes.pod.namespace, operator: Exists}
      - {key: k8s:io.kubernetes.pod.namespace, operator: NotIn, values: [NAMESPACE]}
  XXX
  

  • Add a separate NotIn selector for each label of the constraint. XXX
  • Keep the second selector: in a CiliumNetworkPolicy, fromRequires also implied the policy namespace, so sources from other namespaces were rejected. The Exists condition is required, otherwise the rule also blocks the host, the world, and other sources without a namespace. XXX
  • Rewrite toRequires the same way, as an egressDeny rule with toEndpoints.
  • Add the deny rule first, check the traffic, and only then remove fromRequires and toRequires.

  To list all affected policies, use the d8_cni_cilium_policy_with_requires_fields == 1 expression in Prometheus.

• D8IPv4OptionsGuardAuditMode CE S9
  The CVE-2026-98432 filter only counts packets on VALUE nodes of the cluster.

  The ipv4OptionsGuard.mode of the CNI module is Audit on VALUE nodes: packets with IP options are only counted, not dropped, and MPTCP stays enabled. The nodes remain exposed to CVE-2026-98432.

  Clusters that existed before the mitigation get Audit by default. Once the D8IPv4OptionsGuardMatchedTraffic alert shows that no workload relies on IP options, switch to Enforce in the ModuleConfig of the CNI module (cni-cilium, cni-flannel or cni-simple-bridge):

  spec:
    settings:
      ipv4OptionsGuard:
        mode: Enforce
  

• D8IPv4OptionsGuardBackendMismatch CE S6
  The CVE-2026-98432 filter on the node NODE_NAME uses a different iptables mode than iptables-wrapper selected.

  The node NODE_NAME is protected, but the filter uses XXX: the kernel modules of the mode selected by iptables-wrapper are not loaded, while those of the other mode are.

  The details are in the logs of the ipv4-options-guard container of the CNI agent pod on that node.

  The filter and the other Deckhouse components use different packet filtering frameworks on this node. Such a security configuration is harder to audit and troubleshoot.

  Possible actions:

  • Find out why the kernel modules of the mode selected by iptables-wrapper are not loaded on the node.
  • If loading these kernel modules is acceptable, set ipv4OptionsGuard.allowKernelModuleLoad: true in the CNI module settings.
• D8IPv4OptionsGuardFilterBypassed CE S6
  Packets can skip the CVE-2026-98432 filter in VALUE network namespaces on the node NODE_NAME.

  The node NODE_NAME uses legacy iptables. In VALUE network namespaces, a rule in the PREROUTING chain of the raw table ahead of the jump to D8-IPV4OPT-GUARD can end the traversal of the table (for example, -j ACCEPT). Packets accepted by that rule never reach the filter.

  The guard puts its jump first when it installs the filter, but does not move it later, so as not to fight with other software on the node that manages its own rules. The affected network namespaces are named in the logs of the ipv4-options-guard container of the CNI agent pod on that node.

  Possible actions:

  • Find the software that inserted the rule (iptables-legacy -t raw -S PREROUTING in the network namespace) and make it append the rule instead, or use a target that does not end the traversal.
  • Or put the jump first by hand: iptables-legacy -t raw -I PREROUTING 1 -j D8-IPV4OPT-GUARD. The guard removes the old jump on its next pass.
• D8IPv4OptionsGuardMatchedTraffic CE S7
  IPv4 packets with IP options were seen on the node NODE_NAME in the Audit mode.

  Over the last hour, the ipv4-options-guard sidecar matched VALUE IPv4 packets with IP options (rule XXX) on the node NODE_NAME.

  In the Audit mode they are only counted. In the Enforce mode they would be dropped, and this alert is not raised.

  Legitimate TCP traffic almost never carries IP options. If a workload relies on them (for example, CIPSO labeled networking), exclude the node with the ipv4-options-guard.deckhouse.io/disabled: "true" label before switching ipv4OptionsGuard.mode to Enforce. Otherwise, this may be an attempt to exploit CVE-2026-98432.

• D8IPv4OptionsGuardMetricsAbsent CE S4
  The CVE-2026-98432 filter state is unknown on the node NODE_NAME.

  The CNI agent pod runs on the node NODE_NAME, but Prometheus gets no metrics from its ipv4-options-guard container.

  Without these metrics, the other D8IPv4OptionsGuard* alerts cannot fire for this node, so it may be unprotected unnoticed.

  Check the ipv4-options-guard container of the CNI agent pod on that node and the scraping of its metrics.

• D8IPv4OptionsGuardNodeUnprotected CE S4
  The CVE-2026-98432 filter cannot be installed on the node NODE_NAME.

  The ipv4-options-guard sidecar of the CNI agent could not protect the node NODE_NAME (reason: XXX).

  • iptables_wrapper_failed — iptables-wrapper could not determine the iptables mode of the node.
  • kernel_modules_not_loaded — the kernel modules of the mode selected by iptables-wrapper are not loaded, and loading them is not allowed.

  The details are in the logs of the ipv4-options-guard container of the CNI agent pod on that node.

  The node remains exposed to CVE-2026-98432 until its kernel is updated.

  Possible actions:

  • Update the node kernel to a version with the fix and reboot the node.
  • If the node uses legacy iptables and loading the kernel modules named in the reason is acceptable, set ipv4OptionsGuard.allowKernelModuleLoad: true in the CNI module settings.
• D8IPv4OptionsGuardReconcileFailed CE S6
  The CVE-2026-98432 filter failed in VALUE network namespaces on the node NODE_NAME.

  Some network namespaces on the node NODE_NAME are not protected.

  Check the logs of the ipv4-options-guard container of the CNI agent pod on that node.

• D8NodeHasUnmetKernelRequirements CE S4
  Nodes have unmet kernel requirements.

  Deckhouse has detected that some nodes don’t meet the required kernel constraints. As a result, certain modules can’t run on these nodes.

  Kernel requirements for each module:

  • Cilium module: Kernel version must be >= 5.8.

  To list all affected nodes, use the d8_node_kernel_does_not_satisfy_requirements == 1 expression in Prometheus.

Module cni-flannel

• D8IPv4OptionsGuardAuditMode CE S9
  The CVE-2026-98432 filter only counts packets on VALUE nodes of the cluster.

  The ipv4OptionsGuard.mode of the CNI module is Audit on VALUE nodes: packets with IP options are only counted, not dropped, and MPTCP stays enabled. The nodes remain exposed to CVE-2026-98432.

  Clusters that existed before the mitigation get Audit by default. Once the D8IPv4OptionsGuardMatchedTraffic alert shows that no workload relies on IP options, switch to Enforce in the ModuleConfig of the CNI module (cni-cilium, cni-flannel or cni-simple-bridge):

  spec:
    settings:
      ipv4OptionsGuard:
        mode: Enforce
  

• D8IPv4OptionsGuardBackendMismatch CE S6
  The CVE-2026-98432 filter on the node NODE_NAME uses a different iptables mode than iptables-wrapper selected.

  The node NODE_NAME is protected, but the filter uses XXX: the kernel modules of the mode selected by iptables-wrapper are not loaded, while those of the other mode are.

  The details are in the logs of the ipv4-options-guard container of the CNI agent pod on that node.

  The filter and the other Deckhouse components use different packet filtering frameworks on this node. Such a security configuration is harder to audit and troubleshoot.

  Possible actions:

  • Find out why the kernel modules of the mode selected by iptables-wrapper are not loaded on the node.
  • If loading these kernel modules is acceptable, set ipv4OptionsGuard.allowKernelModuleLoad: true in the CNI module settings.
• D8IPv4OptionsGuardFilterBypassed CE S6
  Packets can skip the CVE-2026-98432 filter in VALUE network namespaces on the node NODE_NAME.

  The node NODE_NAME uses legacy iptables. In VALUE network namespaces, a rule in the PREROUTING chain of the raw table ahead of the jump to D8-IPV4OPT-GUARD can end the traversal of the table (for example, -j ACCEPT). Packets accepted by that rule never reach the filter.

  The guard puts its jump first when it installs the filter, but does not move it later, so as not to fight with other software on the node that manages its own rules. The affected network namespaces are named in the logs of the ipv4-options-guard container of the CNI agent pod on that node.

  Possible actions:

  • Find the software that inserted the rule (iptables-legacy -t raw -S PREROUTING in the network namespace) and make it append the rule instead, or use a target that does not end the traversal.
  • Or put the jump first by hand: iptables-legacy -t raw -I PREROUTING 1 -j D8-IPV4OPT-GUARD. The guard removes the old jump on its next pass.
• D8IPv4OptionsGuardMatchedTraffic CE S7
  IPv4 packets with IP options were seen on the node NODE_NAME in the Audit mode.

  Over the last hour, the ipv4-options-guard sidecar matched VALUE IPv4 packets with IP options (rule XXX) on the node NODE_NAME.

  In the Audit mode they are only counted. In the Enforce mode they would be dropped, and this alert is not raised.

  Legitimate TCP traffic almost never carries IP options. If a workload relies on them (for example, CIPSO labeled networking), exclude the node with the ipv4-options-guard.deckhouse.io/disabled: "true" label before switching ipv4OptionsGuard.mode to Enforce. Otherwise, this may be an attempt to exploit CVE-2026-98432.

• D8IPv4OptionsGuardMetricsAbsent CE S4
  The CVE-2026-98432 filter state is unknown on the node NODE_NAME.

  The CNI agent pod runs on the node NODE_NAME, but Prometheus gets no metrics from its ipv4-options-guard container.

  Without these metrics, the other D8IPv4OptionsGuard* alerts cannot fire for this node, so it may be unprotected unnoticed.

  Check the ipv4-options-guard container of the CNI agent pod on that node and the scraping of its metrics.

• D8IPv4OptionsGuardNodeUnprotected CE S4
  The CVE-2026-98432 filter cannot be installed on the node NODE_NAME.

  The ipv4-options-guard sidecar of the CNI agent could not protect the node NODE_NAME (reason: XXX).

  • iptables_wrapper_failed — iptables-wrapper could not determine the iptables mode of the node.
  • kernel_modules_not_loaded — the kernel modules of the mode selected by iptables-wrapper are not loaded, and loading them is not allowed.

  The details are in the logs of the ipv4-options-guard container of the CNI agent pod on that node.

  The node remains exposed to CVE-2026-98432 until its kernel is updated.

  Possible actions:

  • Update the node kernel to a version with the fix and reboot the node.
  • If the node uses legacy iptables and loading the kernel modules named in the reason is acceptable, set ipv4OptionsGuard.allowKernelModuleLoad: true in the CNI module settings.
• D8IPv4OptionsGuardReconcileFailed CE S6
  The CVE-2026-98432 filter failed in VALUE network namespaces on the node NODE_NAME.

  Some network namespaces on the node NODE_NAME are not protected.

  Check the logs of the ipv4-options-guard container of the CNI agent pod on that node.

Module cni-simple-bridge

• D8IPv4OptionsGuardAuditMode CE S9
  The CVE-2026-98432 filter only counts packets on VALUE nodes of the cluster.

  The ipv4OptionsGuard.mode of the CNI module is Audit on VALUE nodes: packets with IP options are only counted, not dropped, and MPTCP stays enabled. The nodes remain exposed to CVE-2026-98432.

  Clusters that existed before the mitigation get Audit by default. Once the D8IPv4OptionsGuardMatchedTraffic alert shows that no workload relies on IP options, switch to Enforce in the ModuleConfig of the CNI module (cni-cilium, cni-flannel or cni-simple-bridge):

  spec:
    settings:
      ipv4OptionsGuard:
        mode: Enforce
  

• D8IPv4OptionsGuardBackendMismatch CE S6
  The CVE-2026-98432 filter on the node NODE_NAME uses a different iptables mode than iptables-wrapper selected.

  The node NODE_NAME is protected, but the filter uses XXX: the kernel modules of the mode selected by iptables-wrapper are not loaded, while those of the other mode are.

  The details are in the logs of the ipv4-options-guard container of the CNI agent pod on that node.

  The filter and the other Deckhouse components use different packet filtering frameworks on this node. Such a security configuration is harder to audit and troubleshoot.

  Possible actions:

  • Find out why the kernel modules of the mode selected by iptables-wrapper are not loaded on the node.
  • If loading these kernel modules is acceptable, set ipv4OptionsGuard.allowKernelModuleLoad: true in the CNI module settings.
• D8IPv4OptionsGuardFilterBypassed CE S6
  Packets can skip the CVE-2026-98432 filter in VALUE network namespaces on the node NODE_NAME.

  The node NODE_NAME uses legacy iptables. In VALUE network namespaces, a rule in the PREROUTING chain of the raw table ahead of the jump to D8-IPV4OPT-GUARD can end the traversal of the table (for example, -j ACCEPT). Packets accepted by that rule never reach the filter.

  The guard puts its jump first when it installs the filter, but does not move it later, so as not to fight with other software on the node that manages its own rules. The affected network namespaces are named in the logs of the ipv4-options-guard container of the CNI agent pod on that node.

  Possible actions:

  • Find the software that inserted the rule (iptables-legacy -t raw -S PREROUTING in the network namespace) and make it append the rule instead, or use a target that does not end the traversal.
  • Or put the jump first by hand: iptables-legacy -t raw -I PREROUTING 1 -j D8-IPV4OPT-GUARD. The guard removes the old jump on its next pass.
• D8IPv4OptionsGuardMatchedTraffic CE S7
  IPv4 packets with IP options were seen on the node NODE_NAME in the Audit mode.

  Over the last hour, the ipv4-options-guard sidecar matched VALUE IPv4 packets with IP options (rule XXX) on the node NODE_NAME.

  In the Audit mode they are only counted. In the Enforce mode they would be dropped, and this alert is not raised.

  Legitimate TCP traffic almost never carries IP options. If a workload relies on them (for example, CIPSO labeled networking), exclude the node with the ipv4-options-guard.deckhouse.io/disabled: "true" label before switching ipv4OptionsGuard.mode to Enforce. Otherwise, this may be an attempt to exploit CVE-2026-98432.

• D8IPv4OptionsGuardMetricsAbsent CE S4
  The CVE-2026-98432 filter state is unknown on the node NODE_NAME.

  The CNI agent pod runs on the node NODE_NAME, but Prometheus gets no metrics from its ipv4-options-guard container.

  Without these metrics, the other D8IPv4OptionsGuard* alerts cannot fire for this node, so it may be unprotected unnoticed.

  Check the ipv4-options-guard container of the CNI agent pod on that node and the scraping of its metrics.

• D8IPv4OptionsGuardNodeUnprotected CE S4
  The CVE-2026-98432 filter cannot be installed on the node NODE_NAME.

  The ipv4-options-guard sidecar of the CNI agent could not protect the node NODE_NAME (reason: XXX).

  • iptables_wrapper_failed — iptables-wrapper could not determine the iptables mode of the node.
  • kernel_modules_not_loaded — the kernel modules of the mode selected by iptables-wrapper are not loaded, and loading them is not allowed.

  The details are in the logs of the ipv4-options-guard container of the CNI agent pod on that node.

  The node remains exposed to CVE-2026-98432 until its kernel is updated.

  Possible actions:

  • Update the node kernel to a version with the fix and reboot the node.
  • If the node uses legacy iptables and loading the kernel modules named in the reason is acceptable, set ipv4OptionsGuard.allowKernelModuleLoad: true in the CNI module settings.
• D8IPv4OptionsGuardReconcileFailed CE S6
  The CVE-2026-98432 filter failed in VALUE network namespaces on the node NODE_NAME.

  Some network namespaces on the node NODE_NAME are not protected.

  Check the logs of the ipv4-options-guard container of the CNI agent pod on that node.

Module control-plane-manager

• D8ControlPlaneComponentsResourcesInsufficientCapacity CE S5
  Automatic control-plane resource tuning cannot increase XXX requests due to lack of available capacity.

  Automatic control-plane resource tuning attempted to increase XXX requests for one or more control-plane components (kube-apiserver, etcd, kube-controller-manager, kube-scheduler), but the calculated requests exceed the available capacity of at least one master node after accounting for the kubelet reservation and resource requests of other (non-control-plane) pods already scheduled on the node.

  XXX Additional memory required: XXXB. XXX Additional CPU required: VALUEm CPU. XXX

  To resolve this, free up resources on master nodes by moving or reducing other workloads or increase the CPU or RAM available on the nodes.

  To disable automatic tuning for this resource and manage requests manually, set spec.settings.resourcesRequests.XXX in the ModuleConfig of control-plane-manager.

• D8ControlPlaneDefaultVersionDrift CE S5
  Control plane Kubernetes version rollback is blocked (Default soft-guard freeze).

  The cluster tracks the Deckhouse default Kubernetes version (kubernetesVersion: Default in ModuleConfig control-plane-manager, or unset ModuleConfig with no ClusterConfiguration pin). That default is now more than 1 minor below the highest version this cluster has ever run (maxUsed).

  Rollback blocked: Deckhouse freezes the published target (holds it at the current control-plane version, or at the last desired version when the current one is not published yet) and does not roll Default down toward the lower default. Control plane and kubelet stay on the frozen target until the Deckhouse default catches up within the maxUsed−1 window.

  Check the frozen label on the metric: frozen="true" means the digit is held as described above. frozen="false" means the window is violated but nothing was left to hold the digit at (for example the d8-cluster-kubernetes ConfigMap was deleted), so the published target has moved down to the Deckhouse default — pin the intended version explicitly.

  To resolve this alert:

  • Do nothing and wait, if you are happy staying on the version the cluster runs now. This is a valid resolution only while frozen="true": the cluster is not degraded, the version is held, and the alert clears by itself once the Deckhouse default rises back into the maxUsed−1 window. How long that takes depends on why the default dropped — after a Deckhouse rollback, until you upgrade Deckhouse again; when the release default is simply older than the version this cluster runs, until a Deckhouse release ships a high enough default, which may be several releases away.

  • To keep the current (higher) version and stop the alert now, pin it explicitly:

    d8 k patch mc control-plane-manager --type merge -p '{"spec":{"settings":{"kubernetesVersion":"current version"}}}'
    

  • To downgrade deliberately, pin one minor at a time (more than 1 minor below maxUsed is rejected):

    d8 k patch mc control-plane-manager --type merge -p '{"spec":{"settings":{"kubernetesVersion":"current version - 1 minor"}}}'
    

• D8ControlPlaneManagerObsoleteMonitoringKubernetesControlPlaneModuleConfig CE S9
  Obsolete monitoring-kubernetes-control-plane ModuleConfig should be removed.

  The monitoring-kubernetes-control-plane module was merged into the control-plane-manager module, but its ModuleConfig still exists in the cluster and is no longer used.

  Remove it manually:

  d8 k delete moduleconfig monitoring-kubernetes-control-plane
  

  If the ModuleConfig is deployed from an external source (e.g. a Git repository), make sure it is removed from there so it is not created again.

• D8ControlPlaneManagerPodNotRunning CE S6
  Controller pod isn't running on node NODE_NAME.

  The d8-control-plane-manager pod is either failing or hasn’t been scheduled on node NODE_NAME.

  To resolve this issue, check the status of the kube-system/d8-control-plane-manager DaemonSet and its pods by running the following command:

  d8 k -n kube-system get daemonset,pod --selector=app=d8-control-plane-manager
  

• D8ControlPlaneNodeMaintenanceModeEnabled CE S4
  Maintenance mode is enabled for control-plane node NODE_NAME.

  Control-plane node NODE_NAME has maintenance mode enabled with label control-plane-manager.deckhouse.io/maintenance. While this label is present, the control plane is not managed by control-plane-manager.

  If maintenance is complete, remove the label:

  d8 k label cpn NODE_NAME control-plane-manager.deckhouse.io/maintenance-
  

• D8ControlPlaneOperationRunningTooLong CE S4
  Control-plane operation XXX for COMPONENT_NAME on node NODE_NAME runs or retries longer than 10 minutes.

  Control-plane operation XXX for component COMPONENT_NAME on node NODE_NAME has been running or retrying for more than 10 minutes.

  Check the corresponding ControlPlaneOperation and logs of the d8-control-plane-manager Pod on that node:

  d8 k get cpo XXX -o yaml
  d8 k -n kube-system logs -l app=d8-control-plane-manager --field-selector spec.nodeName=NODE_NAME
  

• D8EtcdDatabaseHighFragmentationRatio CE S7
  etcd database size in use is less than 50% of the allocated storage.

  The etcd database size in use on instance INSTANCE_NAME is less than 50% of the allocated disk space, indicating potential fragmentation. Additionally, the total storage size exceeds 75% of the configured quota.

  To resolve this issue, defragment the etcd database by running the following command:

  d8 k -n kube-system exec -ti etcd-NODE_NAME -- /usr/bin/etcdctl \
    --cacert /etc/kubernetes/pki/etcd/ca.crt \
    --cert /etc/kubernetes/pki/etcd/ca.crt \
    --key /etc/kubernetes/pki/etcd/ca.key \
    --endpoints https://127.0.0.1:2379/ defrag --command-timeout=30s
  

• D8EtcdExcessiveDatabaseGrowth CE S4
  etcd cluster database is growing rapidly.

  Based on the growth rate observed over the last six hours, Deckhouse predicts that the etcd database will run out of disk space within one day on instance INSTANCE_NAME.

  To prevent disruptions, investigate the cause and take necessary action.

• D8EtcdHighRequestsLatency CE S5
  Etcd request latency (p99) has exceeded 800 ms multiple times in the past hour.

  The 99th percentile of etcd request duration on INSTANCE_NAME has exceeded 800 ms more than twice in the last hour.

  High etcd latency can cause API server timeouts, slow admission webhooks, and general cluster instability.

  Investigate the following:

  • Node disk I/O: Etcd is extremely sensitive to disk performance. Check disk utilization and I/O wait on the control plane node.
  • Network-attached storage: If the etcd data directory resides on NFS, Ceph, or another network storage, check connectivity and latency to the storage backend. Etcd requires a consistently low-latency local disk (SSD recommended).
  • Control plane summary: Review the Kubernetes Cluster → Control Plane Status Grafana dashboard for an overall picture of API server and etcd health, request rates, and error rates.
  • Node base performance: Check node CPU and RAM utilization and load average. If the node is CPU-bound (load average significantly exceeds the number of CPU cores), consider scaling up the control plane node.
• D8InvalidEtcdDefragCronSchedule CE S6
  Invalid etcd defragmentation cron schedule in ModuleConfig.

  The controlPlaneManager.etcd.defrag.cronSchedule value XXX is not a valid cron expression.

  Deckhouse has fallen back to the default schedule (0 1 * * *) to keep defragmentation running.

  To fix this, set a valid 5-field cron expression:

  d8 k patch moduleconfig control-plane-manager --type merge \
    -p '{"spec":{"settings":{"etcd":{"defrag":{"cronSchedule":"0 1 * * *"}}}}}'
  

  Valid examples:

  • 0 1 * * * — daily at 01:00 UTC
  • 0 */6 * * * — every 6 hours
  • 30 2 * * 0 — weekly on Sunday at 02:30 UTC
• D8KubeEtcdDatabaseSizeCloseToTheLimit CE S3
  etcd database size is approaching the limit.

  The etcd database size on NODE_NAME is nearing its size limit. This may be caused by a high number of events, such as pod evictions or the recent creation of numerous resources in the cluster.

  Possible solutions:

  • Defragment the etcd database by running the following command:

    d8 k -n kube-system exec -ti etcd-NODE_NAME -- /usr/bin/etcdctl \
      --cacert /etc/kubernetes/pki/etcd/ca.crt \
      --cert /etc/kubernetes/pki/etcd/ca.crt \
      --key /etc/kubernetes/pki/etcd/ca.key \
      --endpoints https://127.0.0.1:2379/ defrag --command-timeout=30s
    

  • Increase node memory. Starting from 24 GB, quota-backend-bytes will increase by 1 GB for every extra 8 GB of memory.

    Example:

    Node memory	quota-backend-bytes
    16 GB	2147483648 (2 GB)
    24 GB	3221225472 (3 GB)
    32 GB	4294967296 (4 GB)
    40 GB	5368709120 (5 GB)
    48 GB	6442450944 (6 GB)
    56 GB	7516192768 (7 GB)
    64 GB	8589934592 (8 GB)
    72 GB	8589934592 (8 GB)
    …	…

• D8KubernetesStaleTokensDetected CE S8
  Stale service account tokens detected for XXX/XXX.

  Service account XXX/XXX is using a stale token. This issue may occur if an application reads the token only at startup and does not reload it periodically. As a result, an outdated token might be used, leading to security breach and authentication failures.

  Recommended actions:

  • Ensure your application is configured to periodically reload the token from the file system.
  • Verify that you are using an up-to-date client library that supports automatic token rotation.

  Note that currently these tokens are not blocked because the --service-account-extend-token-expiration flag is enabled by default (Default: true). With this flag enabled, admission-injected tokens are extended up to 1 year during token generation to facilitate a safe transition from legacy tokens to the bound service account token feature, ignoring the value of service-account-max-token-expiration.

  For further investigation: You can find the problematic pods using the following command:

  d8 k get pods -A -o json | jq -r '.items[] | select(.spec.serviceAccountName=="XXX" and .metadata.namespace=="XXX") | .metadata.name'
  

• D8KubernetesVersionIsDeprecated CE S7
  Kubernetes version VERSION_NUMBER is deprecated.

  The current Kubernetes version VERSION_NUMBER has been deprecated, and support for it will be removed in upcoming releases.

  Please migrate to the next kubernetes version (at least 1.34)

  Read documentation about how to update the Kubernetes version in the cluster.

• D8NeedDecreaseEtcdQuotaBackendBytes CE S6
  Deckhouse suggests reducing quota-backend-bytes.

  When the control plane node memory is reduced, Deckhouse may suggest reducing quota-backend-bytes. While Deckhouse is capable of automatically increasing this value, reducing it must be done manually.

  To modify quota-backend-bytes, set the controlPlaneManager.etcd.maxDbSize parameter. Before setting a new value, check the current database usage on every control plane node by running:

  for pod in $(d8 k get pod -n kube-system -l component=etcd,tier=control-plane -o name); do
    d8 k -n kube-system exec -ti "$pod" -- /usr/bin/etcdctl \
      --cacert /etc/kubernetes/pki/etcd/ca.crt \
      --cert /etc/kubernetes/pki/etcd/ca.crt \
      --key /etc/kubernetes/pki/etcd/ca.key \
      endpoint status -w json | jq --arg a "$pod" -r \
      '.[0].Status.dbSize / 1024 / 1024 | tostring | $a + ": " + . + " MB"';
  done
  

  Things to note:

  • The maximum value for controlPlaneManager.etcd.maxDbSize is 8 GB.
  • If control plane nodes have less than 24 GB, set controlPlaneManager.etcd.maxDbSize to 2 GB.

  • Starting from 24 GB, quota-backend-bytes will increase by 1 GB for every extra 8 GB of memory.

    Example:

    Node memory	quota-backend-bytes
    16 GB	2147483648 (2 GB)
    24 GB	3221225472 (3 GB)
    32 GB	4294967296 (4 GB)
    40 GB	5368709120 (5 GB)
    48 GB	6442450944 (6 GB)
    56 GB	7516192768 (7 GB)
    64 GB	8589934592 (8 GB)
    72 GB	8589934592 (8 GB)
    …	…

• D8ObsoleteClusterDomainInClusterConfiguration CE S6
  The obsolete clusterDomain field is still present in ClusterConfiguration.

  ClusterConfiguration still carries the obsolete clusterDomain field. The place to manage the cluster domain is ModuleConfig control-plane-manager.

  This fires simply because the field is present, even if ModuleConfig control-plane-manager already overrides it - the goal is to remove the deprecated field, not just override it.

  To resolve this alert:

  1. If ModuleConfig control-plane-manager does not exist, create it:

     d8 s module enable control-plane-manager
     

  2. Migrate the value straight from the Secret into ModuleConfig:

     DOMAIN=$(d8 k -n kube-system get secret d8-cluster-configuration \
       -o jsonpath='{.data.cluster-configuration\.yaml}' | base64 -d \
       | sed -n 's/^clusterDomain:[[:space:]]*//p' | tr -d "\"'")
     
     d8 k patch moduleconfig control-plane-manager --type merge \
       -p "{\"spec\":{\"version\":3,\"settings\":{\"network\":{\"clusterDomain\":\"$DOMAIN\"}}}}"
     

     The value must match the one the cluster currently runs with; admission rejects a mismatch.

  3. Remove the field from ClusterConfiguration:

     d8 system edit cluster-configuration
     

     Delete clusterDomain, then save.

  Once the field is removed from ClusterConfiguration, this alert is resolved.

• D8ObsoleteEncryptionAlgorithmInClusterConfiguration CE S6
  Obsolete encryptionAlgorithm in ClusterConfiguration needs to be migrated to control-plane-manager ModuleConfig.

  The encryptionAlgorithm field is set to a non-default value in ClusterConfiguration but has not been migrated to ModuleConfig control-plane-manager yet.

  Until the migration is complete, the value from ClusterConfiguration is used to avoid unintended certificate rotation.

  To resolve this alert, migrate the setting:

  1. If ModuleConfig control-plane-manager does not exist, create it first:

     d8 s module enable control-plane-manager
     

  2. Migrate the value from ClusterConfiguration to ModuleConfig in one command:

     d8 k patch moduleconfig control-plane-manager --type merge -p "$(cat <<EOF
     {"spec": {"version": 3, "settings": {"encryptionAlgorithm": "$(d8 k -n kube-system get secret d8-cluster-configuration -o jsonpath='{.data.cluster-configuration\.yaml}' | base64 -d | grep encryptionAlgorithm | awk '{print $2}')"}}}
     EOF
     )"
     

  Once ModuleConfig is updated, this alert will be resolved automatically.

  1. Remove encryptionAlgorithm from ClusterConfiguration:

     d8 system edit cluster-configuration
     

     Delete the encryptionAlgorithm field and save.

• D8ObsoleteGlobalControlPlaneResourcesRequests CE S6
  Obsolete global control-plane resource requests need to be migrated.

  Control-plane resource requests are being migrated from global settings to moduleconfig control-plane-manager.

  The obsolete settings are still used as a fallback, but they should be moved to the control-plane-manager module settings. The migration does not change the meaning of the values: cpu and memory remain the total resource requests for control-plane components on each master node.

  To complete the migration, follow the steps below:

  1. Review the current obsolete settings in moduleconfig global:

     d8 k get mc global -o jsonpath='{.spec.settings.modules.resourcesRequests.controlPlane}'; echo
     

     Example of obsolete settings:

     spec:
       settings:
         modules:
           resourcesRequests:
             controlPlane:
               cpu: 1000m
               memory: 500Mi
     

  2. Make sure the moduleconfig control-plane-manager resource exists and uses settings version 3.

     If the resource does not exist, create it:

     d8 s module enable control-plane-manager
     

     Set the settings version:

     d8 k patch mc control-plane-manager --type merge \
       -p '{"spec":{"version":3}}'
     

  3. Copy the obsolete values from moduleconfig global to moduleconfig control-plane-manager.

     The following command preserves the original JSON value types, including numeric CPU values:

     d8 k patch mc control-plane-manager --type merge -p "$(
       d8 k get mc global -o json | jq -c \
         '{"spec":{"version":3,"settings":{"resourcesRequests": .spec.settings.modules.resourcesRequests.controlPlane}}}'
     )"
     

     After migration, the moduleconfig control-plane-manager resource should contain:

     spec:
       version: 3
       settings:
         resourcesRequests:
           cpu: 1000m
           memory: 500Mi
     

  4. Check that the values were applied:

     d8 k get mc control-plane-manager -o jsonpath='{.spec.settings.resourcesRequests}'; echo
     

  5. Remove the obsolete settings from moduleconfig global:

     d8 k patch mc global --type merge \
       -p '{"spec":{"settings":{"modules":{"resourcesRequests":null}}}}'
     

• D8ObsoleteKubernetesVersionFieldInClusterConfiguration CE S6
  The obsolete kubernetesVersion field is still present in ClusterConfiguration.

  ClusterConfiguration still carries the obsolete kubernetesVersion field. The place to manage the cluster Kubernetes version is settings.kubernetesVersion in ModuleConfig control-plane-manager.

  To resolve this alert:

  1. If ModuleConfig control-plane-manager does not exist, create it:

     d8 s module enable control-plane-manager
     

  2. Read what the obsolete field currently holds:

     d8 k -n kube-system get secret d8-cluster-configuration -o jsonpath='{.data.cluster-configuration\.yaml}' | base64 -d | grep '^kubernetesVersion:'
     

  3. If ModuleConfig has no kubernetesVersion yet, set that same value there, so the cluster keeps the version it runs today (replace 1.34 with what you saw; if you saw Automatic, use Default):

     d8 k patch moduleconfig control-plane-manager --type merge -p '{"spec":{"version":3,"settings":{"kubernetesVersion":"1.34"}}}'
     

     Set Default instead if you want Deckhouse to choose the version. Note that this starts an upgrade whenever the release default is higher than the pinned version:

     d8 k patch moduleconfig control-plane-manager --type merge -p '{"spec":{"version":3,"settings":{"kubernetesVersion":"Default"}}}'
     

  4. Remove kubernetesVersion from ClusterConfiguration:

     d8 system edit cluster-configuration
     

     Delete the kubernetesVersion field and save.

  Once the kubernetesVersion field is removed from ClusterConfiguration, this alert is resolved.

• D8ObsoleteNetworkFieldsInClusterConfiguration CE S6
  Obsolete network fields are still present in ClusterConfiguration.

  ClusterConfiguration still carries one or more of the obsolete podSubnetCIDR, serviceSubnetCIDR, podSubnetNodeCIDRPrefix fields. The place to manage the cluster’s network is the network section of ModuleConfig control-plane-manager.

  This fires simply because a field is present, even if ModuleConfig control-plane-manager already overrides it - the goal is to remove the deprecated field, not just override it.

  To resolve this alert:

  1. If ModuleConfig control-plane-manager does not exist, create it:

     d8 s module enable control-plane-manager
     

  2. Migrate the values straight from the Secret into ModuleConfig using the following command:

     d8 k patch moduleconfig control-plane-manager --type merge -p "$(
       d8 k -n kube-system get secret d8-cluster-configuration -o jsonpath='{.data.cluster-configuration\.yaml}' | base64 -d \
         | yq -o=json '{"podSubnetCIDR": .podSubnetCIDR, "serviceSubnetCIDR": .serviceSubnetCIDR, "podSubnetNodeCIDRPrefix": .podSubnetNodeCIDRPrefix}' \
         | jq -c '{spec: {version: 3, settings: {network: with_entries(select(.value != null))}}}'
     )"
     

     Once a field is set in ModuleConfig this way, it cannot be migrated again: the network group is immutable once set.

  3. Remove all three fields from ClusterConfiguration:

     d8 system edit cluster-configuration
     

     Delete podSubnetCIDR, serviceSubnetCIDR and podSubnetNodeCIDRPrefix, then save.

  Once all three fields are removed from ClusterConfiguration, this alert is resolved.

• D8ObsoletePublishAPIinUserAuthn CE S6
  Obsolete Publish API settings in user-authn need to be migrated.

  Publish API ingress settings are being migrated from mc user-authn to mc control-plane-manager.

  To complete the migration, follow the steps below:

  1. Review current settings for publishAPI in the moduleconfig user-authn resource.

     Note that legacy user-authn spec.version: 1 uses publishAPI.enable instead of publishAPI.enabled. No manual rewrite is required: the command in step 4 uses the normalized payload from the temporary ConfigMap.

  2. If the moduleconfig control-plane-manager resource exists, update its spec to spec.version: 3. If settings.apiserver.loadBalancer is set, migrate it to settings.apiserver.publishAPI.loadBalancer. If the resource does not exist, proceed to step 3.

  3. If moduleconfig control-plane-manager does not exist, create it via d8:

     d8 s module enable control-plane-manager
     

     And set the settings version:

     d8 k patch moduleconfig control-plane-manager --type merge \
     -p '{"spec":{"version":3}}'
     

  4. Apply the migrated settings into moduleconfig control-plane-manager from temporary configMap by executing the following script:

     d8 k patch moduleconfig control-plane-manager --type merge -p "$(cat <<EOF
     {"spec": {"settings": {"apiserver": {"publishAPI": {"ingress": \
     $(d8 k -n kube-system get cm d8-publishapi-config-migration -ojsonpath='{.data.config}')
        }}}}}
     EOF
     )"
     

  5. Remove obsolete spec.settings.publishAPI section from moduleconfig user-authn.

• D8ProblematicFeatureGateInUse CE S9
  Feature gate XXX is STATUS_REFERENCE in current Kubernetes version XXX and should be removed from the configuration. Is not using in the cluster control-plane.

  The feature gate XXX is currently enabled in the control-plane-manager ModuleConfig, but it is STATUS_REFERENCE in the current Kubernetes version XXX.

  XXX This feature gate does not exist in any component (apiserver, kube-controller-manager, kube-scheduler, kubelet) and is not being applied to the cluster. XXX

  Recommended actions:

  1. Remove XXX from the spec.settings.enabledFeatureGates list in the control-plane-manager ModuleConfig:

  d8 k edit mc control-plane-manager
  

  For more information about Kubernetes feature gates, refer to the official documentation.

• D8UsingFeatureGateWillBeDeprecated CE S9
  Feature gate XXX will be deprecated in Kubernetes version XXX.

  The feature gate XXX is currently enabled in the control-plane-manager ModuleConfig, but it will be deprecated in Kubernetes version XXX.

  Current Kubernetes version: XXX

  Recommended actions:

  1. Review the usage of this feature gate in your cluster.
  2. Plan migration away from this feature gate before upgrading to version XXX.
  3. Remove XXX from the spec.settings.enabledFeatureGates list in the control-plane-manager ModuleConfig:

  d8 k edit mc control-plane-manager
  

  For more information about Kubernetes feature gates, refer to the official documentation.

• HelmReleasesHasResourcesWithDeprecatedVersions CE S5
  At least one Helm release contains resources with deprecated apiVersion, which will be removed in Kubernetes version VERSION_NUMBER.

  To list all affected resources, run the following Prometheus query:

  max by (helm_release_namespace, helm_release_name, helm_version, resource_namespace, resource_name, api_version, kind, k8s_version) (resource_versions_compatibility) == 1
  

  For more details on upgrading deprecated resources, refer to the Kubernetes deprecation guide available at https://kubernetes.io/docs/reference/using-api/deprecation-guide/#vXXX.

  Note that the check runs once per hour, so this alert should resolve within an hour after migrating deprecated resources.

• HelmReleasesHasResourcesWithUnsupportedVersions CE S4
  At least one Helm release contains resources with unsupported apiVersion for Kubernetes version VERSION_NUMBER.

  To list all affected resources, run the following Prometheus query:

  max by (helm_release_namespace, helm_release_name, helm_version, resource_namespace, resource_name, api_version, kind, k8s_version) (resource_versions_compatibility) == 2
  

  For more details on migrating deprecated resources, refer to the Kubernetes deprecation guide available at https://kubernetes.io/docs/reference/using-api/deprecation-guide/#vXXX.

  Note that the check runs once per hour, so this alert should resolve within an hour after migrating deprecated resources.

• K8SApiserverDown CE S3
  API servers can't be reached.

  No API servers are reachable, or they have all disappeared from service discovery.

• K8sCertificateExpiration CE S5
  Kubernetes has API clients with soon-to-expire certificates.

  Some clients are connecting to COMPONENT_NAME with certificates that will expire in less than a day on node COMPONENT_NAME.

  To check control plane certificates, use kubeadm:

  1. Install kubeadm using the following command:

     apt install kubeadm=1.24.*
     

  2. Check certificates:

     kubeadm alpha certs check-expiration
     

  To check kubelet certificates, do the following on each node:

  1. Check kubelet configuration:

     ps aux \
       | grep "/usr/bin/kubelet" \
       | grep -o -e "--kubeconfig=\S*" \
       | cut -f2 -d"=" \
       | xargs cat
     

  2. Locate the client-certificate or client-certificate-data field.
  3. Check certificate expiration using OpenSSL.

  Note that there are no tools to find other stale kubeconfig files. Consider enabling the control-plane-manager module for advanced debugging.

• K8sCertificateExpiration CE S6
  Kubernetes has API clients with soon-to-expire certificates.

  Some clients are connecting to COMPONENT_NAME with certificates that will expire in less than 7 days on node NODE_NAME.

  To check control plane certificates, use kubeadm:

  1. Install kubeadm using the following command:

     apt install kubeadm=1.24.*
     

  2. Check certificates:

     kubeadm alpha certs check-expiration
     

  To check kubelet certificates, do the following on each node:

  1. Check kubelet configuration:

     ps aux \
       | grep "/usr/bin/kubelet" \
       | grep -o -e "--kubeconfig=\S*" \
       | cut -f2 -d"=" \
       | xargs cat
     

  2. Locate the client-certificate or client-certificate-data field.
  3. Check certificate expiration using OpenSSL.

  Note that there are no tools to find other stale kubeconfig files. Consider enabling the control-plane-manager module for advanced debugging.

• K8SControllerManagerTargetDown CE S3
  Controller manager is down.

  There is no running kube-controller-manager. As a result, deployments and replication controllers are not progressing.

• K8SSchedulerTargetDown CE S3
  Scheduler is down.

  The Kubernetes scheduler is not running. As a result, new pods are not being assigned to nodes.

• KubeEtcdHighFsyncDurations CE S7
  Syncing (fsync) WAL files to disk is slow.

  In the last 15 minutes, the 99th percentile of the fsync duration for WAL files exceeded 0.5 seconds: VALUE.

  Possible causes:

  • High disk latency where etcd data is stored.
  • High CPU usage on the node.
• KubeEtcdHighNumberOfLeaderChanges CE S5
  The etcd cluster is re-electing the leader too frequently.

  There have been VALUE leader re-elections for the etcd cluster member running on node NODE_NAME in the last 10 minutes.

  Possible causes:

  • High disk latency where etcd data is stored.
  • High CPU usage on the node.
  • Degradation of network connectivity between cluster members in the multi-master mode.
• KubeEtcdInsufficientMembers CE S4
  Insufficient members in the etcd cluster.

  The etcd cluster has too few members, increasing the risk of failure if another member becomes unavailable. To resolve this issue, check the status of etcd Pods: bash d8 k -n kube-system get pod -l component=etcd

• KubeEtcdNoLeader CE S4
  The etcd cluster member running on node NODE_NAME has lost the leader.

  To resolve this issue, check the status of the etcd Pods: bash d8 k -n kube-system get pod -l component=etcd | grep NODE_NAME

• KubeEtcdTargetAbsent CE S5
  There is no etcd target in Prometheus.

  Steps to troubleshoot:

  1. Check the status of the etcd Pods:

     d8 k -n kube-system get pod -l component=etcd
     

  2. Review Prometheus logs:

     d8 k -n d8-monitoring logs -l app.kubernetes.io/name=prometheus -c prometheus
     

• KubeEtcdTargetDown CE S5
  Prometheus is unable to scrape etcd metrics.

  Steps to troubleshoot:

  1. Check the status of the etcd Pods:

     d8 k -n kube-system get pod -l component=etcd
     

  2. Review Prometheus logs:

     d8 k -n d8-monitoring logs -l app.kubernetes.io/name=prometheus -c prometheus
     

• KubernetesServiceExternalIPsDeprecated CE S6
  The cluster has Services that use the deprecated spec.externalIPs field.

  There are VALUE Service(s) in user namespaces that use the spec.externalIPs field. Services in the d8-* and kube-system namespaces are not checked.

  The spec.externalIPs field is officially deprecated in Kubernetes because of the CVE-2020-8554 security vulnerability. Starting with Kubernetes 1.36, the API server returns a warning for every request that sets this field. Support for the field will be removed in future Kubernetes versions.

  Plan the migration of the affected Services to a supported way of exposing them, for example:

  • A Service of the LoadBalancer type. In bare-metal clusters, use the metallb module to assign addresses.
  • A Service of the NodePort type.
  • An Ingress or Gateway API resource for HTTP(S) traffic.

  For details, refer to the Kubernetes announcement.

  To list the affected Services with their namespaces and external IP addresses, run:

  d8 k get services -A -o json | jq -r '["NAMESPACE", "NAME", "TYPE", "EXTERNAL-IPS"], (.items[] | select(.spec.externalIPs) | select(.metadata.namespace | test("^(d8-.*|kube-system)$") | not) | [.metadata.namespace, .metadata.name, .spec.type, (.spec.externalIPs | join(","))]) | @tsv' | column -t
  

  Alternatively, run the following Prometheus query:

  max by (namespace, service, external_ip) (kube_service_spec_external_ip{namespace!~"d8-.*|kube-system"})
  

Module deckhouse

• ApplicationHorizontalPodAutoscalerMissing CE S5
  Application NAME in namespace NAMESPACE has no HorizontalPodAutoscaler for XXX XXX.

  The spec.resourceRequests parameter of Application NAME in namespace NAMESPACE sets a replicas range (minReplicas and maxReplicas) for XXX XXX, but the application deploys no HorizontalPodAutoscaler for this workload. Deckhouse Platform runs the workload with minReplicas replicas instead.

  To fix the issue, do one of the following:

  • Check the application settings: a package can deploy HorizontalPodAutoscaler only when a setting enables it.
  • Replace the range with a fixed number of replicas: set replicas instead of minReplicas and maxReplicas.

  To view the resource settings of the application, run the following command:

  d8 k get application NAME -n NAMESPACE -o jsonpath='{.spec.resourceRequests}'
  

• ApplicationHorizontalPodAutoscalerPinned CE S5
  HorizontalPodAutoscaler of XXX XXX of Application NAME in namespace NAMESPACE doesn't scale the workload.

  The spec.resourceRequests parameter of Application NAME in namespace NAMESPACE sets a fixed number of replicas (replicas) for XXX XXX, which a HorizontalPodAutoscaler of the application scales. Deckhouse Platform sets both bounds of the HorizontalPodAutoscaler to this number, so it doesn’t scale the workload.

  To let HorizontalPodAutoscaler scale the workload, replace replicas with a range: set minReplicas and maxReplicas.

  To view the resource settings of the application, run the following command:

  d8 k get application NAME -n NAMESPACE -o jsonpath='{.spec.resourceRequests}'
  

• ApplicationIsInMaintenanceMode CE S6
  Application NAME in namespace NAMESPACE is running in maintenance mode.

  Application NAME in namespace NAMESPACE is running in maintenance mode. In this mode, its state is not reconciled, which prevents configuration or hook changes from being applied automatically.

  To switch the application back to normal mode, edit the corresponding Application resource with the following command:

  d8 k patch application NAME -n NAMESPACE --type=json -p='[{"op": "remove", "path": "/spec/maintenance"}]'
  

• ApplicationVerticalPodAutoscalerMissing CE S5
  VerticalPodAutoscaler does not manage at least one resource range of container CONTAINER_NAME of XXX XXX of Application NAME in namespace NAMESPACE.

  The spec.resourceRequests parameter of Application NAME in namespace NAMESPACE sets a resource range (min and max) for container CONTAINER_NAME of XXX XXX, but at least one resource in the range is not managed by a VerticalPodAutoscaler of the application. Deckhouse Platform applies the range for each such resource as fixed values instead: the container gets min as the request and max as the limit.

  To fix the issue, do one of the following:

  • Check that the vertical-pod-autoscaler module is enabled: an application deploys VerticalPodAutoscaler only when the module is enabled.
  • Check the VerticalPodAutoscaler of the workload: it doesn’t set the resources in the Off update mode, for a container whose policy mode is Off, for a resource missing from controlledResources of the policy, and for init containers.
  • Replace the range with fixed resources: set requests and limits instead of min and max.

  To view the resource settings of the application, run the following command:

  d8 k get application NAME -n NAMESPACE -o jsonpath='{.spec.resourceRequests}'
  

• ApplicationVerticalPodAutoscalerPinned CE S5
  Fixed resource settings limit VerticalPodAutoscaler for container CONTAINER_NAME of XXX XXX of Application NAME in namespace NAMESPACE.

  The spec.resourceRequests parameter of Application NAME in namespace NAMESPACE sets fixed cpu or memory requests or limits for container CONTAINER_NAME of XXX XXX. If a request is set for a resource managed by VerticalPodAutoscaler, Deckhouse Platform sets both VPA bounds to that request. If a limit is set, Deckhouse Platform sets controlledValues: RequestsOnly: VPA can change requests that are not fixed by the Application, but does not change limits.

  To let VerticalPodAutoscaler manage the requests within a range, replace the fixed cpu and memory requests and limits with min and max.

  To view the resource settings of the application, run the following command:

  d8 k get application NAME -n NAMESPACE -o jsonpath='{.spec.resourceRequests}'
  

• D8CNIEnabledMoreThanOne CE S2
  More than one CNI is enabled in the cluster.

  Deckhouse has detected that multiple CNIs are enabled in the cluster. For the cluster to work correctly, exactly one CNI must be enabled.

  To resolve this issue, follow these steps:

  1. List the CNI modules and their enabled state:

     d8 k get moduleconfig -o custom-columns='NAME:.metadata.name,ENABLED:.spec.enabled' | grep -E 'NAME|cni-'
     

  2. Decide which CNI the cluster must keep. Check which one actually serves the nodes before disabling anything:

     d8 k -n d8-cni-cilium get pods 2>/dev/null; d8 k -n d8-cni-flannel get pods 2>/dev/null; d8 k -n d8-cni-simple-bridge get pods 2>/dev/null
     

  3. Disable every other CNI module, one at a time:

     d8 k patch moduleconfig CNI_MODULE_NAME --type=merge -p '{"spec":{"enabled":false}}'
     

  Changing the CNI of a running cluster disrupts pod networking. If you are unsure which CNI to keep, contact support before disabling anything.

• D8CNIMisconfigured CE S5
  The parameters specified in ModuleConfig of CNI_NAME do not match the ones that are actually being used in the cluster.

  This happened because there were several sources for configuring CNI parameters in the cluster, and ModuleConfig did not have the highest priority previously.

  To resolve this issue, the following steps should be taken:

  1. Find the ConfigMap d8-system/desired-cni-moduleconfig in the cluster, which contains the actual ModuleConfig settings.

     d8 k -n d8-system get configmap desired-cni-moduleconfig -o yaml
     

  2. Apply this prepared ModuleConfig to the cluster. This will not cause any actual reconfigurations in the cluster and is completely safe.

  3. Once the parameters in “ModuleConfig” match those used in the cluster:
     • this alert will be resolved,
     • and the priority of the sources for configuring CNI will change, and ModuleConfig will become the main source of truth.
  4. After that, you can add the desired parameters to the ModuleConfig CNI and they will be applied immediately.
• D8DeckhouseConfigInvalid CE S5
  Deckhouse rejected the cluster configuration: one of the ModuleConfig resources is invalid.

  Deckhouse has detected errors in the cluster configuration. The usual causes are spec.settings that do not match the module’s settings schema, and a failed conversion of spec.settings to the latest schema version.

  This alert does not name the module, because the underlying metric has no labels. Deckhouse puts the exact error into the status.message field of the ModuleConfig that caused it, prefixed with Error:.

  To resolve this issue, follow these steps:

  1. Find the affected configuration. The MESSAGE column holds the error and the VERSION column holds the settings schema version currently in use:

     d8 k get moduleconfig
     

  2. Read the full error for the ModuleConfig that reports one (global is a valid value for MODULE_NAME):

     d8 k get moduleconfig MODULE_NAME -o jsonpath='{.status.message}{"\n"}'
     

  3. Fix spec.settings, or spec.version if the error mentions a settings version or a conversion:

     d8 k edit moduleconfig MODULE_NAME
     

  4. If no ModuleConfig reports an error, check the Deckhouse logs:

     d8 k -n d8-system logs -f -l app=deckhouse
     

• D8DeckhouseCouldNotDeleteModule CE S4
  Deckhouse is unable to delete the MODULE_NAME module.

  To investigate the issue, check the logs by running the following command:

  d8 k -n d8-system logs -f -l app=deckhouse
  

• D8DeckhouseCouldNotDiscoverModules CE S4
  Deckhouse is unable to discover modules.

  To investigate the issue, check the logs by running the following command:

  d8 k -n d8-system logs -f -l app=deckhouse
  

• D8DeckhouseCouldNotRunGlobalHook CE S5
  Deckhouse is unable to run the HOOK_NAME global hook.

  To investigate the issue, check the logs by running the following command:

  d8 k -n d8-system logs -f -l app=deckhouse
  

• D8DeckhouseCouldNotRunModule CE S4
  Deckhouse is unable to start the MODULE_NAME module.

  To investigate the issue, check the logs by running the following command:

  d8 k -n d8-system logs -f -l app=deckhouse
  

• D8DeckhouseCouldNotRunModuleHook CE S7
  Deckhouse is unable to run the MODULE_NAME/HOOK_NAME module hook.

  To investigate the issue, check the logs by running the following command:

  d8 k -n d8-system logs -f -l app=deckhouse
  

• D8DeckhouseCustomTargetDown CE S4
  Prometheus is unable to scrape custom metrics generated by Deckhouse hooks.

  Prometheus cannot reach the endpoint that serves metrics exported by Deckhouse hooks. Deckhouse core metrics may still be scraped, so this usually points at the hook metrics endpoint rather than at Deckhouse being down.

  To investigate the issue, follow these steps:

  1. Check that the pod is up:

     d8 k -n d8-system get pods -l app=deckhouse -o wide
     

  2. Look for hook failures in the logs, since a hook that never completes exports no metrics:

     d8 k -n d8-system logs -l app=deckhouse --tail=100 | grep -i error
     

  3. Check that the scrape configuration for hook metrics is still present:

     d8 k -n d8-monitoring get podmonitor deckhouse -o jsonpath='{range .spec.podMetricsEndpoints[*]}{.path}{"\n"}{end}'
     

• D8DeckhouseDeprecatedConfigmapManagedByArgoCD CE S4
  Deprecated Deckhouse ConfigMap managed by Argo CD.

  The Deckhouse ConfigMap is no longer used.

  To resolve this issue, remove the d8-system/deckhouse ConfigMap from Argo CD.

• D8DeckhouseGlobalHookFailsTooOften CE S9
  The HOOK_NAME Deckhouse global hook is crashing too frequently.

  The HOOK_NAME hook has failed multiple times in the last __SCRAPE_INTERVAL_X_4__.

  To investigate the issue, check the logs by running the following command:

  d8 k -n d8-system logs -f -l app=deckhouse
  

• D8DeckhouseHasNoAccessToRegistry CE S7
  Deckhouse is unable to connect to the registry.

  Deckhouse can’t connect to the registry to check for new releases. With a release channel set, it checks the channel every 3 minutes; without one, it checks the digest of its own image every 15 seconds. Without access to the registry, automatic updates are unavailable.

  Module images come from the same registry, so pods that need a new image fail to start as well, sometimes with errors that do not mention the registry. The usual causes are expired or changed credentials, a registry certificate that has expired or was replaced with one signed by an unknown authority, and a network or proxy change.

  To investigate the issue, follow these steps:

  1. Read the registry error in the logs. UNAUTHORIZED points at the credentials, x509 at the certificate, and a timeout or a refused connection at the network:

     d8 k -n d8-system logs -l app=deckhouse --tail=500 | grep -iE 'check deckhouse release|registry'
     

  2. Check which registry address the cluster is configured to use:

     d8 k -n d8-system get secret deckhouse-registry -o jsonpath='{.data.address}' | base64 -d
     

  3. If the address is correct, the credentials are the next suspect. Check when the secret was last updated:

     d8 k -n d8-system get secret deckhouse-registry
     

  4. If the error mentions x509, check who issued the registry certificate and when it expires. Run the command on a master node and substitute the host from the registry address, and its port if it is not 443:

     openssl s_client -connect REGISTRY_HOST:443 -servername REGISTRY_HOST </dev/null 2>/dev/null | openssl x509 -noout -issuer -enddate
     

  5. Check that the node running Deckhouse can reach the registry at all, for example through a proxy or a firewall rule that changed.

• D8DeckhouseIsHung CE S4
  Deckhouse is down.

  Deckhouse is probably down, since the deckhouse_live_ticks metric in Prometheus has stopped increasing. This metric is expected to increment every 10 seconds.

  To investigate the issue, follow these steps:

  1. Check the pod state and its restart count:

     d8 k -n d8-system get pods -l app=deckhouse -o wide
     

  2. Read the logs. A deadlock or a hook that never returns stops the ticks while the pod still looks healthy:

     d8 k -n d8-system logs -l app=deckhouse --tail=200
     

  3. Check whether a queue stopped moving at the same time:

     d8 system queue list
     

  4. If the pod is alive but stuck, restart it:

     d8 k -n d8-system rollout restart deploy/deckhouse
     

• D8DeckhouseIsNotOnReleaseChannel CE S9
  Deckhouse isn't subscribed to any regular release channels.

  Deckhouse in this cluster isn’t subscribed to any of the regular release channels: Alpha, Beta, EarlyAccess, Stable, or RockSolid.

  To resolve this issue, follow these steps:

  1. Check the current release channel used in the cluster:

     d8 k -n d8-system  get deploy deckhouse -o json | jq '.spec.template.spec.containers[0].image' -r
     

  2. Subscribe to one of the regular release channels by adjusting the deckhouse module configuration.

• D8DeckhouseModuleHookFailsTooOften CE S9
  The MODULE_NAME/HOOK_NAME Deckhouse hook is crashing too frequently.

  The HOOK_NAME hook of the MODULE_NAME module has failed multiple times in the last __SCRAPE_INTERVAL_X_4__.

  To investigate the issue, check the logs by running the following command:

  d8 k -n d8-system logs -f -l app=deckhouse
  

• D8DeckhouseModuleImageDownloadInterrupted CE S4
  Download of the MODULE_NAME module image was interrupted.

  Deckhouse Platform was downloading the image of the MODULE_NAME module, version VERSION_NUMBER, from the container registry to build its erofs image, and the download was interrupted. The module version isn’t put on the node running Deckhouse Platform until a download completes.

  The download is interrupted when it doesn’t finish within the registry request timeout, 2 minutes by default, or when the connection to the container registry drops. A slow connection to the container registry is the usual cause.

  To investigate the issue, follow these steps:

  1. Find the reason in the logs. A timeout reads as context deadline exceeded, a dropped connection as unexpected EOF or connection reset by peer:

     d8 k -n d8-system logs -l app=deckhouse --tail=1000 | grep 'download image layers'
     

  2. Find the ModuleSource the module is downloaded from and the address of its container registry:

     d8 k get module MODULE_NAME -o jsonpath='{.properties.source}'
     d8 k get modulesource SOURCE_NAME -o jsonpath='{.spec.registry.repo}'
     

  3. Check that the image of version VERSION_NUMBER is in the container registry:

     d8 cr ls REGISTRY_ADDRESS/MODULE_NAME
     

     For a private registry, pass the credentials from the ModuleSource:

     d8 cr ls -u USERNAME -p PASSWORD REGISTRY_ADDRESS/MODULE_NAME
     

  4. Check that the node running Deckhouse Platform reaches the container registry and downloads from it fast enough, for example, through a proxy or a firewall rule that changed.

• D8DeckhouseModuleUpdatePolicyNotFound CE S5
  Module update policy not found for MODULE_RELEASE.

  The module update policy for MODULE_RELEASE is missing.

  To resolve this issue, remove the label from the module release using the following command:

  d8 k label mr MODULE_RELEASE modules.deckhouse.io/update-policy-
  

  A new suitable policy will be detected automatically.

• D8DeckhouseModuleValidationError CE S5
  Module configuration failed for module MODULE_NAME.

  Initial config for module MODULE_NAME is not valid.

  You can get more details via

  d8 k get mr -l module=MODULE_NAME
  

  Provided error: XXX

• D8DeckhouseObsoleteMonitoringDeckhouseModuleConfig CE S9
  Obsolete monitoring-deckhouse ModuleConfig should be removed.

  The monitoring-deckhouse module was merged into the deckhouse module, but its ModuleConfig still exists in the cluster and is no longer used.

  Remove it manually:

  d8 k delete moduleconfig monitoring-deckhouse
  

  If the ModuleConfig is deployed from an external source (e.g. a Git repository), make sure it is removed from there so it is not created again.

• D8DeckhousePodIsNotReady CE S4
  The Deckhouse Pod is NOT Ready.

  The Deckhouse pod is running but has not become Ready. While this lasts, Deckhouse is not converging the cluster: module updates and configuration changes are not applied.

  To investigate the issue, follow these steps:

  1. Check the pod state and its restart count:

     d8 k -n d8-system get pods -l app=deckhouse -o wide
     

  2. Read the events and probe failures at the bottom of the pod description:

     d8 k -n d8-system describe pod -l app=deckhouse
     

  3. Check the logs. A readiness probe that never succeeds is most often a startup hook that fails or hangs:

     d8 k -n d8-system logs -l app=deckhouse --tail=200
     

  4. Check whether a queue is stuck, which keeps Deckhouse from reporting Ready:

     d8 system queue list
     

• D8DeckhousePodIsNotRunning CE S4
  The Deckhouse Pod is NOT Running.

  There is no running Deckhouse pod in the d8-system namespace. Nothing is reconciling the cluster: no module updates, no configuration changes, and no Deckhouse releases are applied.

  To investigate the issue, follow these steps:

  1. Check the pod state. Pending points at scheduling, ImagePullBackOff at the registry, and CrashLoopBackOff at Deckhouse failing to start:

     d8 k -n d8-system get pods -l app=deckhouse -o wide
     

  2. Read the events to find out why the pod is not running:

     d8 k -n d8-system describe pod -l app=deckhouse
     

  3. If the pod is restarting, read the logs of the previous run:

     d8 k -n d8-system logs -l app=deckhouse --previous --tail=200
     

  4. Check that the Deployment itself is present and has the replicas it expects:

     d8 k -n d8-system get deploy deckhouse
     

• D8DeckhousePodIsRestartingTooOften CE S9
  Excessive Deckhouse restarts detected.

  Number of restarts in the last hour: VALUE.

  Excessive Deckhouse restarts indicate a potential issue. Normally, Deckhouse should be up and running continuously.

  To investigate the issue, check the logs by running the following command:

  d8 k -n d8-system logs -f -l app=deckhouse
  

• D8DeckhouseQueueIsHung CE S6
  Module queue MODULE_NAME (type XXX, hook HOOK_NAME) is hung.

  The module queue head has remained unchanged for more than 10 minutes.

  Affected queue:

  • Module: MODULE_NAME
  • Task type: XXX
  • Hook: HOOK_NAME

  To investigate the issue, check the logs of the deckhouse controller pods:

  d8 k -n d8-system logs -f -l app=deckhouse
  

• D8DeckhouseQueueIsHungCritical CE S4
  Critical module queue MODULE_NAME (type XXX, hook HOOK_NAME) is hung.

  The critical module queue head has remained unchanged for more than 10 minutes.

  Affected queue:

  • Module: MODULE_NAME
  • Task type: XXX
  • Hook: HOOK_NAME

  To investigate the issue, check the logs of the deckhouse controller pods:

  d8 k -n d8-system logs -f -l app=deckhouse
  

• D8DeckhouseQueueIsHungGlobal CE S4
  Global/Parallel task XXX (hook HOOK_NAME) is hung.

  The global/parallel task head has remained unchanged for more than 10 minutes.

  Affected task:

  • Task type: XXX
  • Hook: HOOK_NAME

  To investigate the issue, check the logs of the deckhouse controller pods:

  d8 k -n d8-system logs -f -l app=deckhouse
  

• D8DeckhouseRegistryModeUnsupported CE S9
  The deckhouse ModuleConfig stores the unsupported registry mode XXX.

  The deckhouse ModuleConfig has spec.settings.registry.mode: XXX. Proxy and Local are modes of the v1 implementation of the registry module, and they have no effect in Deckhouse Platform 1.78: the registry is managed by the registry module and configured in the registry ModuleConfig. These values are removed in Deckhouse Platform 1.79, and a deckhouse ModuleConfig that still stores one will not pass validation there.

  To resolve this issue, do the following before updating to Deckhouse Platform 1.79:

  1. Check the stored value:

     d8 k get moduleconfig deckhouse -o jsonpath='{.spec.settings.registry}'
     

  2. Remove the spec.settings.registry section:

     d8 k patch moduleconfig deckhouse --type json -p '[{"op": "remove", "path": "/spec/settings/registry"}]'
     

     Alternatively, set the mode to Unmanaged and drop the proxy section, which is allowed only in Proxy mode:

     d8 k patch moduleconfig deckhouse --type merge -p '{"spec": {"settings": {"registry": {"mode": "Unmanaged", "proxy": null}}}}'
     

     Both edits are accepted. Switching the mode to Proxy or Local is refused.

  3. If the cluster needs what the stored mode was set for, an in-cluster registry or cache, configure it in the registry ModuleConfig: mode: Managed with primary.upstream, and storage.cache for an in-cluster cache. Removing the stored mode does not change where the cluster pulls images from, so this step is not required to resolve the alert.

  If the deckhouse ModuleConfig is deployed from an external source (e.g. a Git repository), make the same change there so the value is not applied again.

• D8DeckhouseSelfTargetAbsent CE S4
  There is no Deckhouse target in Prometheus.

  Prometheus has no deckhouse target at all. Unlike a target that is down, this means the target was never discovered, so the problem is usually with the Service, its endpoints, or the monitoring configuration rather than with Deckhouse itself.

  To investigate the issue, follow these steps:

  1. Check that the Deckhouse pod exists and is running:

     d8 k -n d8-system get pods -l app=deckhouse -o wide
     

  2. Check that the Service has endpoints. An empty ENDPOINTS column means nothing matches the selector:

     d8 k -n d8-system get endpoints deckhouse
     

  3. Check that the scrape configuration for Deckhouse exists:

     d8 k -n d8-system get servicemonitor,podmonitor -l app=deckhouse
     

  4. Check that the monitoring stack itself is healthy:

     d8 k -n d8-monitoring get pods
     

• D8DeckhouseSelfTargetDown CE S4
  Prometheus is unable to scrape Deckhouse metrics.

  Prometheus cannot reach the deckhouse metrics endpoint. Deckhouse itself may still be working, but its own metrics are missing, so most other Deckhouse alerts are blind while this lasts.

  To investigate the issue, follow these steps:

  1. Check that the pod is up and how long it has been running:

     d8 k -n d8-system get pods -l app=deckhouse -o wide
     

  2. Look for scrape errors, such as a connection refused or a TLS failure:

     d8 k -n d8-system logs -l app=deckhouse --tail=100
     

  3. Check that the pod still exposes the self metrics port:

     d8 k -n d8-system get pod -l app=deckhouse -o jsonpath='{range .items[*].spec.containers[*].ports[*]}{.name}{"="}{.containerPort}{"\n"}{end}'
     

• D8DeckhouseWatchErrorOccurred CE S5
  Possible API server connection error in the client-go informer.

  Deckhouse has detected an error in the client-go informer, possibly due to connection issues with the API server.

  Steps to investigate:

  1. Check Deckhouse logs for more information by running:

     d8 k -n d8-system logs deploy/deckhouse | grep error | grep -i watch
     

  2. This alert attempts to detect a correlation between the faulty snapshot invalidation and API server connection errors, specifically for the handle-node-template hook in the node-manager module.

     To compare the snapshot with the actual node objects for this hook, run the following command:

     diff -u <(d8 k get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'|sort) <(d8 k -n d8-system exec svc/deckhouse-leader -c deckhouse -- deckhouse-controller module snapshots node-manager -o json | jq '."040-node-manager/hooks/handle_node_templates.go"' | jq '.nodes.snapshot[] | .filterResult.Name' -r | sort)
     

• D8HasModuleConfigAllowedToDisable CE S4
  The ModuleConfig annotation for allowing module disabling is set.

  The ModuleConfig is pending module disabling.

  It is recommended that you keep your module configurations clean by removing unnecessary approval annotations.

  If you ignore this alert and do not clear the annotation, the module may be accidentally removed from the cluster, potentially leading to irreversible consequences.

  To resolve this issue and stop the alert, run the following command:

  d8 k annotate moduleconfig MODULE_NAME modules.deckhouse.io/allow-disabling-
  

  With Module v2 enabled, remove the packages.deckhouse.io/allow-disabling annotation instead:

  d8 k annotate moduleconfig MODULE_NAME packages.deckhouse.io/allow-disabling-
  

• D8LicenseClusterKeyInvalid BE S4
  The cluster licensing key is damaged.

  The Secret d8-system/cluster-key does not hold a valid cluster key. Deckhouse Platform keeps computing the license state, but it cannot sign the cluster data file for the customer portal, and license records bound to the cluster key are not accepted.

  Deckhouse Platform does not replace a damaged key on its own: a new key is a new cluster identity, and the license keys issued for the old one may stop applying. Apart from Deckhouse Platform, only system:sudouser may change the Secret.

  If a backup of the Secret is available, restore it as described in the documentation. Otherwise delete the Secret, and Deckhouse Platform creates a new cluster key:

  d8 k --as=system:sudouser -n d8-system delete secret cluster-key
  

  After a new key is created:

  1. Download the cluster data in the Deckhouse web UI section “Licensing”.
  2. Upload the file to the customer portal (https://lk.deckhouse.ru) and issue a new key there.
  3. Upload the key in the same Deckhouse web UI section.
• D8LicenseComplianceGrace BE S5
  The cluster license is in the grace period.

  The license keys of the cluster have expired, and the grace period is running. When it ends, the license state becomes Violation.

  To check the number of remaining grace days in the period, run the following command:

  d8 k get effectivelicense deckhouse-platform -o jsonpath='{.status.key.graceDays}{"\n"}'
  

  To renew the license:

  1. Download the cluster data in the Deckhouse web UI section “Licensing”.
  2. Upload the file to the customer portal (https://lk.deckhouse.ru) and issue a new key there.
  3. Upload the key in the same Deckhouse web UI section.
• D8LicenseComplianceViolation BE S4
  The cluster license is violated.

  The cluster license has been violated: the license keys expired, and the grace period ended (if the key had one).

  Uncovered nodes are reported by the D8LicenseUnlicensedNodes alert.

  Deckhouse Platform and the workloads keep running.

  To restore the license, get a new license key:

  1. Download the cluster data in the Deckhouse web UI section “Licensing”.
  2. Upload the file to the customer portal (https://lk.deckhouse.ru) and issue a new key there.
  3. Upload the key in the same Deckhouse web UI section.
• D8LicenseKeyExpiresSoon BE S7
  The cluster license expires soon.

  The license key of the cluster expires within 7 days.

  To renew the license:

  1. Download the cluster data in the Deckhouse web UI section “Licensing”.
  2. Upload the file to the customer portal (https://lk.deckhouse.ru) and issue a new key there.
  3. Upload the key in the same Deckhouse web UI section.

  To check the expiry date of a key, run the following command:

  d8 k get effectivelicense deckhouse-platform -o jsonpath='{.status.key.validUntil}{"\n"}'
  

• D8LicenseRejectedRecords BE S6
  There are rejected license records or keys.

  A record of an installed license key, or a whole key, was rejected. It grants nothing until it is fixed. Possible causes:

  • The key has been damaged, it is not a Deckhouse Platform license key or was not signed by Deckhouse Platform.
  • The key requires a newer Deckhouse Platform version.
  • The key was issued for another cluster.
  • The same record has already been installed with another key.
  • The record does not match the key format.

  Replaced, expired and future records, and records of a type this Deckhouse Platform version does not know, are counted separately and do not fire the alert.

  The Deckhouse web UI section “Licensing” shows the cause next to each key in the “License keys” table. To obtain the cause via the command line, run the following command:

  d8 k get clusterlicenses -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{range .status.records[?(@.reason)]}  {.id}: {.reason} {.message}{"\n"}{end}{end}'
  

  To replace the key, issue a new one in the customer portal (https://lk.deckhouse.ru).

• D8LicenseUnlicensedNodes BE S4
  The cluster license is violated because of unlicensed nodes.

  The cluster has nodes that the installed license keys do not cover, and the license state is Violation.

  The state changes only after nodes have stayed uncovered for an hour without a break. During a node replacement the new node joins before the old one leaves, and for that short time one node is uncovered. The delay keeps such a replacement silent.

  Removing the extra nodes ends the violation at once. If nodes go over the limit again within 24 hours after that, the violation ends only after every node has stayed covered for 8 hours, and after 24 hours for every further repeat. While every node is covered and a repeat still goes on, status.overLimitEndsAt names the moment it ends.

  To identify the moment the over-limit episode began:

  d8 k get effectivelicense deckhouse-platform -o jsonpath='{.status.overLimitSince}{"\n"}'
  

  The Deckhouse web UI section “Licensing” lists the uncovered nodes in the “License required” group and shows what to add to the license.

  To cover the nodes, get a license key for the current cluster size:

  1. Download the cluster data in the Deckhouse web UI section “Licensing”.
  2. Upload the file to the customer portal (https://lk.deckhouse.ru) and issue a new key there.
  3. Upload the key in the same Deckhouse web UI section.
• D8ModuleOutdatedByMajorVersion CE S9
  Module MODULE_NAME is running XXX major versions behind.

  Module MODULE_NAME is running XXX major versions behind on major version releases. Major version updates may contain breaking changes and require immediate attention and careful planning.

  To check the module releases status, run the following command:

  d8 k get mr -l module=MODULE_NAME
  ```XXXXXX
  
  This major release has from-to version constraints that must be met before applying the update.
  To check the update instructions (from-to versions), run the following command:
  
  ```bash
  d8 k get mr NAME -o jsonpath='{.spec.update.versions[*]}'
  

  This will show you the update paths including the from-to versions defined in the release.XXX

  To approve the module major release update, run the following command:

  bash d8 k annotate mr NAME modules.deckhouse.io/approved="true" XXX

• D8NodeHasDeprecatedOSVersion CE S4
  Nodes with deprecated OS versions detected.

  Deckhouse has detected nodes running deprecated OS versions.

  Steps to troubleshoot:

  1. Get a list of affected nodes by running the following Prometheus query:

     kube_node_info{os_image=~"Ubuntu 18.04.*|Debian GNU/Linux 10.*|CentOS Linux 7.*"}
     

  2. Update the affected nodes to a supported OS version.

• DeckhouseEditionNotFound CE S6
  Incorrect Deckhouse Platform edition.

  Specify a Deckhouse Platform edition in MODULE_NAME ModuleConfig.

  d8 k patch moduleconfig MODULE_NAME --type merge --patch '{"spec": {"settings": {"license": {"edition": "DKP_EDITION"}}}}'
  

  Where DKP_EDITION is the edition you want to use: CE, BE, EE, SE, or SE-plus.

• DeckhouseHighMemoryUsage CE S3
  Deckhouse memory usage is too high.

  Deckhouse pod POD_NAME on node NODE_NAME has high memory usage.

  Please, if possible, get the dumps for debugging purposes (it will take around 30 seconds):

  curl -s "http://$(d8 k -n d8-system get pod POD_NAME -o jsonpath='{.status.hostIP}'):4222/debug/pprof/heap" -o /tmp/heap.pprof
  curl -s "http://$(d8 k -n d8-system get pod POD_NAME -o jsonpath='{.status.hostIP}'):4222/debug/pprof/goroutine" -o /tmp/goroutine.pprof
  curl -s "http://$(d8 k -n d8-system get pod POD_NAME -o jsonpath='{.status.hostIP}'):4222/debug/pprof/profile?seconds=30" -o /tmp/cpu.pprof
  

  and send these files (/tmp/heap.pprof, /tmp/cpu.pprof and /tmp/goroutine.pprof) to the support team.

• DeckhouseMigratedModuleNotDownloaded CE S6
  Migrated module MODULE_NAME has not been downloaded yet.

  Module MODULE_NAME specified in DeckhouseRelease was moved to an external source and is available in a ModuleSource registry, but it has not been downloaded yet (no deployed ModuleRelease exists). DeckhouseRelease will be locked until the module is downloaded.

  This alert is usually temporary while the module source controller downloads the module from the registry. If the alert persists, this may indicate:

  • The module’s update policy or ModuleSource is misconfigured, preventing the module from being downloaded.
  • A network issue is preventing the module from being downloaded.

  To investigate the issue, do the following:

  • Check the ModuleRelease objects for the module:

    d8 k get mr -l module=MODULE_NAME
    

  • Check the configured ModuleSources:

    d8 k get modulesources
    

  • Check the deckhouse logs for more details:

    d8 k -n d8-system logs -f -l app=deckhouse
    

• DeckhouseMigratedModuleNotFound CE S6
  Migrated module MODULE_NAME was not found in any ModuleSource registry.

  Module MODULE_NAME specified in DeckhouseRelease was moved to an external source and was not found in any available ModuleSource registry. DeckhouseRelease will be locked until the module becomes available in one of the configured ModuleSource registries.

  This may indicate:

  • A ModuleSource registry is misconfigured.
  • The module has not been published to the registry yet.
  • A network issue is preventing access to the registry.

  To investigate the issue, do the following:

  • Check the configured ModuleSources:

    d8 k get modulesources
    

  • List the available module sources:

    d8 k -n d8-system exec -it deployment/deckhouse -c deckhouse -- deckhouse-controller registry get sources
    

  • List the modules available in a specific source:

    d8 k -n d8-system exec -it deployment/deckhouse -c deckhouse -- deckhouse-controller registry get modules source_name
    

  • Check the deckhouse logs for more details:

    d8 k -n d8-system logs -f -l app=deckhouse
    

• DeckhouseModuleUpdatingFailedBrokenSequence CE S4
  Module update has failed. Updating sequence is broken.

  The ‘MODULE_NAME’ module update has failed.

  Current version: XXX. Desired version: VERSION_NUMBER.

  Attempt to download interim releases failed. Deckhouse updates modules one minor version at a time, so every intermediate version must exist in the registry.

  Possible reasons:

  • Some of the necessary versions of the module image are not available in the registry.
  • The registry is unreachable.

  To resolve this issue, follow these steps:

  1. Check which module releases exist and where the sequence stopped:

     d8 k get modulereleases -l module=MODULE_NAME
     

  2. Read the reason recorded on the release that is not deployed:

     d8 k get modulereleases -l module=MODULE_NAME -o custom-columns='NAME:.metadata.name,VERSION:.spec.version,PHASE:.status.phase,MESSAGE:.status.message'
     

  3. Find the module source that serves this repo. The registry label above is a repo URL, not a source name:

     d8 k get modulesource -o custom-columns='NAME:.metadata.name,REPO:.spec.registry.repo'
     

  4. Ensure that every minor version of the ‘MODULE_NAME’ module image between XXX and VERSION_NUMBER is available in the registry ‘XXX’. If the cluster uses a mirror, the missing versions have to be pulled into it first.

• DeckhouseModuleUpdatingFailedModuleIsNotValid CE S4
  Module update has failed. Module is not valid.

  The ‘MODULE_NAME’ module update has failed.

  Desired version: VERSION_NUMBER.

  Possible reasons:

  • The module image is corrupted.

  To resolve this issue, ensure that the VERSION_NUMBER image in the registry ‘XXX’ is valid.

  Check which versions of the ‘MODULE_NAME’ module are available in the registry:

  d8 cr ls XXX/MODULE_NAME/release
  

  For a private registry, pass the credentials from the corresponding ModuleSource:

  d8 cr ls -u USERNAME -p PASSWORD XXX/MODULE_NAME/release
  

  To get the exact error, check Deckhouse logs: d8 k -n d8-system logs deploy/deckhouse | grep "failed to download module"

• DeckhouseReleaseDisruptionApprovalRequired CE S4
  Deckhouse release disruption approval required.

  The new Deckhouse release includes a disruptive update that requires manual approval.

  To check the details, run the following command:

  d8 k describe DeckhouseRelease NAME
  

  To approve the disruptive update, run the following command:

  d8 k annotate DeckhouseRelease NAME release.deckhouse.io/disruption-approved=true
  

• DeckhouseReleaseIsBlocked CE S5
  Deckhouse release requirements haven't been met.

  The requirements for the Deckhouse release haven’t been met.

  To check the details, run the following command:

  d8 k describe DeckhouseRelease NAME
  

• DeckhouseReleaseIsWaitingManualApproval CE S3
  A new Deckhouse release is awaiting manual approval.

  A new Deckhouse release is available but requires manual approval before it can be applied.

  To approve the release, run the following command:

  d8 k patch DeckhouseRelease NAME --type=merge -p='{"approved": true}'
  

• DeckhouseReleaseIsWaitingManualApproval CE S6
  A new Deckhouse release is awaiting manual approval.

  A new Deckhouse release is available but requires manual approval before it can be applied.

  To approve the release, run the following command:

  d8 k patch DeckhouseRelease NAME --type=merge -p='{"approved": true}'
  

• DeckhouseReleaseIsWaitingManualApproval CE S9
  A new Deckhouse release is awaiting manual approval.

  A new Deckhouse release is available but requires manual approval before it can be applied.

  To approve the release, run the following command:

  d8 k patch DeckhouseRelease NAME --type=merge -p='{"approved": true}'
  

• DeckhouseReleaseNotificationNotSent CE S4
  Deckhouse release notification webhook hasn't been sent.

  The Deckhouse release notification webhook failed to send.

  To check the notification webhook address, run the following command:

  d8 k get mc deckhouse -o yaml
  

• DeckhouseUpdating CE S4

  Deckhouse is being updated to XXX.

• DeckhouseUpdatingFailed CE S4
  Deckhouse update has failed.

  The Deckhouse update to version VERSION_NUMBER has failed.

  Possible reasons:

  • The next minor or patch version of the Deckhouse image is not available in the registry.
  • The Deckhouse image is corrupted.
  • Deckhouse cannot reach the registry.

  To resolve this issue, follow these steps:

  1. Check the state of the releases. The one stuck in Pending is the one that failed:

     d8 k get deckhousereleases
     

  2. Read the reason recorded on that release:

     d8 k get deckhousereleases -o custom-columns='NAME:.metadata.name,PHASE:.status.phase,MESSAGE:.status.message'
     

  3. Read the corresponding error in the logs:

     d8 k -n d8-system logs -l app=deckhouse --tail=200 | grep -i release
     

  4. Ensure that version VERSION_NUMBER of the Deckhouse image is present in the registry. If the cluster uses a mirror, the missing version has to be pulled into it first.

• ModuleAtConflict CE S4
  Conflict detected for module MODULE_NAME.

  Deckhouse has detected conflicting sources for the MODULE_NAME module: it is available from more than one ModuleSource and no source is selected, so the module is not deployed.

  To resolve this issue, follow these steps:

  1. List the sources that provide the module:

     d8 k get module MODULE_NAME -o jsonpath='{.properties.availableSources}{"\n"}'
     

  2. Compare what each source offers, so you pick the one you actually want:

     d8 k get modulerelease -l module=MODULE_NAME
     

  3. Select the source in the module configuration:

     d8 k patch moduleconfig MODULE_NAME --type=merge -p '{"spec":{"source":"SOURCE_NAME"}}'
     

  4. Verify that the module has left the Conflict phase. The SOURCE and PHASE columns show the result:

     d8 k get module MODULE_NAME
     

• ModuleConfigObsoleteVersion CE S4
  ModuleConfig NAME uses obsolete settings version VERSION_NUMBER, the latest one is XXX.

  Deckhouse has detected that ModuleConfig NAME has spec.version: VERSION_NUMBER, while the latest settings schema version of the NAME module is XXX.

  Deckhouse converts spec.settings from version VERSION_NUMBER to version XXX on the fly, so the module keeps working, but the stored configuration remains outdated. Support for version VERSION_NUMBER may be dropped in a future release.

  To resolve this issue, follow these steps:

  1. Review the migration notes for the NAME module settings, one line per conversion step:

     d8 k get modulesettingsdefinition NAME -o jsonpath='{range .spec.versions[*].conversions[*]}{.descriptions.en}{"\n"}{end}'
     

  2. Migrate spec.settings accordingly and set spec.version to XXX:

     d8 k edit moduleconfig NAME
     

  3. Verify that the version has been updated and no warnings are left:

     d8 k get moduleconfig NAME -o jsonpath='{.spec.version}{"\t"}{.status.message}{"\n"}'
     

• ModuleHasDeprecatedUpdatePolicy CE S4
  Module MODULE_NAME is matched by the deprecated update policy UPDATE_POLICY_NAME.

  Deckhouse has detected that the module MODULE_NAME is using the deprecated update policy UPDATE_POLICY_NAME. The v1alpha1 policy has a selector that no longer functions.

  To specify the update policy in the module configuration, run the following command:

  d8 k patch moduleconfig MODULE_NAME --type='json' -p='[{"op": "add", "path": "/spec/updatePolicy", "value": "UPDATE_POLICY_NAME"}]'
  

  After resolving all alerts related to the update policy UPDATE_POLICY_NAME, clear the selector by running the following command:

  d8 k patch moduleupdatepolicies.v1alpha1.deckhouse.io UPDATE_POLICY_NAME --type='json' -p='[{"op": "replace", "path": "/spec/moduleReleaseSelector/labelSelector/matchLabels", "value": {"": ""}}]'
  

• ModuleIsDeprecated CE S9
  Module MODULE_NAME is deprecated.

  The module MODULE_NAME is deprecated and will not receive new updates. The module may be removed in a future release, so disable it in a controlled manner rather than waiting for it to disappear.

  To resolve this issue, follow these steps:

  1. Check the module state before changing anything. The STAGE column reads Deprecated, and the conditions show whether anything still runs:

     d8 k get module MODULE_NAME
     

  2. Find out whether other modules declare a dependency on it:

     d8 k get modules -o json | jq -r '.items[] | select(.properties.requirements.modules | has("MODULE_NAME")) | .metadata.name'
     

  3. Once nothing depends on it and its workloads are no longer needed, disable the module:

     d8 k patch moduleconfig MODULE_NAME --type=merge -p '{"spec":{"enabled":false}}'
     

• ModuleIsInMaintenanceMode CE S6
  Module MODULE_NAME is running in maintenance mode.

  Module MODULE_NAME is running in maintenance mode. In this mode, its state is not reconciled, which prevents configuration or hook changes from being applied automatically.

  To switch the module back to normal mode, edit the corresponding ModuleConfig resource with the following command:

  d8 k patch moduleconfig MODULE_NAME --type=json -p='[{"op": "remove", "path": "/spec/maintenance"}]'
  

• ModuleReleaseIsBlockedByRequirements CE S6
  A new release for module MODULE_NAME has been blocked due to unmet requirements.

  A new release for module MODULE_NAME has been blocked because it doesn’t meet the required conditions.

  To check the requirements, run the following command:

  d8 k  get mr NAME -o json | jq .spec.requirements
  

• ModuleReleaseIsOutdated CE S5
  Module MODULE_NAME is XXX or more minor versions behind the latest available release.

  Module MODULE_NAME is running XXX or more minor versions behind the latest available release. This module is significantly outdated and may have compatibility issues.

  To approve the module release update, run the following command:

  d8 k annotate mr NAME modules.deckhouse.io/approved="true"
  

• ModuleReleaseIsOutdated CE S7
  Module MODULE_NAME is 2 minor versions behind the latest available release.

  Module MODULE_NAME is running 2 minor versions behind the latest available release. Updating is recommended to maintain compatibility and access to latest features.

  To approve the module release update, run the following command:

  d8 k annotate mr NAME modules.deckhouse.io/approved="true"
  

• ModuleReleaseIsOutdated CE S9
  Module MODULE_NAME is 1 minor version behind the latest available release.

  Module MODULE_NAME is running 1 minor version behind the latest available release. Consider updating to maintain compatibility and security.

  To approve the module release update, run the following command:

  d8 k annotate mr NAME modules.deckhouse.io/approved="true"
  

• ModuleReleaseIsWaitingManualApproval CE S6
  A new release for module MODULE_NAME is awaiting manual approval.

  A new release for module MODULE_NAME is available but requires manual approval before it can be applied.

  To approve the module release, run the following command:

  d8 k annotate mr NAME modules.deckhouse.io/approved="true"
  

Module descheduler

Module documentation

• D8DocumentationModulesListNotBuilt CE S8
  The page of the list of modules in the documentation is not updated.

  docs-builder could not build the page of the list of modules for 30 minutes. The page shows the list as it was built before, or says that the list is being prepared.

  Check the logs of docs-builder:

  d8 k -n d8-system logs -l app=documentation -c builder --tail=-1 | grep "write the list of modules"
  

• D8DocumentationModuleVersionsNotUpdated CE S8
  The versions of modules in the documentation search were not updated for an hour.

  docs-builder updates the versions of modules the search and the AI export use every 15 minutes; the last update finished more than an hour ago, or none has finished in the hour since docs-builder started reading the registry.

  Check the logs of docs-builder:

  d8 k -n d8-system logs -l app=documentation -c builder --tail=-1 | grep -i -E "resolve|render pinned"
  

• D8DocumentationNoLeader CE S7
  No documentation replica reads the registry.

  No replica of the documentation holds the docs-builder-ondemand-leader Lease in d8-system. The versions of modules the search uses are not updated; versions users open are still prepared.

  Check the Lease and the logs of docs-builder:

  d8 k -n d8-system get lease docs-builder-ondemand-leader -o yaml
  d8 k -n d8-system logs -l app=documentation -c builder --tail=-1 | grep -i lead
  

• D8DocumentationPinnedVersionsTooLarge CE S8
  The module versions the documentation keeps at all times take more space than planned.

  docs-builder keeps the versions of modules the search uses on the cache volume of the pod and never evicts them. They take XXXB, more than the space set aside for them.

  Versions users open are evicted sooner and prepared again more often. If the volume fills up, Kubernetes evicts the documentation pod, and the versions are prepared again unless they are kept in S3.

  The size is in the docs_builder_ondemand_cache_pinned_bytes metric, the space set aside in docs_builder_ondemand_cache_pinned_budget_bytes: the cache volume of the pod less 2 GiB of versions users open. The space is not configurable: report the alert to the Deckhouse technical support with the number of modules enabled in the cluster.

• D8DocumentationReplicaReadsRegistry CE S8
  A documentation replica can not read S3 and reads the registry itself.

  A replica of the documentation could not read the versions of the leader from S3 for 10 minutes and prepares them from the registry itself until S3 answers again.

  Check the S3 storage of the documentation (moduleStorage.s3 of the documentation ModuleConfig) and the logs of docs-builder:

  d8 k -n d8-system logs -l app=documentation -c builder --tail=-1 | grep -i s3
  

• D8DocumentationS3NotReady CE S9
  Module versions documentation is not kept in S3, the S3 storage is not ready.

  The documentation ModuleConfig keeps the prepared documentation of module versions in S3 (moduleStorage.s3.type), but the S3 storage has not been usable for 15 minutes. Until it is, the documentation is kept in the documentation pod only and is prepared again after a restart of the pod.

  With type: External, check that the Secret from moduleStorage.s3.external.secretRef exists and holds the keys listed on the usage page of the module. The reason is in the log of the builder container:

  d8 k -n d8-system logs deploy/documentation -c builder | grep -i s3
  

• D8DocumentationSystemS3Disabled CE S9
  Module versions documentation is not kept in S3, the system-s3 module is disabled.

  moduleStorage.s3.type: System is set in the documentation ModuleConfig, but the system-s3 module is disabled. Until it is enabled, the prepared documentation of module versions is kept in the documentation pod only and is prepared again after a restart of the pod.

  Enable the system-s3 module, or set moduleStorage.s3.type to Disabled in the documentation ModuleConfig.

• D8DocumentationVersionNotRenderedAnew CE S9
  Some module versions in S3 can not be prepared with the current look of the documentation.

  After an update of the documentation look, VALUE module versions kept in S3 could not be prepared again. They open as they were prepared before the update.

  Find them in the logs of docs-builder:

  d8 k -n d8-system logs -l app=documentation -c builder --tail=-1 | grep "the template can not render the version"
  

• ModuleConfigDeprecated CE S9
  Deprecated deckhouse-web ModuleConfig detected.

  The deckhouse-web module has been renamed to documentation, and a new documentation ModuleConfig is generated automatically.

  Steps to troubleshoot:

  1. Remove the deprecated deckhouse-web ModuleConfig from the CI deployment process.

  2. Delete it using the following command:

     d8 k delete mc deckhouse-web
     

Module istio

• D8IstioActiveWaypointsWithAmbientDisabled EE S6
  Active Istio waypoint exists while ambient mesh is disabled.

  Deckhouse has detected an active waypoint Deployment DEPLOYMENT_NAME in namespace NAMESPACE, created for WaypointInstance XXX, while ambient mesh is disabled.

  This usually happens if ambient mesh was enabled, a WaypointInstance was created, and then ambient mesh was disabled. In this state the waypoint-controller is not running, so it cannot reconcile or clean up WaypointInstance resources.

  To inspect affected resources, run:

  d8 k -n NAMESPACE get waypointinstance XXX -o yaml
  d8 k -n NAMESPACE get deploy,svc,sa,gateway,pdb,hpa,vpa -l istio.deckhouse.io/waypoint-instance=XXX
  

  Recommended remediation:

  1. Temporarily enable ambient mesh.
  2. Wait until waypoint-controller starts.
  3. Delete the affected WaypointInstance or change configuration as needed.
  4. Wait until managed waypoint resources are reconciled as needed or removed.
  5. Disable ambient mesh again.

  If ambient mesh cannot be re-enabled, manually remove the managed waypoint resources and then remove the WaypointInstance finalizer only after verifying that no managed resources remain.

• D8IstioActualDataPlaneVersionNotEqualDesired EE S8
  There are pods with Istio data plane version VERSION_NUMBER, but desired version is XXX.

  There are pods in the NAMESPACE namespace with Istio data plane version VERSION_NUMBER, while the desired version is XXX. As a result, the Istio version will be changed after the pod is restarted.

  To resolve the issue, use the following cheat sheet:

  ### Namespace-wide configuration
  # `istio.io/rev=vXYZ`: Use a specific revision.
  # `istio-injection=enabled`: Use the global revision.
  d8 k get ns NAMESPACE --show-labels
  
  ### Pod-wide configuration
  d8 k -n NAMESPACE get pods -l istio.io/rev=DESIRED_VISION
  

• D8IstioActualVersionIsNotInstalled EE S4
  The control plane version for pods with injected sidecars isn't installed.

  There are pods in the NAMESPACE namespace with injected sidecars of version VERSION_NUMBER (revision REVISION_NUMBER), but the corresponding control plane version is not installed. As a result, these pods have lost synchronization with the state in Kubernetes.

  To resolve this issue, install the required control plane version. Alternatively, update the namespace or pod configuration to match an installed control plane version.

  To identify orphaned pods, run the following command:

  d8 k -n NAMESPACE get pods -l 'service.istio.io/canonical-name' -o json | jq --arg revision REVISION_NUMBER '.items[] | select(.metadata.annotations."sidecar.istio.io/status" // "{}" | fromjson | .revision == $revision) | .metadata.name'
  

• D8IstioAdditionalControlplaneDoesntWork CE S4
  Additional control plane isn't working.

  Deckhouse has detected that the additional Istio control plane ISTIO_REVISION_LABEL isn’t working.

  As a result, sidecar injection for pods with ISTIO_REVISION_LABEL isn’t working as well.

  To check the status of the control plane pods, run the following command:

  d8 k get pods -n d8-istio -l istio.io/rev=ISTIO_REVISION_LABEL
  

• D8IstioAllianceAmbientGatewayAddressUnusable EE S6
  The ambient east-west gateway has no address peers can dial.

  The ambient east-west gateway is deployed in this cluster, but publishes no address peer clusters can dial: its Service is missing, it has no external address, no node is running a gateway pod, or every address it has is neither an IP address nor a usable DNS name. The metadata-exporter pod logs in d8-istio say which.

  Peer clusters keep using the addresses they fetched last, if any. Those that have none cannot reach ambient-mode workloads here.

  To fix, set alliance.ambientGateway.advertise to the IP address or DNS name peers should dial. Or make the load balancer assign an address (alliance.ambientGateway.loadBalancerClass, .serviceAnnotations). Or switch alliance.ambientGateway.inlet to NodePort and check that gateway pods are scheduled (.nodeSelector, .tolerations). Or, if ambient multicluster is not needed, switch all multicluster members back to sidecar mode.

• D8IstioAllianceDataPlaneNotConnected EE S4
  Data-plane connectivity check failed for XXX NAME.

  The data-plane health check of XXX NAME with uuid XXX has been failing for more than 5 minutes.

  This means that actual service mesh traffic between clusters is not flowing properly, even though the control plane metadata exchange may be working.

  Possible causes:

  • The remote cluster’s alliance-healthcheck pod is not running.
  • Istio sidecar injection is not working properly on the healthcheck pods.
  • The ingress gateway between clusters is misconfigured or unreachable.
  • mTLS or trust domain configuration issues between clusters.

  To check the status:

  d8 k get XXX NAME -o json | jq '.status.conditions[] | select(.type=="DataplaneConnectionReady")'
  

• D8IstioAllianceDiscoveryMetadataEndpointRequestedViaDeprecatedSubdomain EE S5
  A peer cluster requests private metadata using a deprecated subdomain.

  Remote XXX NAME, (UUID: XXX) requests private metadata from hostname XXX, that hostname uses the deprecated istio subdomain.

  Update the remote cluster XXX resource: set spec.metadataEndpoint to (publicDomainTemplate + istio-metadata)/metadata/.

  Migration steps on the remote cluster:

  1. Ensure DNS resolves the new target (publicDomainTemplate + istio-metadata).
  2. Update spec.metadataEndpoint to use the new domain name. Keep the path /metadata/ unchanged.
  3. Verify remote metadata fetch and alliance health after the change.
• D8IstioAllianceIngressGatewayAddressUnusable EE S6
  The ingress gateway for federation and multicluster has no address peers can dial.

  The ingress gateway that peer clusters reach this cluster through is deployed, but publishes no address they can dial: its Service is missing, it has no external address, or no node is running a gateway pod. The metadata-exporter pod logs in d8-istio say which.

  Peer clusters keep using the addresses they fetched last, if any. Those that have none cannot reach federated services or, over multicluster, sidecar-mode workloads here.

  To fix, set alliance.ingressGateway.advertise to the addresses peers should dial. Or make the load balancer assign an address (alliance.ingressGateway.loadBalancerClass, .serviceAnnotations). Or switch alliance.ingressGateway.inlet to NodePort and check that gateway pods are scheduled (.nodeSelector, .tolerations).

• D8IstioConfigAnalysisConflictingGateways EE S5
  Conflicting Istio Gateways detected (IST0145).

  The Istio configuration analyzer found conflicting Gateways in revision REVISION_NUMBER.

  Code: XXX (ConflictingGateways) Namespace: NAMESPACE Resource: XXX

  Two or more Gateways share the same selector, port, and host match. Ingress traffic may be routed unpredictably or return 404.

  Steps to resolve:

  1. List Gateways and compare spec.selector, server ports, and hosts for overlaps involving XXX.
  2. Deduplicate or narrow hosts/ports so each combination is unique.
  3. See https://istio.io/latest/docs/reference/config/analysis/ist0145/

  List matching issues in Prometheus (Grafana → Explore):

  d8_istio_config_analysis_issue{type="ConflictingGateways",revision="REVISION_NUMBER"}
  

  Check analyzer logs:

  d8 k -n d8-istio logs -l app=istio-config-analyzer,istio.io/rev=REVISION_NUMBER --tail=100
  

• D8IstioConfigAnalysisError EE S5
  Istio configuration analysis detected an error-level issue.

  The Istio configuration analyzer found an error in revision REVISION_NUMBER.

  Type: ERROR_TYPE (XXX) Namespace: NAMESPACE Resource: XXX

  Inspect the reported resource in namespace NAMESPACE.

  Documentation for analysis codes: https://istio.io/latest/docs/reference/config/analysis/

  List analysis issues in Prometheus (Grafana → Explore):

  d8_istio_config_analysis_issue{revision="REVISION_NUMBER",severity="Error"}
  

  Check analyzer pods and logs:

  d8 k -n d8-istio get pods -l app=istio-config-analyzer,istio.io/rev=REVISION_NUMBER
  d8 k -n d8-istio logs -l app=istio-config-analyzer,istio.io/rev=REVISION_NUMBER --tail=100
  

• D8IstioConfigAnalysisImageAutoWithoutInjection EE S5
  Pod uses image auto without Istio injection (IST0147).

  The Istio configuration analyzer found pods with image: auto that are not targeted for sidecar injection in revision REVISION_NUMBER.

  Code: XXX (ImageAutoWithoutInjectionError) Namespace: NAMESPACE Resource: XXX

  image: auto is rewritten only when the Istio injector mutates the pod. Without injection the container image stays literally auto and the pod will not run correctly.

  Steps to resolve:

  1. Enable sidecar injection for the namespace or workload (istio-injection=enabled / istio.io/rev=REVISION_NUMBER, or a matching RevisionTag).
  2. Restart / recreate the affected pods after injection is enabled.
  3. See https://istio.io/latest/docs/reference/config/analysis/ist0147/

  List matching issues in Prometheus (Grafana → Explore):

  d8_istio_config_analysis_issue{type="ImageAutoWithoutInjectionError",revision="REVISION_NUMBER"}
  

  Check analyzer logs:

  d8 k -n d8-istio logs -l app=istio-config-analyzer,istio.io/rev=REVISION_NUMBER --tail=100
  

• D8IstioConfigAnalysisInvalidGatewayCredential EE S5
  Gateway TLS credential is invalid (IST0161).

  The Istio configuration analyzer found an invalid Gateway credential in revision REVISION_NUMBER.

  Code: XXX (InvalidGatewayCredential) Namespace: NAMESPACE Resource: XXX

  The Secret referenced by the Gateway for TLS is missing or does not contain a usable certificate/key. HTTPS listeners on that Gateway will fail.

  Steps to resolve:

  1. Open the Gateway XXX in namespace NAMESPACE and note credentialName / TLS secret references.
  2. Verify the Secret exists in the same namespace as the ingress Gateway workload and has valid tls.crt / tls.key (or the expected keys).
  3. See https://istio.io/latest/docs/reference/config/analysis/ist0161/

  List matching issues in Prometheus (Grafana → Explore):

  d8_istio_config_analysis_issue{type="InvalidGatewayCredential",revision="REVISION_NUMBER"}
  

  Check analyzer logs:

  d8 k -n d8-istio logs -l app=istio-config-analyzer,istio.io/rev=REVISION_NUMBER --tail=100
  

• D8IstioConfigAnalysisReferencedResourceNotFound EE S5
  Istio resource references a missing dependency (IST0101).

  The Istio configuration analyzer found a missing reference in revision REVISION_NUMBER.

  Code: XXX (ReferencedResourceNotFound) Namespace: NAMESPACE Resource: XXX

  An Istio object points to a Gateway, Service, ServiceEntry, or another dependency that does not exist (typo, wrong namespace, or resource not created yet). Traffic to that route may fail.

  Steps to resolve:

  1. Inspect the reported resource in namespace NAMESPACE and fix the broken reference (use FQDN for cross-namespace hosts, e.g. reviews.backend.svc.cluster.local).
  2. Confirm the target object exists.
  3. See https://istio.io/latest/docs/reference/config/analysis/ist0101/

  List matching issues in Prometheus (Grafana → Explore):

  d8_istio_config_analysis_issue{type="ReferencedResourceNotFound",revision="REVISION_NUMBER"}
  

  Check analyzer logs:

  d8 k -n d8-istio logs -l app=istio-config-analyzer,istio.io/rev=REVISION_NUMBER --tail=100
  

• D8IstioConfigAnalysisSchemaValidationError EE S5
  Istio resource failed schema validation (IST0106).

  The Istio configuration analyzer found a schema validation error in revision REVISION_NUMBER.

  Code: XXX (SchemaValidationError) Namespace: NAMESPACE Resource: XXX

  The resource does not match the Istio API schema and may be ignored or rejected by the control plane.

  Steps to resolve:

  1. Inspect XXX in namespace NAMESPACE and compare fields with the Istio API for your control plane version.
  2. Fix invalid or unknown fields, then re-apply the manifest.
  3. See https://istio.io/latest/docs/reference/config/analysis/ist0106/

  List matching issues in Prometheus (Grafana → Explore):

  d8_istio_config_analysis_issue{type="SchemaValidationError",revision="REVISION_NUMBER"}
  

  Check analyzer logs:

  d8 k -n d8-istio logs -l app=istio-config-analyzer,istio.io/rev=REVISION_NUMBER --tail=100
  

• D8IstioConfigAnalysisWarning EE S4
  Istio configuration analysis detected a warning-level issue.

  The Istio configuration analyzer found a warning in revision REVISION_NUMBER.

  Type: ERROR_TYPE (XXX) Namespace: NAMESPACE Resource: XXX

  Inspect the reported resource in namespace NAMESPACE.

  Documentation for analysis codes: https://istio.io/latest/docs/reference/config/analysis/

  List analysis warnings in Prometheus (Grafana → Explore):

  d8_istio_config_analysis_issue{revision="REVISION_NUMBER",severity="Warning"}
  

  Check analyzer pods and logs:

  d8 k -n d8-istio get pods -l app=istio-config-analyzer,istio.io/rev=REVISION_NUMBER
  d8 k -n d8-istio logs -l app=istio-config-analyzer,istio.io/rev=REVISION_NUMBER --tail=100
  

• D8IstioCustomCAMaterialInvalid CE S4
  The Istio custom CA material is invalid but is being served to istiod.

  The Istio module is publishing CA material (XXX) that failed validation to istiod.

  The module never hard-blocks on module-owned or last-good material (doing so would rotate the mesh root and break mTLS cluster-wide), so it keeps serving this material with only a warning. However, istiod may reject it, which can prevent it from issuing or renewing workload certificates.

  Depending on the source:

  • cacerts — the d8-istio/cacerts Secret (or the CA reused from it) is not a valid CA. Inspect the Secret and restore valid ca-cert.pem/ca-key.pem (and root-cert.pem/cert-chain.pem if used).
  • secretRef:namespace/name — the last-good CA that was previously resolved from this ca.secretRef has become invalid (for example, its chain expired). Fix the referenced Secret so the next re-resolution (every 5 minutes) succeeds.

  Refer to the custom CA documentation.

• D8IstioCustomCASecretRefUnresolved CE S5
  The Istio custom CA ca.secretRef cannot be resolved.

  The Istio module could not re-resolve the configured ca.secretRef (XXX) and is keeping the last successfully-resolved CA.

  The mesh is not regressed — istiod keeps running on the exact CA material it already had — but any change made to the source Secret is not being picked up. Common causes:

  • the referenced Secret was deleted or renamed;
  • the referenced Secret was made malformed (not valid PEM, not a CA certificate, or a mismatched key pair);
  • a transient Kubernetes API problem.

  The module re-resolves the Secret every 5 minutes; this alert clears automatically once resolution succeeds. Inspect the referenced Secret and restore valid CA material.

  Refer to the custom CA documentation.

• D8IstioDataPlaneVersionMismatch EE S8
  There are pods with data plane version different from the control plane version.

  There are pods in the NAMESPACE namespace with Istio data plane version VERSION_NUMBER, which is different from the control plane version DESIRED_VERSION.

  Steps to resolve the issue:

  1. Restart affected pods and use the following PromQL query to get a full list:

     max by (namespace, dataplane_pod) (d8_istio_dataplane_metadata{full_version="VERSION_NUMBER"})
     

  2. Use the automatic Istio data plane upgrade.

• D8IstioDataPlaneWithoutIstioInjectionConfigured EE S4
  Detected pods with Istio sidecars but istio-injection isn't configured.

  There are pods in the NAMESPACE namespace with Istio sidecars, but istio-injection isn’t configured. As a result, these pods will lose their Istio sidecars after being recreated.

  To identify the affected pods, run the following command:

  d8 k -n NAMESPACE get pods -o json | jq -r --arg revision REVISION_NUMBER '.items[] | select(.metadata.annotations."sidecar.istio.io/status" // "{}" | fromjson | .revision == $revision) | .metadata.name'
  

• D8IstioDeprecatedIstioVersionInstalled CE
  The installed Istio version has been deprecated.

  Deckhouse has detected that a deprecated Istio version VERSION_NUMBER is installed.

  Support for this version will be removed in upcoming Deckhouse releases. The higher the alert severity, the greater the probability of support being discontinued.

  To learn how to upgrade Istio, refer to the upgrade guide.

• D8IstioDesiredVersionIsNotInstalled EE S6
  Desired control plane version isn't installed.

  There is a desired Istio control plane version XXX (revision REVISION_NUMBER) configured for pods in the NAMESPACE namespace, but that version isn’t installed. As a result, pods can’t be recreated in the NAMESPACE namespace.

  To resolve this issue, install the desired control plane version. Alternatively, update the namespace or pod configuration to match an installed control plane version.

  Use the following cheat sheet:

  ### Namespace-wide configuration
  # `istio.io/rev=vXYZ`: Use a specific revision.
  # `istio-injection=enabled`: Use the global revision.
  d8 k get ns NAMESPACE --show-labels
  
  ### Pod-wide configuration
  d8 k -n NAMESPACE get pods -l istio.io/rev=REVISION_NUMBER
  

• D8IstioFederationMetadataEndpointDoesntWork EE S6
  IstioFederation FEDERATION_NAME gets no usable metadata from the remote cluster.

  The Deckhouse hook could not get usable metadata from the endpoint ENDPOINT_NAME for IstioFederation FEDERATION_NAME.

  The reason and the details are in the PublicMetadataExchangeReady and PrivateMetadataExchangeReady conditions of the IstioFederation. The endpoint may respond normally while the metadata is still unusable, for example, when the ingress gateway of the remote cluster has no address yet.

  To view the conditions, run the following command:

  d8 k get istiofederation FEDERATION_NAME -o jsonpath='{range .status.conditions[*]}{.type}: {.status}: {.reason}: {.message}{"\n"}{end}'
  

  To reproduce the request to the public endpoint, run the following command:

  curl ENDPOINT_NAME
  

  To reproduce the request to private endpoints (run from the Deckhouse pod), run the following:

  KEY="$(deckhouse-controller module values istio -o json | jq -r .internal.remoteAuthnKeypair.priv)"
  LOCAL_CLUSTER_UUID="$(deckhouse-controller module values -g istio -o json | jq -r .global.discovery.clusterUUID)"
  REMOTE_CLUSTER_UUID="$(d8 k get istiofederation FEDERATION_NAME -o json | jq -r .status.metadataCache.public.clusterUUID)"
  TOKEN="$(deckhouse-controller helper gen-jwt --private-key-path <(echo "$KEY") --claim iss=d8-istio --claim sub=$LOCAL_CLUSTER_UUID --claim aud=$REMOTE_CLUSTER_UUID --claim scope=private-federation --ttl 1h)"
  curl -H "Authorization: Bearer $TOKEN" ENDPOINT_NAME
  

• D8IstioFederationRemoteClusterUUIDMismatch EE S4
  IstioFederation peer cluster UUID changed.

  Public metadata from ENDPOINT_NAME for IstioFederation FEDERATION_NAME reports cluster UUID XXX, but the previously pinned UUID is XXX.

  Deckhouse stops private metadata exchange and does not send a federation JWT until the peer identity is confirmed again.

  Check whether spec.metadataEndpoint was pointed to another cluster or the remote endpoint was replaced.

  To inspect the resource:

  d8 k get istiofederation FEDERATION_NAME -o yaml
  curl ENDPOINT_NAME
  

  If the peer change is intentional, recreate the IstioFederation resource so a new UUID can be pinned.

• D8IstioGlobalControlplaneDoesntWork CE S4
  Global control plane isn't working.

  Deckhouse has detected that the global Istio control plane ISTIO_REVISION_LABEL isn’t working.

  As a result, sidecar injection for pods with global revision isn’t working as well, and the validating webhook for Istio resources is absent.

  To check the status of the control plane pods, run the following command:

  d8 k get pods -n d8-istio -l istio.io/rev=ISTIO_REVISION_LABEL
  

• D8IstioMulticlusterMetadataEndpointDoesntWork EE S6
  IstioMulticluster MULTICLUSTER_NAME gets no usable metadata from the remote cluster.

  The Deckhouse hook could not get usable metadata from the endpoint ENDPOINT_NAME for IstioMulticluster MULTICLUSTER_NAME.

  The reason and the details are in the PublicMetadataExchangeReady and PrivateMetadataExchangeReady conditions of the IstioMulticluster. The endpoint may respond normally while the metadata is still unusable, for example, when the remote cluster does not deploy the gateway this cluster reaches it through.

  To view the conditions, run the following command:

  d8 k get istiomulticluster MULTICLUSTER_NAME -o jsonpath='{range .status.conditions[*]}{.type}: {.status}: {.reason}: {.message}{"\n"}{end}'
  

  To reproduce the request to the public endpoint, run the following command:

  curl ENDPOINT_NAME
  

  To reproduce the request to private endpoints (run from the d8-system/deckhouse pod), run the following:

  KEY="$(deckhouse-controller module values istio -o json | jq -r .internal.remoteAuthnKeypair.priv)"
  LOCAL_CLUSTER_UUID="$(deckhouse-controller module values -g istio -o json | jq -r .global.discovery.clusterUUID)"
  REMOTE_CLUSTER_UUID="$(d8 k get istiomulticluster MULTICLUSTER_NAME -o json | jq -r .status.metadataCache.public.clusterUUID)"
  TOKEN="$(deckhouse-controller helper gen-jwt --private-key-path <(echo "$KEY") --claim iss=d8-istio --claim sub=$LOCAL_CLUSTER_UUID --claim aud=$REMOTE_CLUSTER_UUID --claim scope=private-multicluster --ttl 1h)"
  curl -H "Authorization: Bearer $TOKEN" ENDPOINT_NAME
  

• D8IstioMulticlusterRemoteAPIHostDoesntWork EE S6
  Multicluster remote API host health check failed.

  The remote API host API_HOST for IstioMulticluster MULTICLUSTER_NAME has failed the health check performed by the Deckhouse monitoring hook.

  To reproduce the request (run from the d8-system/deckhouse pod), run the following:

  TOKEN="$(deckhouse-controller module values istio -o json | jq -r --arg ah API_HOST '.internal.multiclusters[]| select(.apiHost == $ah)| .apiJWT ')"
  curl -H "Authorization: Bearer $TOKEN" https://API_HOST/version
  

• D8IstioMulticlusterRemoteClusterUUIDMismatch EE S4
  IstioMulticluster peer cluster UUID changed.

  Public metadata from ENDPOINT_NAME for IstioMulticluster MULTICLUSTER_NAME reports cluster UUID XXX, but the previously pinned UUID is XXX.

  Deckhouse stops private metadata exchange and does not send a multicluster JWT until the peer identity is confirmed again.

  Check whether spec.metadataEndpoint was pointed to another cluster or the remote endpoint was replaced.

  To inspect the resource:

  d8 k get istiomulticluster MULTICLUSTER_NAME -o yaml
  curl ENDPOINT_NAME
  

  If the peer change is intentional, recreate the IstioMulticluster resource so a new UUID can be pinned.

• D8IstioNetworkNameIsNotLabelValue CE S6
  The Istio network name is not a valid Kubernetes label value.

  The Istio network name XXX of this cluster is not a valid Kubernetes label value. Because of this, the name is left out of every place where a label value is required.

  Contact Deckhouse technical support to resolve the issue.

• D8IstioOperatorReconcileError CE S5
  The istio-operator is unable to reconcile Istio control plane setup.

  Deckhouse has detected an error in the istio-operator reconciliation loop.

  To investigate the issue, check the operator logs:

  d8 k -n d8-istio logs -l app=operator,revision=REVISION_NUMBER
  

• D8IstioPodsWithoutIstioSidecar EE S4
  Detected pods without Istio sidecars but with istio-injection configured.

  There is a pod POD_NAME in the NAMESPACE namespace without Istio sidecars, but with istio-injection configured.

  To identify the affected pods, run the following command:

  d8 k -n NAMESPACE get pods -l '!service.istio.io/canonical-name' -o json | jq -r '.items[] | select(.metadata.annotations."sidecar.istio.io/inject" != "false") | .metadata.name'
  

• D8IstioRemoteClusterNotSynced EE S4
  XXXIstioMulticluster MULTICLUSTER_NAME is not synchronized.XXXIstio remote cluster CLUSTER_ID is not synchronized.XXX

  XXXThe Istio control plane instance INSTANCE_NAME cannot synchronize with the remote cluster described by IstioMulticluster MULTICLUSTER_NAME (cluster ID CLUSTER_ID).XXXThe Istio control plane instance INSTANCE_NAME cannot synchronize with the remote cluster CLUSTER_ID.XXX

  Possible causes:

  • The remote cluster is offline.
  • The remote API endpoint is not reachable.
  • The remote ServiceAccount token is invalid or expired.
  • There is a TLS or certificate issue between the clusters.
  • Remote and local Istio versions mismatch.

  XXXCheck the conditions of the IstioMulticluster:

  d8 k get istiomulticluster MULTICLUSTER_NAME -o jsonpath='{range .status.conditions[*]}{.type}: {.status}: {.reason}: {.message}{"\n"}{end}'
  

  All conditions must be True. If a condition is not True, its reason and message say what is wrong. Access to the remote API is reported by the RemoteAPIServerReady condition.

  If there is no such resource, this cluster was registered by the secret XXX alone — check the API address and the token in the kubeconfig it stores. XXXNo IstioMulticluster resource registered this cluster, so check the API address and the token in the kubeconfig stored in the secret XXX. XXX

• D8IstioReservedUIDConflict CE S4
  Application container is running as UID 1337 (reserved by Istio).

  Container CONTAINER_NAME in pod POD_NAME in namespace NAMESPACE is running as UID 1337.

  UID 1337 is reserved by Istio for the istio-proxy sidecar container. When application containers run with this UID, their traffic bypasses Istio completely — no routing rules, mTLS, or telemetry will be applied.

  To fix the issue, change the runAsUser in the container’s or pod’s securityContext to a different UID.

• D8IstioVersionIsIncompatibleWithK8sVersion CE S3
  The installed Istio version is incompatible with the Kubernetes version.

  The installed Istio version VERSION_NUMBER may not work properly with the current Kubernetes version VERSION_NUMBER because it’s not supported officially.

  To resolve the issue, upgrade Istio following the guide.

• IstioIrrelevantExternalServiceFound CE S5
  External service found with irrelevant ports specifications.

  A service NAME in the NAMESPACE namespace has an irrelevant port specification.

  The .spec.ports[] field isn’t applicable for services of the ExternalName type. However, Istio renders port listeners for external services as 0.0.0.0:port, which captures all traffic to the specified port. This can cause problems for services that aren’t registered in the Istio registry.

  To resolve the issue, remove the .spec.ports section from the service configuration. It is safe.

Module kube-dns

• KubernetesCoreDNSHasCriticalErrors CE S5
  Critical errors found in CoreDNS.

  Deckhouse has detected at least one critical error in the CoreDNS pod POD_NAME.

  To resolve the issue, review the container logs:

  d8 k -n kube-system logs POD_NAME
  

Module metallb

• D8MetallbBGPSessionDown EE S4
  MetalLB BGP session is down.

  JOB_NAME, MetalLB CONTAINER_NAME on POD_NAME has BGP session PEER down.

  Check the logs for details:

  d8 k -n d8-metallb logs daemonset/speaker -c speaker
  

• D8MetallbConfigNotLoaded EE S4
  The MetalLB configuration hasn't been loaded.

  JOB_NAME, MetalLB CONTAINER_NAME on POD_NAME hasn’t been loaded.

  To find the cause of the issue, review the controller logs:

  d8 k -n d8-metallb logs deploy/controller -c controller
  

• D8MetallbConfigStale EE S4
  MetalLB is running on a stale configuration.

  JOB_NAME, MetalLB CONTAINER_NAME on POD_NAME is running on a stale configuration because the latest configuration failed to load.

  To find the cause of the issue, review the controller logs:

  d8 k -n d8-metallb logs deploy/controller -c controller
  

• D8MetallbNotSupportedServiceAnnotationsDetected SE S4
  The annotation 'ANNOTATION_NAME' has been deprecated for the service 'NAME' in the 'NAMESPACE' namespace.

  The annotation ‘ANNOTATION_NAME’ has been deprecated for the service ‘NAME’ in the ‘NAMESPACE’ namespace.

  The following service annotations are no longer effective:

  • metallb.universe.tf/ip-allocated-from-pool: Remove this annotation.
  • metallb.universe.tf/address-pool: Replace it with the .spec.loadBalancerClass parameter or use the network.deckhouse.io/metal-load-balancer-class annotation, referencing the appropriate MetalLoadBalancerClass.
  • metallb.universe.tf/loadBalancerIPs: Replace it with network.deckhouse.io/load-balancer-ips: IP.
  • metallb.universe.tf/allow-shared-ip: Replace it with network.deckhouse.io/load-balancer-shared-ip-key.
  • metallb.io/ip-allocated-from-pool: Remove this annotation.
  • metallb.io/address-pool: Replace it with the .spec.loadBalancerClass parameter or use the network.deckhouse.io/metal-load-balancer-class annotation, referencing the appropriate MetalLoadBalancerClass.
  • metallb.io/loadBalancerIPs: Replace it with network.deckhouse.io/load-balancer-ips: IP.
  • metallb.io/allow-shared-ip: Replace it with network.deckhouse.io/load-balancer-shared-ip-key.

  Please note. Existing LoadBalancer services of Deckhouse have been migrated automatically, but the new ones will not be.

• D8MetallbObsoleteLayer2PoolsAreUsed SE S7
  The metallb module has obsolete layer2 pools configured.

  In ModuleConfig version 2, addressPool ‘NAME’ of type “layer2” are ignored. They should be removed from the configuration.

• D8MetallbUpdateMCVersionRequired SE S5
  The metallb ModuleConfig settings are outdated.

  D8 MetalLB settings are outdated.

  To resolve this issue, increase version in the ModuleConfig metallb.

• L2LoadBalancerOrphanServiceFound SE S4
  Orphaned service with an irrelevant L2LoadBalancer name has been found.

  The cluster contains an orphaned service NAME in the NAMESPACE namespace with an irrelevant L2LoadBalancer name.

  To resolve this issue, verify the L2LoadBalancer name specified in the annotation network.deckhouse.io/l2-load-balancer-name.

Module multitenancy-manager

• ClusterResourceGrantPolicyViolation CE S6
  Objects in namespace XXX use cluster-wide resources that are not available to its project.

  Objects in namespace XXX reference cluster-wide resources (for example, a StorageClass or a ClusterRole) that the current ClusterResourceGrantPolicy objects do not make available to the project of this namespace. The project label of the metric holds the namespace, not the project name. Such an object was created before the policy changed. It keeps working, and when it is updated, only new values are checked.

  Steps to troubleshoot:

  1. Find the objects, their fields and the resources on the “Security” → “Cluster Resource Grant Violations” Grafana dashboard, or in the d8_cluster_objects_grant_violated{project="XXX"} metric (labels violating_object_name, violating_field, violating_resource and grant).

  2. Check which resources are available to the project:

     d8 k get available RESOURCE_NAME -n XXX -o yaml
     

  3. Make the resource available to the project in a ClusterResourceGrantPolicy, or change the field of the object to an available value.
• D8NamespaceNotAdopted CE S6
  Namespace XXX belongs to no project.

  Namespace XXX was created directly more than 10 minutes ago and is still not a project. The multitenancy-manager controller turns every namespace outside the system ones into a project of its own within seconds, whatever the allowNamespacesWithoutProjects parameter says. Until it does, the objects created in the namespace skip the checks of a project: the availability of cluster-wide resources, the project template with its network and pod security policies, and the quota.

  The controller has no leader, fails to adopt the namespace, or has adopted it into a Project that does not reconcile. Check its pods and logs:

  d8 k -n d8-multitenancy-manager get pods
  d8 k -n d8-multitenancy-manager logs deploy/multitenancy-manager -c multitenancy-manager --tail=200 | grep XXX
  

  If the controller fails to adopt the namespace, it tries again with a delay of up to 30 seconds, so the namespace becomes a project and the alert resolves soon after the cause is gone, for example an admission policy or webhook that refuses the Project. Once the controller adopts the namespace, check the objects created in it in the meantime.

  If a Project named like the namespace exists already, the controller takes the adoption as done and does not try again, and the logs of the namespace controller show nothing. The project may be in Error, for example with the HelmOwnership condition when the namespace belongs to another Helm release, and then it never labels the namespace: until the conflict is removed, this Project is not updated at all. Check the project:

  d8 k get project XXX -o jsonpath='{.status.state}{"\n"}{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'
  

  Remove the cause the condition names, for example the foreign release or its meta.helm.sh/* annotations on the namespace, and the project labels the namespace. This also happens to the namespaces that existed before the upgrade.

• D8ProjectQuotaExceeded CE S7
  Project XXX has a ProjectQuota set below its usage.

  The usage of project XXX, summed over all of its namespaces, is above the limits of its ProjectQuota: the limits were set below the usage, by a change of the quota or when it was created (the HardLowered reason of the Exceeded condition).

  Nothing running is evicted. Creating more of the resources whose limits are exceeded is denied until the usage is back within the limits.

  1. See which limits are exceeded and by how much:

     d8 k get projectquota XXX -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'
     

  2. See which namespaces use the most (the usage of every namespace is in its d8-project-quota ResourceQuota):

     d8 k get projectquota XXX -o jsonpath='{range .status.topNamespaces[*]}{.name}: {.used}{"\n"}{end}'
     d8 k get resourcequota -A -l projects.deckhouse.io/project=XXX
     

  3. Raise the limits in spec.hard of the ProjectQuota (or in spec.quota of the Project, if the quota comes from there), or delete what the project does not need.

• D8ProjectQuotaLeaderFlapping CE S5
  The leader of the project quota ledger changes too often.

  The lease of the ledger changed hands more than three times in ten minutes; every change refuses counted creates for a second or more. The reason label of d8_multitenancy_quota_ledger_handover_total says why: expired points at late renewals (the API server, the leader-election priority level), stepdown at the canary, graceful at restarts of the pods (rollouts, evictions).

• D8ProjectQuotaLedgerCanaryFailing CE S4
  Most API servers do not reach the leader of the project quota ledger.

  The canary writes of the ledger do not reach its webhook through most of the API servers. The leader hands the lease over when another replica is ready; this alert says it could not, or that the new leader is not reached either. Check the network from the master nodes to the pods of d8-multitenancy-manager and the CNI on the masters. The steps of D8ProjectQuotaLedgerDown apply.

• D8ProjectQuotaLedgerDown CE S4
  No replica of the project quota ledger answers.

  No replica of the quota-ledger container of the multitenancy-manager module holds its lease with synced informers. Creating the objects a ProjectQuota counts in the namespaces of projects with a quota fails with project quota ledger unavailable, or with failed calling webhook "quota-ledger.multitenancy-manager.deckhouse.io". Deletions, running objects and other namespaces are not affected.

  1. Check the pods, the lease and the route:

     d8 k -n d8-multitenancy-manager get pods,lease,endpointslices
     d8 k -n d8-multitenancy-manager logs deploy/multitenancy-manager -c quota-ledger --tail=200
     

     The logs say whether the ledger is fenced (its lease renewals are late) or syncing.

  2. If the cause cannot be removed quickly, delete the webhook as system:sudouser. While no replica holds the route, the API servers reach none of them, so suspending the ledger with the quota-ledger-suspend ConfigMap does not help here. The sum check stops at once; Deckhouse puts the webhook back within about 4 minutes, so this buys time rather than switching the check off. Disabling the multitenancy-manager module removes the check together with the rest of the module.

     d8 k --as=system:sudouser delete validatingwebhookconfiguration d8-multitenancy-manager-quota-ledger
     

• D8ProjectQuotaLedgerDrift CE S6
  The project quota ledger differs from the API server.

  The ledger holds the usage of some quota differently from a quorum read of the same version of it. The state of the ledger is inconsistent. Save the logs of the quota-ledger container, then restart the pod of the leader once so that another replica takes over. Do not restart it in a loop.

• D8ProjectQuotaLedgerSlow CE S5
  The project quota ledger answers slowly.

  The 99th percentile of the calls of the ledger webhook, as the API servers see them, is above 100 ms. The ledger answers within 25 ms of its own, so the delay comes from the network or from the CPU of the quota-ledger container. Check the CPU of the master nodes and of the container, the d8_multitenancy_quota_ledger_inflight and d8_multitenancy_quota_ledger_requests_total{result="busy"} series, and other webhooks on resourcequotas/status.

• D8ProjectQuotaLedgerUnreachableFromAPIServer CE S5
  The API server XXX does not reach the project quota ledger.

  The canary writes through the API server XXX fail on the webhook of the ledger, while the other API servers reach it. Counted creates that land on this API server in the namespaces of projects with a quota wait for the webhook timeout and fail. Check the network of that master node to the pods of d8-multitenancy-manager; the webhooks of other modules are likely affected the same way.

• D8ProjectQuotaNotEnforced CE S6
  The ProjectQuota of project XXX does not apply in every namespace of the project.

  The Enforced condition of the ProjectQuota of project XXX has been False for 15 minutes for one of these reasons:

  • KeyRollout: the d8-project-quota ResourceQuota of some namespace of the project is missing or not calculated with every limit of the quota. The resource quota controller of kube-controller-manager calculates a new quota within seconds, so a quota that stays uncalculated points at kube-controller-manager or at the multitenancy-manager controller; the message of the condition names the namespaces and the error of the controller, if any. Until it is calculated, the objects of users are refused in a namespace labelled projects.deckhouse.io/quota: pending. In a namespace labelled projects.deckhouse.io/quota: enforced, the objects the quota counts are refused only while its d8-project-quota has no calculated limit at all or has scopes; a limit added to a calculated quota holds nothing back.
  • KeyNotServed: some keys of the quota count nothing, because a count/ key names a resource of an API group that the API server does not serve, or because the module that serves the resources of a key, such as virtualization, is disabled. Enable that module, or remove the key.
  • ProjectNotFound: there is no project named after the ProjectQuota.
  • WebhookMissing: the d8-multitenancy-manager-quota-ledger webhook is missing or its caBundle is not the CA of the quota-ledger-certs Secret, so the sum is not checked. Deckhouse puts it back on the next run of the module; see the D8ProjectQuotaWebhookMisconfigured alert.
  • WebhookMisconfigured: the d8-multitenancy-manager-quota-ledger webhook is changed (its rules, namespaceSelector, objectSelector, matchConditions, failurePolicy or service) so that writes of d8-project-quota do not reach the ledger. The message of the condition names the field. Deckhouse puts the webhook back on the next run of the module; see the D8ProjectQuotaWebhookMisconfigured alert.
  • WebhookNotCalled: the canary of the ledger found API servers that admit objects without calling the d8-multitenancy-manager-quota-ledger webhook, so the sum is not checked for what they admit. The message of the condition names them; check the rules, namespaceSelector and matchConditions of the webhook.

  Check the condition and the quotas of the namespaces:

  d8 k get projectquota XXX -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'
  d8 k get resourcequota -A -l projects.deckhouse.io/project=XXX -o custom-columns=NAMESPACE:.metadata.namespace,HARD:.spec.hard,CALCULATED:.status.hard
  

• D8ProjectQuotaNotSupported CE S8
  The sum of a project quota is not enforced, because Deckhouse Platform does not manage the control plane.

  The cluster has ProjectQuota objects, but the control-plane-manager module is disabled: Deckhouse Platform does not manage the control plane of the cluster. The sum over the namespaces of a project is enforced only where it does. Every ProjectQuota reports Enforced=False with the ControlPlaneNotManaged reason; the usage is counted and shown, and every namespace is limited by its d8-project-quota ResourceQuota, so no namespace alone uses more than the quota of the whole project.

  This is expected on a cluster whose control plane is managed outside Deckhouse Platform, for example by a cloud provider. Silence the alert if that is the case.

• D8ProjectQuotaOvershoot CE S5
  Usage of project XXX has grown past its ProjectQuota.

  The usage of project XXX grew past its quota. The ledger allows that only for a status write of a quota saved long after it was approved (a change of the ledger leader at that moment, a stalled etcd, another slow webhook) or when kube-controller-manager recounts a quota, and then by at most a tenth of a limit. The Exceeded condition of the ProjectQuota names the keys, the difference and the probable cause. Nothing running is evicted; creating more of these resources is denied until the usage is back within the quota.

• D8ProjectQuotaSuspended CE S6
  The project quota ledger is suspended.

  The quota-ledger-suspend ConfigMap in d8-multitenancy-manager suspends the ledger: every write passes and the sum of the project quotas is not checked. Find who created it in the audit log and delete it when the reason is gone:

  d8 k -n d8-multitenancy-manager delete configmap quota-ledger-suspend
  

• D8ProjectQuotaWebhookErrors CE S5
  The API servers fail to call the project quota ledger.

  For 10 minutes the API servers have been failing to call the webhook of the ledger: connections refused, timeouts or TLS errors. A change of the leader causes such errors for a second or two only. The steps of D8ProjectQuotaLedgerDown apply.

• D8ProjectQuotaWebhookMisconfigured CE S4
  The webhook of the project quota ledger is missing or misconfigured.

  The d8-multitenancy-manager-quota-ledger ValidatingWebhookConfiguration is missing, does not carry the CA of the quota-ledger-certs Secret, is changed so that writes of d8-project-quota do not reach the ledger (its rules, namespaceSelector, objectSelector, matchConditions, failurePolicy or service), or the API servers let the canary writes through without calling it. The sum of the project quotas is not checked; the Enforced condition of every ProjectQuota names the reason (WebhookMissing, WebhookMisconfigured or WebhookNotCalled) and the field or the API servers. Deckhouse puts the webhook back within about 4 minutes; if it does not, contact Deckhouse technical support. Compare the CA:

  d8 k get validatingwebhookconfiguration d8-multitenancy-manager-quota-ledger -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d | openssl x509 -noout -subject
  d8 k -n d8-multitenancy-manager get secret quota-ledger-certs -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -subject
  

• D8ResourceQuotaAdmissionDegraded CE S4
  The quota admission of the API servers is slow.

  The 99th percentile of the ResourceQuota admission plugin is above one second. The plugin has a few workers for all creates in the cluster, so this slows down creates everywhere. The cause may be a slow webhook on resourcequotas/status (the project quota ledger or another one) or a slow etcd. Find the slow webhook by apiserver_admission_webhook_admission_duration_seconds and check etcd_disk_wal_fsync_duration_seconds.

• MultitenancyManagerControllerUnavailable CE S4
  The multitenancy-manager controller has had no available pods for more than 10 minutes.

  The multitenancy-manager Deployment in the d8-multitenancy-manager namespace has had no available pods for more than 10 minutes.

  While this alert is active, projects are not reconciled, and the admission webhooks of the module, which refuse a request when they cannot be reached, refuse to create or change Project, ProjectTemplate and the other objects they check.

  Steps to troubleshoot:

  1. Check the pods of the controller:

     d8 k -n d8-multitenancy-manager get pods -l app=multitenancy-manager
     d8 k -n d8-multitenancy-manager describe pods -l app=multitenancy-manager
     

  2. A pod that is running but not ready may wait for the conversion of the CRDs. The controller does not start until each of the clusterresourcegrantpolicies.multitenancy.deckhouse.io, projects.deckhouse.io and projecttemplates.deckhouse.io CRDs converts through the webhook: until then, the API server would show it objects of an older version with only apiVersion rewritten. Look for the wait in the log:

     d8 k -n d8-multitenancy-manager logs deploy/multitenancy-manager -c multitenancy-manager | grep 'conversion'
     

     • hold the controller until the webhook-handler of the deckhouse module registers the conversion webhook of the CRDs means that the controller waits; the line names the CRDs and their current conversion strategy.
     • the CRDs convert through the webhook or the CRDs already convert through the webhook means that the wait is over.

  3. If the controller waits, check the conversion strategy of the CRDs and the ConversionWebhook resources of the module. The webhook-handler of the deckhouse module sets the strategy to Webhook for every ConversionWebhook resource:

     d8 k get crd clusterresourcegrantpolicies.multitenancy.deckhouse.io projects.deckhouse.io projecttemplates.deckhouse.io -o custom-columns=NAME:.metadata.name,STRATEGY:.spec.conversion.strategy
     d8 k get conversionwebhooks.deckhouse.io clusterresourcegrantpolicies.multitenancy.deckhouse.io projects.deckhouse.io projecttemplates.deckhouse.io
     d8 k -n d8-system logs deploy/webhook-handler -c handler --tail=200
     

     Do not set the strategy by hand: the conversion has to be served by the webhook-handler, which registers it once it has loaded the ConversionWebhook resource.

• MultitenancyManagerHelmProjectTemplate CE S9
  ProjectTemplate XXX still renders its projects from a Helm text in resourcesTemplate.

  ProjectTemplate XXX describes the objects of its projects with a Helm text in resourcesTemplate, and its projects are rendered from that text. They keep working as before. This form of a template is deprecated, and resourcesTemplate goes away in a future release, so move the template to structured fields and manifests before then. Moving a template from a Helm text describes the move step by step.

  The built-in templates default, secure and secure-with-dedicated-nodes are structured and accept the same parameters as their Helm texts did, except administrators and resourceQuota, so they show how a text maps to the fields. The full mapping is in Moving a template from a Helm text. Objects of the text that have no structured field go to manifests, which the controller applies with the d8:project:admin role of the project. An AuthorizationRule of the Admin access level goes to .spec.administrators of each Project instead, because manifests does not create one. Cluster-scoped objects have no place in a template.

  Once the template has structured fields in place of resourcesTemplate, the AuthorizationRule and ResourceQuota objects that .spec.administrators and .spec.quota replace are deleted after these fields work, and an AuthorizationRule that names nobody from .spec.administrators any more is deleted at once. Every other object of the text that the structured fields do not render under the same kind and name is left in place, and the MultitenancyManagerProjectKeepsHelmTemplateObjects alert names the project until manifests takes the object over or it is deleted. The administrators then have the d8:project:admin role in place of the Admin access level, and the two roles do not grant the same permissions. See Switching a project from a Helm text.

  Steps:

  1. Read the template and compare it with the built-in template it was copied from, if any:

     d8 k get projecttemplate XXX -o yaml
     d8 k get projecttemplate default -o yaml
     

  2. List the projects of the template. A template that no project uses can be deleted, and the alert goes with it:

     d8 k get projects -l projects.deckhouse.io/project-template=XXX
     

  3. Find the projects of the template that still set administrators or resourceQuota in .spec.parameters:

     d8 k get projects -l projects.deckhouse.io/project-template=XXX -o json | jq -r '.items[] | select((.spec.parameters // {}) | has("administrators") or has("resourceQuota")) | .metadata.name'
     

     These values get no standard fields and are not handed over, so move them to .spec.administrators and .spec.quota first, and only as far as they mean the same there. .spec.administrators gives users and groups the d8:project:admin role, so an administrator with another access level or another kind of subject needs a ProjectRoleBinding of its own. .spec.quota enforces every limit it holds, 0 included, so leave out a limit that the text did not render, such as a 0 that it skipped.

  4. If the structured fields and manifests cover what the projects need, which they do unless the projects need cluster-scoped objects of the text or objects that the d8:project:admin role does not allow, write the template as deckhouse.io/v1alpha2 with these fields and manifests and without resourcesTemplate, in one change, where the template is applied from, for example in Git or in CI. A template that projects use cannot lose resourcesTemplate without structured fields or manifests in its place, and once it has fields that deckhouse.io/v1alpha1 cannot describe, a write of it through deckhouse.io/v1alpha1 is refused. If the template is an unchanged copy of a built-in template, you can instead move its projects to that template in .spec.projectTemplateName and then delete the copy.

  5. Check each project after the switch. Its HandoverPending condition is removed once the replaced objects are deleted, its ManifestsApplied condition says whether the objects of manifests are applied, and .status.keptObjects lists the objects left in place:

     d8 k get project PROJECT_NAME -o jsonpath='{range .status.conditions[*]}{.type}={.status}: {.message}{"\n"}{end}{range .status.keptObjects[*]}{.kind} {.namespace}/{.name}{"\n"}{end}'
     

• MultitenancyManagerPreviousRevisionStillRunning CE S6
  Pods of several revisions of the multitenancy-manager controller have run side by side for more than 15 minutes.

  Pods of VALUE revisions of the multitenancy-manager Deployment in the d8-multitenancy-manager namespace have been pending or running side by side for more than 15 minutes.

  The rollout of the controller is stuck: the new pod is pending or not ready, or an old pod is stuck terminating on a lost node. The new pod waits for the pods of older revisions for 10 minutes at most and then starts its controllers anyway. If an old pod of a revision that runs without leader election is still running, both pods write the Helm releases of the projects, and objects in the project namespaces may be deleted and created again.

  Steps to troubleshoot:

  1. List the controller pods with their revisions and nodes:

     d8 k -n d8-multitenancy-manager get pods -L pod-template-hash -o wide
     

  2. Find out whether the new pod has started its controllers:

     d8 k -n d8-multitenancy-manager logs NEW_POD_NAME -c multitenancy-manager | grep 'other revisions'
     

     • hold the controllers until the pods of other revisions finish means that the new pod holds the lease and waits.
     • no pods of other revisions to wait for means that it has started its controllers at once, because the old pods had finished or were deleted more than 10 minutes ago.
     • the pods of other revisions finished means that it has started its controllers after waiting, because the old pods finished or were deleted more than 10 minutes ago.
     • the pods of other revisions did not finish in time, starting the controllers anyway means that it has started its controllers after 10 minutes of waiting.

     Without any of these lines, the new pod waits for the lease that the old pod holds. When the wait ends after 10 minutes, the new pod also writes a Warning event with the reason PreviousRevisionStillRunning, and the message of the event names the old pods. An event is kept for an hour by default:

     d8 k -n d8-multitenancy-manager get events --field-selector reason=PreviousRevisionStillRunning
     

  3. Check the node of the old pod, shown in the NODE column:

     d8 k get node NODE_NAME
     

  4. Act on what you see:

     • The old pod is Running, not terminating, and the new pod is not ready. The Deployment removes the old pods only after the new ones are ready, and deleting an old pod does not help, since its ReplicaSet creates it again. Find out why the new pod is not ready:

       d8 k -n d8-multitenancy-manager describe pod NEW_POD_NAME
       

     • The old pod is Terminating on a node that is NotReady. Such a pod stays until its node comes back or the Node object is deleted. If the node is lost for good, delete the Node object, and its pods are deleted with it. If you know the machine of the node is powered off, you can force-delete the old pod instead:

       d8 k -n d8-multitenancy-manager delete pod OLD_POD_NAME --force --grace-period=0
       

       Do not force-delete a pod whose containers may still run. The pod disappears from the API, while its process keeps writing the Helm releases.

• MultitenancyManagerProjectHandoverStuck CE S6
  Project XXX still keeps the AuthorizationRule and ResourceQuota objects its standard fields replace.

  An upgrade of the release of project XXX dropped AuthorizationRule or ResourceQuota objects that the Helm text of its template had rendered for the administrators and the quota of the project. These are the AuthorizationRule objects with the Admin access level of the administrators and a quota such as all-pods whose every limit .spec.quota sets. That happens when a template switches from resourcesTemplate to structured fields, when a project moves to such a template, or when the text no longer renders these objects. The controller kept them through the upgrade, so that the administrators never lost access and the namespace never lost its quota, and deletes them once the standard fields of the project replace them. The handover began more than 30 minutes ago and has not finished.

  Until it does, the kept AuthorizationRule objects keep granting the Admin access level, the user-authz:admin role, next to the d8:project:admin role of the d8-administrators ProjectRoleBinding. The kept quotas keep applying next to the d8-project-quota ResourceQuota, and the tighter of them wins.

  The kept objects are deleted by a reconcile of the project that succeeds once all of the following hold, and not earlier than 2 minutes after d8-administrators became Ready and after the handover began:

  • the d8-administrators ProjectRoleBinding names the administrators from .spec.administrators and is Ready for its current generation, when a kept AuthorizationRule names one of them;
  • the d8:project:admin ClusterRole has its aggregated rules, when a kept AuthorizationRule names one of the administrators;
  • the d8-project-quota ResourceQuota has the limits from .spec.quota and is calculated, that is, its status.hard matches its spec.hard, when a kept quota has limits.

  A kept AuthorizationRule that names nobody from .spec.administrators any more is deleted at once, and a subject removed from .spec.administrators is removed at once from a kept AuthorizationRule that names other administrators too. A reconcile that cannot find the template, validate the project against it, upgrade the release or apply the standard fields stops before the deletion, and the kept objects stay. A template that grants a role forbidden in projects puts the project in the Error state too, but it does not stop the deletion.

  The HandoverPending condition of the project says what the controller waits for.

  Steps to troubleshoot:

  1. Read what the controller waits for, and the state and the other conditions of the project:

     d8 k get project XXX -o jsonpath='{.status.state}{"\n"}{range .status.conditions[*]}{.type}={.status}: {.message}{"\n"}{end}'
     

  2. Check the administrators binding. Its status.observedGeneration must match metadata.generation, and its Ready condition must be True:

     d8 k -n XXX get projectrolebinding d8-administrators -o yaml
     

     If the binding lags behind, the controller could not create its RoleBinding in some namespace of the project. The logs of the controller name the namespace and the error:

     d8 k -n d8-multitenancy-manager logs -l app=multitenancy-manager -c multitenancy-manager --tail=-1 --prefix | grep d8-administrators
     

  3. Check that the d8:project:admin ClusterRole has rules. The aggregation controller of kube-controller-manager fills them:

     d8 k get clusterrole d8:project:admin -o jsonpath='{.rules}'
     

  4. Check the project quota and the events in the namespace of the project:

     d8 k -n XXX get resourcequota d8-project-quota -o yaml
     d8 k -n XXX get events --sort-by=.lastTimestamp
     

     A quota whose status.hard stays empty is not processed by the resource quota controller of kube-controller-manager.

  5. Fix what the checks show, for example a RoleBinding that an admission policy refuses in one of the namespaces of the project, a kube-controller-manager that does not process the quota, or the condition that stops the reconcile of the project. The controller checks again whenever d8-administrators or d8-project-quota changes, and at least every minute, and deletes the kept objects once the conditions hold.

• MultitenancyManagerProjectKeepsHelmTemplateObjects CE S7
  Project XXX has VALUE objects its release left in place.

  The release of project XXX no longer holds VALUE objects, and the controller left them in place instead of deleting them. That happens when the project switches off a Helm text in resourcesTemplate and the structured fields do not render the object. It also happens when .spec.quota stops covering a quota kept for a handover. The release of the structured fields leaves an object in place too when manifests of the template render it, so that manifests take it over. Examples are AuthorizationRule or RoleBinding objects, quotas, network policies, workloads or their volumes. The objects left in place behave as follows:

  • They keep working as before. The rules and the bindings keep granting access, the quotas keep applying, and the workloads keep running.
  • The project no longer manages them. A change of the template or of the project does not change them. When the project is deleted, the ones in its namespaces go with the namespaces, and the controller deletes the cluster-scoped ones that still carry the label.
  • They carry the projects.deckhouse.io/kept-from-helm-template: "true" label and the helm.sh/resource-policy: keep annotation, and they have lost the heritage: multitenancy-manager label. While they carry the label, only a requester who may make the same change to objects of their kind across the cluster may change or delete them, so the users of the project may not.

  The project lists them in .status.keptObjects until each of them is deleted, loses the label, is rendered by the template again, or is taken over by the manifests of the template. Decide for each object:

  • If it is still needed, describe it in the template or in the project, for example with a ProjectRoleBinding in place of an AuthorizationRule or a RoleBinding, or with a limit in .spec.quota, and then delete the object. Or remove the label to hand it over to the project, whose users may then change and delete it.
  • If it is in a namespace of the project, manifests of the template can take it over. Once manifests renders an object of the same kind, namespace and name, the controller applies it as the system:multitenancy-manager:project:XXX user, removes the label and takes the object off the list. The object then has what manifests renders, and a field that the release set and manifests leaves out is removed.
  • If it is not needed, delete it. An access it grants ends only when it is deleted.
  • If the template renders an object of the same kind and name again, the project release takes it over. The controller then removes the label, puts the heritage: multitenancy-manager label back and removes the helm.sh/resource-policy: keep annotation, unless the template sets it itself.

  Steps:

  1. List the objects:

     d8 k get project XXX -o jsonpath='{range .status.keptObjects[*]}{.kind} {.namespace}/{.name}{"\n"}{end}'
     

  2. Keep an object as your own:

     d8 k -n NAMESPACE label KIND NAME projects.deckhouse.io/kept-from-helm-template-
     

  3. Delete an object:

     d8 k -n NAMESPACE delete KIND NAME
     

  The controller looks at the list every 10 minutes, and at a reconcile of the project at most once a minute.

• MultitenancyManagerProjectNamedLikeAdditionalNamespace CE S8
  Project XXX is named like an additional namespace of project XXX.

  The name of project XXX is the name of project XXX, a dash and a suffix. Additional namespaces of a project are named this way (project-spec.name of a ProjectNamespace), but XXX is not one of them. It is a separate project with a namespace of its own. The administrators, quota, bindings and template of project XXX do not apply to it.

  Such a pair appears when both namespaces existed before they became projects, or when a project is created under such a name. Nothing is broken, but the names read as one project to anyone who looks at them.

  If the two projects are meant to be separate, nothing has to change in the cluster. The alert keeps firing for as long as both projects exist, so silence it for this pair in Alertmanager.

  If XXX is meant to be an additional namespace of project XXX, recreate it as one. A namespace cannot move between projects in place. Deleting the project deletes its namespace with everything in it.

  1. Make sure that everything running in the XXX namespace can be deployed again, or move it elsewhere first.

  2. Find the ClusterProjectRoleBinding objects that name the ServiceAccounts of the XXX namespace. While a cluster-wide binding names them, the ProjectNamespace in step 4 is refused, so remove these subjects from the bindings first and add them back once the namespace is there again:

     d8 k get clusterprojectrolebindings -o json | jq -r --arg ns XXX '.items[] | select(any(.spec.subjects[]?; (.kind == "ServiceAccount" and .namespace == $ns) or (.kind == "User" and (.name | startswith("system:serviceaccount:" + $ns + ":"))) or (.kind == "Group" and .name == "system:serviceaccounts:" + $ns))) | .metadata.name'
     

  3. Delete the project. The command returns once the namespace is gone:

     d8 k delete project XXX
     

  4. Create a ProjectNamespace in the XXX namespace, with spec.name set to the part of the name after XXX-. The controller creates the XXX namespace again, as an additional namespace of project XXX.
  5. Deploy the workloads into the namespace.

Module node-local-dns

• D8NodeLocalDNSCacheDenialEvictionsHigh BE S6
  node-local-dns denial cache is being frequently evicted.

  The denial (NXDOMAIN) cache in node-local-dns is evicting entries frequently.

  This might indicate that there are Services that are being recreated too often in cluster. So, denial cache size is not enough (denial 65536 is currently configured).

  Affected node: NODE_NAME

• D8NodeLocalDNSCacheSuccessEvictionsHigh BE S6
  node-local-dns success cache is being frequently evicted.

  The success cache in node-local-dns is evicting entries frequently (>100 in 5 minutes).

  This could mean there are too many different DNS records application have to resolve (to many Services?). So, node-local-dns cache capacity is too small (success 120000 is currently configured), or traffic patterns changed.

  Affected node: NODE_NAME

• D8NodeLocalDNSForwardMaxConcurrentRejects BE S5
  The forward plugin is rejecting DNS queries due to max concurrency limit.

  More than 20 queries were rejected within the last 5 minutes on node NODE_NAME because the max concurrent query limit was reached.

• D8NodeLocalDNSKubeforwardRequestLatencyP95High BE S4
  High DNS request latency in kubeforward (p95 > 300ms under load).

  The 95th percentile DNS request latency for A-records through kubeforward exceeded 300 milliseconds over the last 5 minutes while handling more than 0.1 QPS.

  This may indicate slow upstreams, network issues, or DNS overload on this node.

  Affected node: NODE_NAME

  For more detailed diagnostics, inspect Top 5 upstreams by slow kubeforward requests in Grafana.

  If the alert fires only on one node, run the checks below from node NODE_NAME (for example, from a debug pod on this node). If the alert fires on many nodes, treat it as a potential systemic issue first.

  Get kubeforward upstream IPs from a master host:

  d8 k -n kube-system get po -l k8s-app=kube-dns -o jsonpath='{.items[*].status.podIP}'
  

  Then run latency checks on the affected node NODE_NAME (or from a debug pod on that node):

  UPSTREAMS="paste kube-dns pod IPs here"
  for ip in $UPSTREAMS; do
    echo "== upstream ${ip} =="
    for i in $(seq 1 20); do
      dig @"$ip" -p 5353 kubernetes.default.svc.cluster.local A +tries=1 +time=1 +stats | grep "Query time"
    done
  done
  

  To identify slow requests, enable module logs (enableLogs) and inspect slow query messages:

  pod=$(kubectl -n kube-system get po -l k8s-app=node-local-dns --field-selector spec.nodeName=NODE_NAME -o jsonpath='{.items[0].metadata.name}')
  d8 k -n kube-system logs "$pod" | grep '\[kubeforward\] slow query'
  

  Example log line: [kubeforward] slow query A kubernetes.default.svc.cluster.local. took 425ms (rcode=NOERROR, upstream=10.96.0.10:53)

  Interpretation:

  • If most upstreams are slow and many nodes are affected, suspect systemic network/CoreDNS overload.
  • If one upstream is slow or only one node is affected, suspect a localized issue.
• D8NodeLocalDNSUpstreamHealthcheckFailed BE S5
  Too many upstream health check failures in forward plugin.

  The health check system of the forward plugin failed more than 10 times within 5 minutes on node NODE_NAME. As a result, all upstreams were considered unhealthy.

Module node-manager

• CapsInstanceUnavailable CE S8
  There are unavailable instances in the MACHINE_DEPLOYMENT_NAME MachineDeployment.

  The MachineDeployment MACHINE_DEPLOYMENT_NAME has VALUE unavailable instances.

  Check the status of the instances in the cluster with the following command:

  d8 k get instance -l node.deckhouse.io/group=MACHINE_DEPLOYMENT_NAME
  

• ClusterHasOrphanedDisks CE S6
  Cloud data discoverer found orphaned disks in the cloud.

  The cloud data discoverer has found disks in the cloud that do not have a corresponding PersistentVolume in the cluster.

  You can safely delete these disks manually from your cloud provider:

  ID: ID, Name: NAME

• D8BashibleApiserverLocked CE S6
  Bashible-apiserver has been locked for too long.

  Bashible-apiserver has been locked for an extended period.

  To resolve the issue, check if the bashible-apiserver Pods are up-to-date and running:

  d8 k -n d8-cloud-instance-manager get pods -l app=bashible-apiserver
  

• D8CAPIControllerManagerNotAvailable CE S4
  No capi-controller-manager replica is available.

  No replica of capi-controller-manager has been available for more than 5 minutes.

  Its admission webhooks reject changes to Cluster API objects, so node groups served by Cluster API do not create or remove nodes.

  To inspect the pods:

  d8 k -n d8-cloud-instance-manager get pods -l app=capi-controller-manager -o wide
  

• D8CAPIControllerManagerReplicasUnavailable CE S7
  Part of the capi-controller-manager replicas is unavailable.

  Some replicas of capi-controller-manager have been unavailable for more than 30 minutes.

  The controller keeps working while at least one replica is available. Usually a master is down, cordoned, or its port is taken.

  To inspect the pods:

  d8 k -n d8-cloud-instance-manager get pods -l app=capi-controller-manager -o wide
  

• D8CloudDataDiscovererCloudRequestError CE S6
  Cloud data discoverer cannot get data from the cloud.

  Refer to the cloud data discoverer’s logs for details:

  d8 k -n NAMESPACE logs deploy/cloud-data-discoverer
  

• D8CloudDataDiscovererSaveError CE S6
  Cloud data discoverer cannot save data to a Kubernetes resource.

  Refer to the cloud data discoverer’s logs for details:

  d8 k -n NAMESPACE logs deploy/cloud-data-discoverer
  

• D8ClusterAutoscalerManagerPodIsNotReady CE S8
  The POD_NAME Pod is NOT Ready.

  To check the Pod’s status, run the following command:

  d8 k -n NAMESPACE logs POD_NAME
  

• D8ClusterAutoscalerPodIsNotRunning CE S8
  The cluster-autoscaler Pod is NOT Running.

  The POD_NAME Pod is STATUS.

  To check the Pod’s status, run the following command:

  d8 k -n NAMESPACE get pods POD_NAME -o json | jq .status
  

• D8ClusterAutoscalerPodIsRestartingTooOften CE S9
  Too many cluster-autoscaler restarts detected.

  The cluster-autoscaler has restarted VALUE times in the past hour.

  Frequent restarts may indicate a problem. The cluster-autoscaler is expected to run continuously without interruption.

  Check the logs for details:

  d8 k -n d8-cloud-instance-manager logs -f -l app=cluster-autoscaler -c cluster-autoscaler
  

• D8ClusterAutoscalerTargetAbsent CE S8
  Cluster-autoscaler target is missing in Prometheus.

  The cluster-autoscaler automatically scales nodes in the cluster. If it’s unavailable, it will be impossible to add new nodes when there’s not enough resources for scheduling Pods. It may also lead to unnecessary cloud costs due to unused but still provisioned cloud instances.

  To resolve the issue, follow these steps:

  1. Check availability and status of cluster-autoscaler Pods:

     d8 k -n d8-cloud-instance-manager get pods -l 'app in (cluster-autoscaler,cluster-autoscaler-mcm)'
     

  2. Verify that the cluster-autoscaler Deployment exists:

     d8 k -n d8-cloud-instance-manager get deploy -l 'app in (cluster-autoscaler,cluster-autoscaler-mcm)'
     

  3. Check the Deployment’s status:

     d8 k -n d8-cloud-instance-manager describe deploy deployment-name
     

• D8ClusterAutoscalerTargetDown CE S8

  Prometheus is unable to scrape cluster-autoscaler’s metrics.

• D8ClusterAutoscalerTooManyErrors CE S8
  Cluster-autoscaler is issuing too many errors.

  The cluster-autoscaler encountered multiple errors from the cloud provider when attempting to scale the cluster.

  Check the logs for details:

  d8 k -n d8-cloud-instance-manager logs -f -l app=cluster-autoscaler -c cluster-autoscaler
  

• D8FencingAgentDown CE S6
  The fencing agent does not work on node NODE_NAME.

  The node NODE_NAME is Ready, but Prometheus does not get the metrics of its fencing agent for 5 minutes.

  Without the agent, the node does not detect a loss of the control plane, and its neighbors do not see it.

  To check the pod of the agent:

  d8 k -n d8-cloud-instance-manager get pods -l app=fencing-agent --field-selector spec.nodeName=NODE_NAME -o wide
  d8 k -n d8-cloud-instance-manager describe pods -l app=fencing-agent --field-selector spec.nodeName=NODE_NAME
  

• D8FencingAgentInvalidKey CE S6
  The fencing agent on node NODE_NAME cannot read the memberlist key.

  The fencing agent on the node NODE_NAME cannot read the memberlist key from the Secret fencing-memberlist-key in the namespace d8-cloud-instance-manager, because the key has a wrong format. The agent does not connect to the other agents and does not arm the watchdog, so the node is not protected.

  The primary key must have exactly 32 bytes. A frequent error is a key written as a base64 string into stringData. To check the length of the key:

  d8 k -n d8-cloud-instance-manager get secret fencing-memberlist-key -o jsonpath='{.data.primary}' | base64 -d | wc -c
  

  The command must show 32. To write the key correctly, refer to the module FAQ.

• D8FencingAgentRestarted CE S7
  The fencing agent restarted on node NODE_NAME.

  The container fencing-agent restarted on the node NODE_NAME in the last 15 minutes.

  After a crash of the agent, the watchdog stays armed. At the start, the new agent process disarms the watchdog if the control plane is available. If the control plane is not available, the watchdog timeout expires and the kernel panics. The node then stays stopped, because kernel.panic is 0. Restart the node manually.

  If the agent crashes again and again, the watchdog of the node is not armed, and fencing does not stop the node if the node fails.

  To read the logs of the agent that crashed:

  d8 k -n d8-cloud-instance-manager logs -l app=fencing-agent --field-selector spec.nodeName=NODE_NAME -c fencing-agent --previous
  

• D8FencingEvictionNotProgressing CE S4
  The evacuation of node NODE_NAME is not complete after 60 seconds.

  The fencing controller put the taint node.kubernetes.io/out-of-service on the node NODE_NAME, but the evacuation is not complete after 60 seconds. The FencingFailedNodeState object stays in the phase Evicting and has the condition Progressing with the status False and the reason XXX. The controller examines the node again every 5 seconds. When the cause is gone, the evacuation completes, and the alert stops.

  The reasons:

  • VolumesAttached: the node still has VolumeAttachment objects. The volumes of its pods are not free for other nodes. Usually the CSI driver or the storage cannot detach the volumes.
  • PodsRemaining: the controller could not delete pods of the node, or another error of the API stops the evacuation. The message gives the pods and the errors.
  • TooManyUnhealthyMachines: the controller does not delete the node of a cloud machine, because the MachineHealthCheck does not replace it. More than 50 percent of the machines are unhealthy, or the cluster has no MachineHealthCheck that reports its machines. The message gives the numbers.

  To read the message of the condition:

  d8 k get fencingfailednodestate NODE_NAME -o yaml
  

  To find the VolumeAttachment objects of the node:

  d8 k get volumeattachments -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,ATTACHER:.spec.attacher,PV:.spec.source.persistentVolumeName,ATTACHED:.status.attached | awk 'NR==1 || $2=="NODE_NAME"'
  

  To read the errors of the detach:

  d8 k get volumeattachments -o json | jq '.items[] | select(.spec.nodeName=="NODE_NAME") | {name: .metadata.name, detachError: .status.detachError}'
  

• D8FencingKernelPanicMismatch CE S6
  The value of kernel.panic on node NODE_NAME is not the value that fencing needs.

  The value of kernel.panic on the node NODE_NAME stays wrong for 30 minutes. The fencing agent does not arm the watchdog, so the node is not protected.

  Fencing needs kernel.panic set to 0. After a fencing panic, the node must stay stopped and must not restart by itself.

  Bashible sets the value. Check that bashible works on the node and that no other tool changes kernel.panic:

  d8 k get node NODE_NAME -o jsonpath='{.status.conditions[?(@.type=="Fencing")]}' | jq
  sysctl kernel.panic
  

• D8FencingKeyRegenerated CE S9
  The memberlist key of fencing was generated again.

  Deckhouse generated a new memberlist key of fencing, because the Secret fencing-memberlist-key in the namespace d8-cloud-instance-manager was deleted. The agents use the new key. The old key is lost.

  Agents that did not get the new key cannot talk with the other agents. If you keep a copy of the key outside the cluster, replace it.

  The alert goes out 24 hours after the key was generated. It does not rise when Deckhouse creates the key for the first time or when you rotate the key yourself.

• D8FencingNodeAliveAfterAttestation CE S4
  Node NODE_NAME sends heartbeats after a witness confirmed its stop.

  A witness node confirmed that the node NODE_NAME stopped. After that, the fencing controller saw a heartbeat of the node. Thus the node can still work.

  The fencing controller does not put the taint node.kubernetes.io/out-of-service on the node while the node sends heartbeats. When the heartbeats stop, the node gets the taint after the silent delay: controlPlaneLostAfter + 0.2s + watchdog.timeout + watchdog.margin + 0.9s of the profile after the last sign of life of the node.

  Examine the node. To read the state of the incident:

  d8 k get fencingfailednodestate NODE_NAME -o yaml
  

• D8FencingNodeFailed CE S4
  Fencing found a failure of node NODE_NAME.

  The node NODE_NAME has a FencingFailedNodeState object. The fencing agents of other nodes found that this node failed. The watchdog of the node is armed.

  The fencing controller puts the taint node.kubernetes.io/out-of-service on the node when the node is stopped, for one of these reasons:

  • A witness node confirmed that the node stopped, and the watchdog timeout of the node passed after the confirmation.
  • The node is silent: it does not send its heartbeat through the API server. A node that loses the control plane stops itself. The controller waits for the time that the node needs to stop: the control plane loss threshold, the watchdog timeout and a margin after the last sign of life.

  After the taint, the pods of the node move to other nodes. The phase of the object shows the progress. For the cases when the controller does not put the taint, refer to the FAQ of the node-manager module.

  If the phase is FallbackAlive, the node works. Refer to the alert D8FencingNodeIsolatedFromPeers.

  To read the state of the incident:

  d8 k get fencingfailednodestate NODE_NAME
  d8 k describe fencingfailednodestate NODE_NAME
  

  The alert stops when the object is deleted.

• D8FencingNodeIPMismatch CE S6
  The fencing agent on node NODE_NAME cannot find the IP address of the node.

  The fencing agent on the node NODE_NAME cannot use the IP address of its Node object. The Node object has no IPv4 address, or the node network interface does not have this address. The agent waits for the address and does not arm the watchdog, so the node is not protected.

  To compare the addresses:

  d8 k get node NODE_NAME -o jsonpath='{.status.addresses}' | jq
  

• D8FencingNodeIsolatedFromPeers CE S5
  Node NODE_NAME works, but the fencing agents of other nodes cannot connect to it.

  The fencing agents of other nodes found that the node NODE_NAME failed. But the node connects to the control plane and sends its heartbeat. Thus the node works, and the network between this node and the other nodes is broken.

  Usually a firewall rule or an ACL stops the traffic of the fencing agents. The fencing controller does not put the taint node.kubernetes.io/out-of-service on a node that sends its heartbeat.

  To read the state of the incident:

  d8 k get fencingfailednodestate NODE_NAME -o yaml
  

  Make sure that the network lets the fencing agents connect to each other.

• D8FencingProfileIsUnavailable CE S5
  Fencing of node NODE_NAME is blocked, because the controller cannot use its SLA profile.

  The node NODE_NAME has a fencing incident, but the fencing controller cannot use the SLA profileXXX XXXXXX of the incident. The controller does not put the taint node.kubernetes.io/out-of-service on the node, and the pods of the node do not move to other nodes.

  The controller calculates the times of the incident from the profile:

  • the time from the confirmation of a witness to the taint: watchdog.timeout + watchdog.margin;
  • the time from the last sign of life of a silent node to the taint: controlPlaneLostAfter, watchdog.timeout, watchdog.margin and a fixed margin;
  • the time after the last heartbeat of the node: heartbeat.interval and heartbeat.apiTimeout.

  The message of the condition ConfigurationError of the incident gives the cause, for example the profile that is missing.

  The profile of the incident is the profile of the node group of the node. Only an armed node needs a profile. For a node that is not armed, the controller uses a fixed time of 20 seconds after the last heartbeat and reads no profile.

  The controller tries again with backoff. Thus the alert shows that the profile is not usable for some minutes.

  The module installs the built-in profiles. If a profile is missing, a user usually deleted or changed it. XXX To read the profile:

  d8 k get fencingslaprofile XXX -o yaml
  

  XXX The controller did not find the name of the profile. The message of the condition ConfigurationError gives the cause. XXX

  To read the cause and the events of the incident:

  d8 k get fencingfailednodestate NODE_NAME -o jsonpath='{.status.conditions}' | jq
  d8 k describe fencingfailednodestate NODE_NAME
  

  To read the logs of the controller:

  d8 k -n d8-cloud-instance-manager logs deploy/fencing-controller -c fencing-controller --all-pods=true
  

• D8FencingSystemNetworkNotFound CE S4
  The SystemNetwork XXX of fencing does not exist.

  The ModuleConfig node-manager names the SystemNetwork XXX in settings.fencing.systemNetworkFallback.systemNetworkName, but the cluster has no such SystemNetwork. The agents cannot use the second network.

  Deckhouse does not change the setting. Create the SystemNetwork, or set the name of another one, or delete the name to turn the second network off.

  To check the network:

  d8 k get systemnetwork XXX
  d8 k get moduleconfig node-manager -o jsonpath='{.spec.settings.fencing}'
  

  The alert goes out about 10 minutes after the network is back, because it looks at the last 10 minutes.

  While all node groups have the mode Disabled, the rules of fencing are not loaded, so the alert does not rise. The agents do not use the second network then. The alert rises when you turn fencing on.

• D8FencingWatchdogDeviceMissing CE S6
  The fencing agent on node NODE_NAME has no watchdog device.

  The fencing agent on the node NODE_NAME has no watchdog device for 30 minutes. The agent does not arm the watchdog, so the node is not protected.

  A new agent pod does not start without /dev/watchdog. Thus, usually this agent runs in a pod of the previous revision without the watchdog device, and the DaemonSet did not replace this pod yet. On the node, softdog and /dev/watchdog can be present.

  Do these checks:

  1. Check the rollout of the DaemonSet of the node group. Find the agent pods of the group in the states ContainerCreating (with the FailedMount event) or Terminating on other nodes. Such a pod stops the rollout. A pod does not start if the NodeGroupConfiguration step failed on its node. A pod stays Terminating if its node is NotReady.

     d8 k -n d8-cloud-instance-manager rollout status ds/fencing-agent-group
     d8 k -n d8-cloud-instance-manager get pods -l app=fencing-agent,node-group-name=group -o wide
     d8 k -n d8-cloud-instance-manager get events --field-selector reason=FailedMount
     

  2. On the node, check that bashible ran the NodeGroupConfiguration enable-watchdog-for-nodegroup-* without errors, that softdog is loaded and that /dev/watchdog is present:

     d8 k get node NODE_NAME -o jsonpath='{.status.conditions[?(@.type=="Fencing")]}' | jq
     journalctl -u bashible --since -1h
     lsmod | grep softdog
     ls -l /dev/watchdog
     

• D8FencingWatchdogTimeoutMismatch CE S6
  The watchdog timeout on node NODE_NAME differs from the timeout of the fencing profile.

  The watchdog of the node NODE_NAME reports a timeout that is not the timeout of the fencing profile of its node group. The fencing agent does not arm the watchdog, so the node is not protected.

  A watchdog driver can round the timeout or accept only some values. To read the reason:

  d8 k get node NODE_NAME -o jsonpath='{.status.conditions[?(@.type=="Fencing")]}' | jq
  

• D8KubeletCheckpointStateUnpinnedContainers CE S4
  Pod NAMESPACE/POD_NAME may run in a NUMA zone other than the one recorded by kubelet.

  Kubelet on Node NODE_NAME reset the memory manager checkpoint and recorded new NUMA zones for Pod NAMESPACE/POD_NAME. Zones are applied only when a container is created, so the running containers stay in their previous zones, which may differ from the recorded ones.

  As a result, the zone the Pod really uses may also be assigned to another Guaranteed Pod, causing both Pods to share its memory resources. The Pod’s CPUs and memory may also be assigned to different NUMA zones, which slows down its memory access.

  The Pod is not restarted automatically. The restart time depends on the workload. To restore resource isolation, do the following:

  • To recreate the Pod’s containers in newly assigned NUMA zones, delete the Pod:

    d8 k -n NAMESPACE delete pod POD_NAME
    

  • For a virtual machine, perform a live migration to another node instead, without stopping it.
  • If the node runs many such Pods, you can drain it instead of deleting them one by one.

  The alert resolves within a few minutes after the Pod is recreated. No kubelet restart is needed.

• D8MachineControllerManagerPodIsNotReady CE S8

  The POD_NAME Pod is NOT Ready.

• D8MachineControllerManagerPodIsNotRunning CE S8
  The machine-controller-manager Pod is NOT Running.

  The POD_NAME Pod is STATUS.

  To check the Pod’s status, run the following command:

  d8 k -n NAMESPACE get pods POD_NAME -o json | jq .status
  

• D8MachineControllerManagerPodIsRestartingTooOften CE S9
  Too many machine-controller-manager restarts detected.

  The machine-controller-manager has restarted VALUE times in the past hour.

  Frequent restarts may indicate a problem. The machine-controller-manager is expected to run continuously without interruption.

  Check the logs for details:

  d8 k -n d8-cloud-instance-manager logs -f -l app=machine-controller-manager -c controller
  

• D8MachineControllerManagerTargetAbsent CE S8
  Machine-controller-manager target is missing in Prometheus.

  Machine-controller-manager controls ephemeral nodes in the cluster. If it becomes unavailable, it will be impossible to create or delete nodes in the cluster.

  To resolve the issue, follow these steps:

  1. Check availability and status of machine-controller-manager Pods:

     d8 k -n d8-cloud-instance-manager get pods -l app=machine-controller-manager
     

  2. Verify availability of the machine-controller-manager Deployment:

     d8 k -n d8-cloud-instance-manager get deployment machine-controller-manager
     

  3. Check the Deployment’s status:

     d8 k -n d8-cloud-instance-manager describe deployment machine-controller-manager
     

• D8MachineControllerManagerTargetDown CE S8

  Prometheus is unable to scrape the machine-controller-manager’s metrics.

• D8NeedMigrateStateToOpenTofu CE S4
  Terraform state must be migrated to OpenTofu.

  This likely means the automatic migration to OpenTofu was canceled due to destructive changes detected in the cluster state.

  Verify the following:

  • Current Terraform version:

    d8 k -n d8-system exec -it deployments/terraform-state-exporter -c exporter -- terraform version
    

  • Version in the Terraform state:

    d8 k -n d8-system get secret d8-cluster-terraform-state -o json | jq -r '.data["cluster-tf-state.json"] | @base64d | fromjson | .terraform_version'
    

  • Check for destructive changes:

    d8 k exec -it deploy/terraform-state-exporter -n d8-system -- dhctl terraform check
    

  To resolve the issue and migrate to OpenTofu manually, follow these steps:

  1. Using the install container with the previous major Deckhouse release (for example, 1.69.X, while the cluster is now on 1.70.X), run:

     dhctl converge
     

     This should resolve all destructive changes in the cluster.

  2. After that, the terraform-auto-converger Pod should complete the migration automatically.

  If the automatic migration doesn’t happen, use the install container with the current Deckhouse release and run the following command to enforce migration of the Terraform state to OpenTofu:

  dhctl converge-migration
  

  If this alert appears in a cluster created by Deckhouse Commander, it means the cluster has been updated to a new Deckhouse version without approving state changes (for example, destructive changes).

  To resolve this, follow these steps:

  1. In Deckhouse Commander, click Delete manually to temporarily remove the cluster from control.

  2. In the cluster, remove the commander-agent module using the following command:

     d8 k delete mc commander-agent
     

  3. Using the install container with the previous major Deckhouse release (for example, 1.69.X, while the cluster is now on 1.70.X), run the following command to resolve all destructive changes in the cluster:

     dhctl converge
     

  4. In Deckhouse Commander, click Attach to reattach the cluster.
• D8NodeCgroupV2NotSupported CE S8
  Node NODE_NAME is using deprecated cgroup v1 – migration to v2 required.

  Node NODE_NAME in NodeGroup NODE_GROUP_NAME does not support cgroup v2. Starting from Kubernetes 1.35, cgroup v1 is deprecated. To migrate to cgroup v2, please upgrade the kernel to version 5.8 or newer. For more information, please refer to the documentation.

• D8NodeContainerdV2NotSupported CE S8
  Node NODE_NAME is using deprecated containerd v1 – migration to v2 required.

  Node NODE_NAME in NodeGroup NODE_GROUP_NAME does not support containerd v2. Starting from Kubernetes 1.38, containerd v1.y will no longer be supported. Please check the requirements for containerd v2 and schedule your migration plan.

  The exact reason is listed in the node.deckhouse.io/containerd-v2-err annotation on the node.

  If the reason is erofs and the node runs Ubuntu in AWS, the module is shipped in a separate linux-modules-extra package. DKP installs it automatically, but for some kernels, mostly outdated ones, the package is no longer published in the APT repository. In that case, the installation is skipped and a warning is written to the node logs. The problem has been confirmed for the 5.15.0-1028-aws, 6.8.0-1024-aws, and 6.8.0-1029-aws kernels. Switch the node group to a newer AMI or kernel version that includes erofs.

• D8NodeControllerReconcileErrorsHigh CE S8
  Node-controller reconcile errors are high for CONTROLLER_NAME.

  Controller CONTROLLER_NAME has a reconcile error rate above 0.1 errors per second for more than 10 minutes.

  Check node-controller logs:

  kubectl -n d8-cloud-instance-manager logs deploy/node-controller --all-pods=true
  

• D8NodeControllerReconcileP99High CE S8
  Node-controller reconcile p99 is high for CONTROLLER_NAME.

  The p99 reconcile duration for controller CONTROLLER_NAME has been above 5 seconds for more than 10 minutes.

  This may indicate slow reconciliation, Kubernetes API latency, or degraded controller performance.

  To inspect node-controller logs:

  kubectl -n d8-cloud-instance-manager logs deploy/node-controller --all-pods=true
  

• D8NodeControllerWorkqueueDepthHigh CE S8
  Node-controller workqueue NAME has a high backlog.

  The node-controller workqueue NAME has more than 50 queued objects for more than 15 minutes.

  This may indicate that the controller cannot process reconciliation events fast enough.

  To inspect node-controller logs:

  kubectl -n d8-cloud-instance-manager logs deploy/node-controller --all-pods=true
  

• D8NodeGroupFencingDeprecatedSettings CE S9
  NodeGroup NAME uses a deprecated fencing setting.

  NodeGroup NAME has a deprecated fencing setting in spec.fencing:

  • mode: Watchdog works as mode: Slow. To remove the alert, set mode: Slow, or choose another mode.
  • watchdog.timeout has no effect. The timeout comes from the fencing profile of the mode. To remove the alert, delete watchdog.timeout.

  To read the settings of the NodeGroup:

  d8 k get nodegroup NAME -o jsonpath='{.spec.fencing}'
  

• D8NodeGroupIsNotUpdating CE S8
  NodeGroup NODE_GROUP_NAME is not handling the update correctly.

  There is a new update available for Nodes in the NODE_GROUP_NAME NodeGroup. Although Nodes have detected the update, none of them have received approval to start the update process.

  Most likely, there is a problem with the update_approval hook of the node-manager module.

• D8NodeIsNotUpdating CE S7
  Node NODE_NAME cannot complete the update.

  Node NODE_NAME in NodeGroup NODE_GROUP_NAME detected a new update, requested and received approval, started the update, and encountered a step that could cause downtime. The update manager (the update_approval hook of the node-group module) granted downtime approval, but no success message was received, which indicates that the update has not completed.

  To investigate the details, view Bashible logs on the Node:

  journalctl -fu bashible
  

• D8NodeIsNotUpdating CE S8
  Node NODE_NAME cannot complete the update.

  Node NODE_NAME in NodeGroup NODE_GROUP_NAME has detected a new update, requested and received approval, but failed to complete the update.

  To investigate the details, view Bashible logs on the Node:

  journalctl -fu bashible
  

• D8NodeIsNotUpdating CE S9
  Node NODE_NAME is not updating.

  Node NODE_NAME in NodeGroup NODE_GROUP_NAME has a pending update but it’s neither receiving it nor attempting to.

  Most likely, Bashible is not handling the update correctly. It should annotate the Node with update.node.deckhouse.io/waiting-for-approval before the update can proceed.

  Options to investigate the details:

  • Check the expected configuration checksum for the NodeGroup:

    d8 k -n d8-cloud-instance-manager get secret configuration-checksums -o jsonpath={.data.NODE_GROUP_NAME} | base64 -d
    

  • Check the current configuration checksum on the Node:

    d8 k get node NODE_NAME -o jsonpath='{.metadata.annotations.node\.deckhouse\.io/configuration-checksum}'
    

  • View Bashible logs on the Node:

    journalctl -fu bashible
    

• D8NodeIsUnmanaged CE S9
  The NODE_NAME Node is not managed by the node-manager module.

  The NODE_NAME Node is not managed by the node-manager module.

  To resolve this issue, follow the instructions how to clean up the node and add it to the cluster.

• D8NodeOutOfServiceTainted CE S6
  Node NODE_NAME has the taint node.kubernetes.io/out-of-service for 30 minutes.

  The node NODE_NAME has the taint node.kubernetes.io/out-of-service for 30 minutes. Kubernetes does not run pods on the node.

  The fencing controller or a user puts the taint on a node that stopped. Start the node again or remove it from the cluster.

  To read the taints of the node:

  d8 k get node NODE_NAME -o jsonpath='{.spec.taints}' | jq
  

  If a user put the taint and the node works, remove the taint:

  d8 k taint node NODE_NAME node.kubernetes.io/out-of-service-
  

• D8NodeUpdateStuckWaitingForDisruptionApproval CE S8
  Node NODE_NAME cannot obtain disruption approval.

  Node NODE_NAME in NodeGroup NODE_GROUP_NAME detected a new update, requested and received initial approval, and started the update. However, it reached a stage that could cause downtime and was unable to obtain disruption approval. A disruption approval is normally issued automatically by the update_approval hook of the node-manager module.

  To resolve this issue, investigate why the approval could not be granted to proceed with the update.

• D8ProblematicNodeGroupConfiguration CE S8
  Node NODE_NAME cannot begin the update.

  There is a new update available for Nodes in the NODE_GROUP_NAME NodeGroup. However, Node NODE_NAME cannot begin the update.

  The Node is missing the node.deckhouse.io/configuration-checksum annotation, which may indicate that its bootstrap process did not complete correctly.

  Troubleshooting options:

  • Check the cloud-init log (/var/log/cloud-init-output.log) on the node.
  • Check the NodeGroupConfiguration resource associated with the NODE_GROUP_NAME NodeGroup for potential issues.
• NodeGroupHasStaticInternalNetworkCIDRsField CE S9
  NodeGroup NAME uses deprecated field spec.static.internalNetworkCIDRs.

  The spec.static.internalNetworkCIDRs field is deprecated and has been moved to the static cluster configuration.

  To resolve this issue, delete the field from NodeGroup NAME. This is safe, as the setting has already been migrated automatically.

• NodeGroupMasterTaintIsAbsent CE S4
  The master NodeGroup is missing the required control-plane taint.

  The master NodeGroup doesn’t have the node-role.kubernetes.io/control-plane: NoSchedule taint. This may indicate a misconfiguration where control-plane nodes can run non-control-plane Pods.

  To resolve the issue, add the following to the master NodeGroup spec:

    nodeTemplate:
      taints:
      - effect: NoSchedule
        key: node-role.kubernetes.io/control-plane
  

  Note that the taint key: node-role.kubernetes.io/master is deprecated and has no effect starting from Kubernetes 1.24.

• NodeGroupProviderTypeIsInvalid CE S4
  The spec.providerType field is set incorrectly in NodeGroup NAME.

  The spec.providerType field of NodeGroup NAME is set to XXX, which is not the cloud provider the nodes of the group run in.

  To resolve this issue, set the field to a correct value:

  • If the nodes of the group run in a cloud, set the field to the type of that cloud provider, as published in its registration secret:

    kubectl -n kube-system get secrets -l cloud-provider.deckhouse.io/registration -o json \
    | jq -r '.items[] | .data.type | @base64d'
    

  • If the cluster is static, or the NodeGroup has nodeType: Static, set the field to None or leave it empty:

    kubectl patch nodegroup NAME --type=merge -p '{"spec":{"providerType":"None"}}'
    

• NodeGroupReplicasUnavailable CE S7
  NodeGroup NODE_GROUP_NAME has no available instances.

  This probably means that machine-controller-manager is unable to create Machines using the cloud provider module.

  Possible causes:

  1. Cloud provider resource limits.
  2. No access to the cloud provider API.
  3. Misconfiguration of the cloud provider or instance class.
  4. Problems with bootstrapping the Machine.

  Recommended actions:

  1. Check the status of the NodeGroup:

     d8 k get ng NODE_GROUP_NAME -o yaml
     

     Look for errors in the .status.lastMachineFailures field.

  2. If no Instances stay in the Pending state for more than a couple of minutes, it likely means that Instances are being continuously created and deleted due to an error:

     d8 k get instances
     

  3. If logs don’t show errors, and an Instance continues to be Pending, check its bootstrap status:

     d8 k get instances INSTANCE_NAME -o yaml | grep 'bootstrapStatus' -B0 -A2
     

  4. If the output looks like the example below, use the logsEndpoint (or the command in description) to examine bootstrap logs:

     bootstrapStatus:
       description: Use 'curl -N http://192.168.199.158:8000' to get bootstrap logs.
       logsEndpoint: http://192.168.199.158:8000
     

  5. If there’s no bootstrap log endpoint, cloudInit may not be working correctly. This could indicate a misconfigured instance class in the cloud provider.

• NodeGroupReplicasUnavailable CE S8
  Too many unavailable instances in the NODE_GROUP_NAME NodeGroup.

  The number of simultaneously unavailable instances in the NODE_GROUP_NAME NodeGroup exceeds the allowed threshold. This may indicate that the autoscaler has provisioned too many nodes at once. Check the state of the Machine in the cluster. This probably means that machine-controller-manager is unable to create Machines using the cloud provider module.

  Possible causes:

  1. Cloud provider resource limits.
  2. No access to the cloud provider API.
  3. Misconfiguration of the cloud provider or instance class.
  4. Problems with bootstrapping the Machine.

  Recommended actions:

  1. Check the status of the NodeGroup:

     d8 k get ng NODE_GROUP_NAME -o yaml
     

     Look for errors in the .status.lastMachineFailures field.

  2. If no Instances stay in the Pending state for more than a couple of minutes, it likely means that Instances are being continuously created and deleted due to an error:

     d8 k get instances
     

  3. If logs don’t show errors, and an Instance continues to be Pending, check its bootstrap status:

     d8 k get instances INSTANCE_NAME -o yaml | grep 'bootstrapStatus' -B0 -A2
     

  4. If the output looks like the example below, use the logsEndpoint (or the command in description) to examine bootstrap logs:

     bootstrapStatus:
       description: Use 'curl -N http://192.168.199.158:8000' to get bootstrap logs.
       logsEndpoint: http://192.168.199.158:8000
     

  5. If there’s no bootstrap log endpoint, cloudInit may not be working correctly. This could indicate a misconfigured instance class in the cloud provider.

• NodeGroupReplicasUnavailable CE S8
  NodeGroup NODE_GROUP_NAME has unavailable instances.

  There are VALUE unavailable instances in the NodeGroup NODE_GROUP_NAME. This probably means that machine-controller-manager is unable to create Machines using the cloud provider module.

  Possible causes:

  1. Cloud provider resource limits.
  2. No access to the cloud provider API.
  3. Misconfiguration of the cloud provider or instance class.
  4. Problems with bootstrapping the Machine.

  Recommended actions:

  1. Check the status of the NodeGroup:

     d8 k get ng NODE_GROUP_NAME -o yaml
     

     Look for errors in the .status.lastMachineFailures field.

  2. If no Instances stay in the Pending state for more than a couple of minutes, it likely means that Instances are being continuously created and deleted due to an error:

     d8 k get instances
     

  3. If logs don’t show errors, and an Instance continues to be Pending, check its bootstrap status:

     d8 k get instances INSTANCE_NAME -o yaml | grep 'bootstrapStatus' -B0 -A2
     

  4. If the output looks like the example below, use the logsEndpoint (or the command in description) to examine bootstrap logs:

     bootstrapStatus:
       description: Use 'curl -N http://192.168.199.158:8000' to get bootstrap logs.
       logsEndpoint: http://192.168.199.158:8000
     

  5. If there’s no bootstrap log endpoint, cloudInit may not be working correctly. This could indicate a misconfigured instance class in the cloud provider.

• NodeGroupUsesDeprecatedInCoreGPU CE S9
  NodeGroup NAME uses the deprecated built-in GPU support.

  NodeGroup NAME has spec.gpu configured while the gpu module is disabled. As a result, GPU nodes in the NodeGroup use the built-in GPU support, which is deprecated and will be removed in a future DKP release.

  To use the supported GPU implementation, enable the gpu module.

• NodeRequiresDisruptionApprovalForUpdate CE S8
  Node NODE_NAME requires disruption approval to proceed with the update.

  Node NODE_NAME in NodeGroup NODE_GROUP_NAME has detected a new update, received initial approval, and started the update. However, it encountered a stage that may cause downtime and requires manual disruption approval, because the NodeGroup is configured with disruptions.approvalMode: Manual.

  To resolve the issue, ensure the Node is ready for unsafe updates (drained) and grant disruption approval by annotating the Node with update.node.deckhouse.io/disruption-approved=.

  Caution:

  • Nodes in manual mode aren’t drained automatically.
  • Do not drain master nodes.

  1. To drain the Node and grant the update approval, run the following command:

     d8 k drain NODE_NAME --delete-emptydir-data --ignore-daemonsets --force=true &&
       d8 k annotate node NODE_NAME update.node.deckhouse.io/disruption-approved=
     

  2. Uncordon the Node once the update is complete and the annotation update.node.deckhouse.io/approved is removed:

     while d8 k get node NODE_NAME -o json | jq -e '.metadata.annotations | has("update.node.deckhouse.io/approved")' > /dev/null; do sleep 1; done
     d8 k uncordon NODE_NAME
     

  If the NodeGroup has multiple Nodes, repeat this process for each one, since only one Node is updated at a time. Consider temporarily switching to automatic disruption approval (disruptions.approvalMode: Automatic).

• NodeStuckInDraining CE S6
  Node NODE_NAME keeps failing to drain.

  Node NODE_NAME in NodeGroup NODE_GROUP_NAME is stuck in the draining process.

  The Node has the error from the last attempt in the update.node.deckhouse.io/drain-failed annotation, truncated to 1 KB. To read it, run:

  d8 k get node NODE_NAME -o json | jq -r '.metadata.annotations["update.node.deckhouse.io/drain-failed"]'
  

  The full error text is in the DrainFailed events on the Node:

  d8 k -n default get event --field-selector involvedObject.name=NODE_NAME,reason=DrainFailed --sort-by='.metadata.creationTimestamp'
  

  The drain is retried until it succeeds, so the alert clears on its own once the cause is gone. Until then the Node holds the update slot of its NodeGroup, and no other Node in that group is updated.

  The usual causes are a PodDisruptionBudget that never allows an eviction and a pod that never terminates. Fix the cause named in the error above.

• NodeStuckInDrainingForDisruptionDuringUpdate CE S6
  Node NODE_NAME is stuck in draining.

  Node NODE_NAME in NodeGroup NODE_GROUP_NAME has detected a new update, requested and received approval, started the update, and reached a step that could cause downtime. It is currently stuck in the draining process while waiting for automatic disruption approval.

  To get more details, run the following:

  d8 k -n default get event --field-selector involvedObject.name=NODE_NAME,reason=ScaleDown --sort-by='.metadata.creationTimestamp'
  

• StandbyHolderDeploymentReplicasUnavailable CE S5
  No available replicas in Deployment standby-holder-NODE_GROUP_NAME.

  Deckhouse has detected that there are no available replicas in the standby-holder-NODE_GROUP_NAME Deployment in the d8-cloud-instance-manager namespace.

  Check the Deployment and associated Pods for issues.

• UnmetCloudConditions CE S6
  Deckhouse update is unavailable due to unmet cloud provider conditions.

  Deckhouse has detected that some cloud provider–specific conditions have not been met. Until these issues are resolved, updating to the new Deckhouse release is not possible.

  Troubleshooting details:

  • Name: NAME
  • Message: MESSAGE_CONTENTS

Module okmeter

• D8OkmeterAgentPodIsNotReady CE S6

  Okmeter agent is not Ready.

Module openvpn

• OpenVPNClientCertificateExpired CE S4
  OpenVPN client certificate expired for XXX

  The OpenVPN client certificate for XXX has expired.

  Check the certificate information and make sure it actually expires:

  d8 k -n d8-openvpn get secrets -l name="XXX"  -o jsonpath='{.items[0].data.tls\.crt}' | base64 -d | openssl x509 -text -noout
  

  Renew or delete the expired certificate. Check the documentation for details.

• OpenVPNClientCertificateExpired CE S5
  OpenVPN client certificate expires for XXX in less than 7 days.

  The OpenVPN client certificate for XXX is set to expire in VALUE days.

  Renewal of certificate required. Check the documentation for details.

  View certificate details to display the exact expiration date:

  d8 k -n d8-openvpn get secrets -l name=XXX  -o jsonpath='{.items[0].data.tls\.crt}' | base64 -d | openssl x509 -text -noout
  

• OpenVPNClientCertificateExpired CE S6
  OpenVPN client certificate expires for XXX in less than 30 days.

  The OpenVPN client certificate for XXX is set to expire in VALUE days.

  Renewal of certificate required. Check the documentation for details.

  View certificate details to display the exact expiration date:

  d8 k -n d8-openvpn get secrets -l name=XXX  -o jsonpath='{.items[0].data.tls\.crt}' | base64 -d | openssl x509 -text -noout
  

• OpenVPNServerCACertificateExpired CE S4
  OpenVPN CA certificate has expired.

  OpenVPN CA certificate renews automatically 1 days before expiration. Automatic rotation did not work. Check the module status or perform manual rotation.

  View certificate details to display the exact expiration date:

  d8 k -n d8-openvpn get secrets openvpn-pki-ca  -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout
  

  The hook should rotate CA and server certificates before expiry.

• OpenVPNServerCACertificateExpiresTomorrow CE S4
  OpenVPN CA certificate will expire tomorrow

  OpenVPN CA certificate renews automatically 1 days before expiration. Automatic rotation did not work. Check the module status or perform manual rotation.

  View certificate details to display the exact expiration date:

  d8 k -n d8-openvpn get secrets openvpn-pki-ca  -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout
  

  The hook should rotate CA and server certificates before expiry.

• OpenVPNServerCACertificateExpiringInAWeek CE S5
  OpenVPN CA certificate expires in less than 7 days.

  The OpenVPN CA certificate is set to expire in VALUE days.

  Renew the CA certificate if necessary. After renewing the certificate, you will need to reissue all client and server certificates. Check the documentation for details.

  View certificate details to display the exact expiration date:

  d8 k -n d8-openvpn get secrets -l name=server  -o jsonpath='{.items[0].data.tls\.crt}' | base64 -d | openssl x509 -text -noout
  

• OpenVPNServerCACertificateExpiringSoon CE S5
  OpenVPN CA certificate expires in less than 30 days.

  The OpenVPN CA certificate is set to expire in VALUE days.

  Renew the CA certificate if necessary. After renewing the certificate, you will need to reissue all client and server certificates. Check the documentation for details.

  View certificate details to display the exact expiration date:

  d8 k -n d8-openvpn get secrets -l name=server  -o jsonpath='{.items[0].data.tls\.crt}' | base64 -d | openssl x509 -text -noout
  

• OpenVPNServerCertificateExpired CE S4
  OpenVPN server certificate has expired.

  The OpenVPN server certificate has expired.

  OpenVPN server certificate renews automatically 1 days before expiration. Automatic rotation did not work. Check the module status or perform manual rotation.

  Check the certificate information and make sure it actually expires.

  d8 k -n d8-openvpn get secrets -l name=server  -o jsonpath='{.items[0].data.tls\.crt}' | base64 -d | openssl x509 -text -noout
  

• OpenVPNServerCertificateExpiresTomorrow CE S4
  OpenVPN server certificate will expire tomorrow.

  The OpenVPN server certificate is set to expire in less than 1 day.

  OpenVPN server certificate renews automatically 1 days before expiration. Automatic rotation did not work. Check the module status or perform manual rotation.

  Check the certificate information and make sure it actually expires:

  d8 k -n d8-openvpn get secrets -l name=server  -o jsonpath='{.items[0].data.tls\.crt}' | base64 -d | openssl x509 -text -noout
  

Module registry

• D8RegistryAgentNotReady CE S4
  The registry agent on node NODE_NAME is not ready.

  The registry agent on node NODE_NAME is not answering, or is not running at all. Every pull on the node goes through it, and without it containerd pulls straight from the registry an image names. Images of the platform itself, and of registries that need the cluster’s credentials or certificate authority, then fail on that node; other images still pull, bypassing the in-cluster cache and any RegistryUpstream.

  The kubelet restarts an agent that stops answering for a minute, so a state that lasts this long is one a restart does not fix: an agent that fails on start, or one that is not on the node at all.

  What the pod says:

  d8 k -n kube-system get pod registry-agent-NODE_NAME -o wide
  d8 k -n kube-system describe pod registry-agent-NODE_NAME
  d8 k -n kube-system logs registry-agent-NODE_NAME --previous --tail=50
  

  An agent that exits on start names in its log what it could not use, usually its certificate material under /etc/kubernetes/registry-agent/pki or containerd’s drop-in directory.

  No such pod means either that the kubelet has no manifest for it — on a node bashible configures, that is /etc/kubernetes/manifests/registry-agent.yaml — or that the API server refused the mirror pod, in which case the agent may well be running. The kubelet’s log on the node says which:

  journalctl -u kubelet --since "1 hour ago" | grep registry-agent
  

  A node of a NodeGroup with systemType: DeckhouseEngine has no SSH: start a debug pod on it with d8 k debug node/NODE_NAME -it --image=ubuntu and run chroot /host first.

  Fifteen minutes, so that a node taking over the agent after the module starts managing it, or an update that replaces the agent on every node, does not raise it.

• D8RegistryAgentRestartingTooOften CE S5
  The registry agent on node NODE_NAME keeps restarting.

  The registry agent on node NODE_NAME has restarted five times or more in the last hour. Each restart drops every pull in flight on the node, and until the agent is back containerd pulls without it, where images of the platform cannot be pulled.

  Either the agent exits on its own, or the kubelet kills it because it stopped answering its liveness probe for a minute. The last state of the container says which:

  d8 k -n kube-system get pod registry-agent-NODE_NAME -o jsonpath='{.status.containerStatuses[0].lastState}' | jq
  d8 k -n kube-system logs registry-agent-NODE_NAME --previous --tail=100
  d8 k -n kube-system get events --field-selector involvedObject.name=registry-agent-NODE_NAME
  

  Liveness probe failed in the events is an agent that hung: the log before the restart shows what it was doing. An exit code with an error in the log is an agent that gave up: the log names what it could not use.

• D8RegistryAirGapTransitionHeld CE S5
  The registry upstream cannot be removed yet.

  The configuration has no upstream, so the cluster is meant to be air-gapped, and the module is still holding the previous upstream: either the cache is not complete enough to stand alone, or the cache leader’s report was not written by the syncer running on its node now, or workloads in the platform namespaces (kube-system and d8-*) still reference the upstream and would have nowhere to pull from without it. Workloads in other namespaces are not counted and do not hold the transition.

  This is the safe outcome and the cluster is pulling images normally. It is also the one transition in this module that could cut every node off from images, which is why it waits rather than proceeds — and why a wait this long is worth looking at: nothing about it resolves on its own if the cache has stopped filling, or if a rollout onto the in-cluster address is stuck.

  Check the cache and the current count of references to the upstream (upstreamReferences):

  d8 k get registrystorage registry -o jsonpath='{.status}' | jq '{phase,fill,safeToDropUpstream,upstreamReferences,leader}'
  d8 k get registryconfig registry -o jsonpath='{.status.effectiveUpstream}' | jq
  

  The registry-controller log names up to five of the referencing objects each time the count changes. To see the last such line:

  d8 k -n d8-system logs -l app=registry-controller -c registry-controller --prefix --tail=-1 | grep 'still names the upstream' | tail -1
  

  If the leader is full and upstreamReferences is 0, the upstream is held on the leader’s report: its syncerStartedAt is missing or earlier than the start of the syncer container in the leader’s storage pod. The syncer stamps its report once it has read its store, normally within two minutes of starting. The hold does not end while the leader’s storage pod is not Ready, or while that pod runs a syncer from before syncerStartedAt existed (during a module update), which stamps nothing. To see the leader’s report and the controller’s reason:

  d8 k get registrystorage registry -o json | jq '.status as $s | $s.replicas[] | select(.node == $s.leader) | {node,full,error,syncerStartedAt}'
  d8 k -n d8-system logs -l app=registry-controller -c registry-controller --prefix --tail=-1 | grep 'the upstream is held' | tail -1
  

  Deleting the leader’s storage pod is safe while the upstream is held: the replacement syncer reads its store and stamps its report, normally within two minutes.

• D8RegistryConfigInvalid CE S4
  The registry configuration was rejected.

  The registry module cannot use its configuration, so the cluster keeps whatever arrangement it already had. Nothing has broken yet; nothing new will take effect either.

  What was wrong is in the resource:

  d8 k get registryconfig registry -o jsonpath='{.status.conditions}' | jq
  

• D8RegistryControllerDown CE S5
  The registry controller is down or cannot be scraped.

  No replica of registry-controller in d8-system is both ready and answering Prometheus. Either every replica is down or not ready, or they run and their metrics cannot be read — through kube-rbac-proxy, which needs a TokenReview to let Prometheus in.

  Pulls are not affected: every node keeps pulling the way its agent was last told. What stops is everything that changes that. A change to the registry ModuleConfig, a new RegistryUpstream, a leader moving to another master, an air-gap transition — none of it reaches the nodes until the controller is back.

  Beyond the registry, no namespace in the cluster can finish deleting: the controller also serves the registry.deckhouse.io API (the RegistryImage listing), and while nobody answers for it, discovery of that group fails and namespaces stay Terminating with the condition NamespaceDeletionDiscoveryFailure. They are deleted once a replica answers again.

  And while this is firing, most other D8Registry* alerts are blind: they read what the controller exports, so they stay silent whatever happens in the meantime. The exceptions are D8RegistryStorageLeaderAbsent, which reads the cache replicas themselves, and the migration and drain alerts, which Deckhouse feeds; those keep working.

  d8 k -n d8-system get deploy registry-controller
  d8 k -n d8-system get pods -l app=registry-controller -o wide
  d8 k -n d8-system logs deploy/registry-controller -c registry-controller --all-pods=true --tail=100
  d8 k -n d8-monitoring get podmonitor registry-controller
  

  If the pods are ready, the problem is in the scrape: look at the kube-rbac-proxy container’s log for authentication errors.

• D8RegistryDrainStuck CE S6
  The registry module cannot finish leaving the pull path.

  The registry ModuleConfig asks for Unmanaged, and the module is still running its components and still serving registry.d8-system.svc:5001. That is deliberate: workloads in the cluster still name that address, and taking it away before they move would leave them unable to pull. It has been waiting for half an hour, which is longer than it should take.

  Nothing is degraded while this alert is firing. The cluster pulls, the module serves, and the only thing not happening is the withdrawal.

  What to look at:

  1. What still names the address. The module logs the first few on every pass, and the count is a metric:

     d8 k -n d8-system logs deploy/deckhouse -c deckhouse | grep "still serving the in-cluster registry"
     

  2. Then the workloads themselves, which is where the answer usually is:

     d8 k get pods,deployments,daemonsets,statefulsets -A -o json \
       | jq -r '.items[] | select([.spec.template.spec.containers[]?.image, .spec.containers[]?.image] | any(startswith("registry.d8-system.svc:5001"))) | "\(.kind) \(.metadata.namespace)/\(.metadata.name)"'
     

  A third cause is a cluster with nowhere to go, and it is the one to check first on a cluster that was air-gapped or installed with no upstream. Leaving the pull path means pointing the cluster back at the registry the module was fetching from, and a module serving purely from its own store has no such registry to name. It says so in the log:

  d8 k -n d8-system logs deploy/deckhouse -c deckhouse | grep "no upstream to point the cluster back at"
  

  Give the cluster a registry to pull from, or ask for Managed again. Nothing is degraded while it waits, and the withdrawal will not complete until one of the two happens.

  Otherwise two causes account for it. Either a release is not being re-rendered — check that the operator queue is not stuck, since the references move only as each module’s release is rendered again:

  d8 k -n d8-system exec deploy/deckhouse -c deckhouse -- deckhouse-controller queue main
  

  Or something outside the platform’s own renders names the address: a workload of yours that hardcoded it, or a manually edited manifest. Those will never move on their own, and until they are changed to pull from the registry the cluster was installed with, the module will keep serving them — which is the intended behaviour, not a failure.

  One shape of this is worth naming, because it was hit the first time this ran on a live cluster: a rolling update stuck for reasons of its own. The new pod is already on the upstream registry and never becomes ready, so the superseded ReplicaSet keeps its old pod, and that pod names the in-cluster address. A pod that has already pulled is not waited on, so this alone does not hold the withdrawal — but a pod stuck BEFORE its containers start does, and rightly, since it is exactly what would fail. Fix the rollout, or delete the pod that cannot start:

  d8 k get pods -A --field-selector status.phase=Pending
  

  A drain can also be abandoned by asking for Managed again: the module then keeps the pull path it never finished giving up, and this alert goes away.

• D8RegistryMigrationPending CE S8
  The registry module is held on its v1 implementation.

  This cluster still pulls images the way the v1 implementation of the registry module arranged it, and the handover to the current implementation has not happened. This alert does not resolve on its own while the reason stays, and the following DP releases stay blocked by the registryImplementation requirement.

  Reason: XXX

  The handover waits because both implementations configure the same thing on every node: which registry the container runtime asks, and with which credentials. Running both would not merge those two answers, it would race them. So the current implementation takes over only from a state it can take over from.

  What to do depends on the reason:

  • The reason names the Direct mode. This release can take over on its own once it knows where images come from: set mode: Managed in the registry ModuleConfig together with primary.upstream. Take the values from the registry.direct section of the deckhouse ModuleConfig, as the module’s FAQ shows for the Direct mode: imagesRepo splits into host and path, the credentials stay the same, plus ca if the registry requires one. The handover happens on the module’s next reconciliation and this alert clears.

  • The reason names the Proxy or Local mode, says the v1 implementation is transitioning, has not recorded a mode, or that its state or node configuration cannot be read. This state is not supported. A cluster gets here only through an update that bypassed the registryImplementation release requirement: a forced release, an edited image, or a jump from a release older than DP 1.77.1. DP 1.78 has no code that can manage or migrate this state.

    DP 1.78 does not deploy the v1 implementation, and the update removed the components that managed its node services on the masters. Depending on the timing of the update, those services were stopped, or keep running with nothing to manage them or renew their certificates. The nodes keep the registry configuration the v1 implementation gave them. Images already on a node keep working; pulling any other image may fail.

    Do not change the registry or deckhouse ModuleConfigs and do not delete the registry-* secrets in d8-system: those secrets keep the handover closed, so that the current implementation is not deployed next to what the v1 implementation left. Contact Deckhouse Platform technical support.

  To see the mode the v1 implementation recorded:

  d8 k -n d8-system get secret registry-state -o jsonpath='{.data.state}' | base64 -d | head
  

  If that secret is gone, the node configuration the v1 implementation left still shows the mode. This prints the mode and the proxy endpoints only, without credentials:

  d8 k -n d8-system get secret registry-bashible-config -o jsonpath='{.data.config}' | base64 -d \
    | awk '/^(agent|mode):/ {print; next} /^proxyEndpoints:/ {p=1; print; next} p && /^ *- / {print; next} {p=0}'
  

• D8RegistryNodeForeignRegistryConfig CE S5
  Node NODE_NAME carries container runtime registry configuration the registry module did not write.

  Node NODE_NAME has a file under /etc/containerd/conf.d/ (containerd v1) or /etc/containerd/conf2.d/ (containerd v2) that configures a registry, and the registry module manages this cluster’s registry configuration. Two writers of that configuration is not a state the platform supports, so bashible refuses to merge such a file.

  One exception for now: on containerd v1 with the module in the Unmanaged mode, a conf.d file is still merged. The refusal described below starts there once the module is switched to mode: Managed, so fix the file before that switch.

  Right now nothing is broken, and that is the danger. The refusal happens inside the bashible step that builds the container runtime configuration, and that step only runs when something makes it run. Until then the file has no effect at all: bashible compares the node’s configuration checksum, finds it unchanged and reports “Configuration is in sync, nothing to do”.

  The step does run on its own, though: a node re-applies its whole configuration periodically (roughly every four hours), and also on any registry change, platform upgrade or reboot. At that point the step fails with

  Failed to merge /etc/containerd/conf.d/file: it contains custom registry fields,
  configure them in /etc/containerd/registry.d instead
  

  and retries, so the node stops applying configuration entirely — hours after the change that caused it, and with nothing else to connect the two.

  What to do, before that happens:

  1. Find the file on the node:

     ls /etc/containerd/conf.d/ /etc/containerd/conf2.d/ 2>/dev/null
     grep -l registry /etc/containerd/conf.d/*.toml /etc/containerd/conf2.d/*.toml 2>/dev/null
     

  2. Move the registry part of it to a host directory, which is where this platform configures registries and where the module’s node agent leaves other people’s entries alone:

     /etc/containerd/registry.d/registry host/hosts.toml
     

  3. Remove the registry section from the conf.d file (anything non-registry in it can stay), and remove the NodeGroupConfiguration that writes it if that is where it comes from.

  Doing it across a fleet with a NodeGroupConfiguration has one catch worth knowing in advance: give it a weight below 32. The step that refuses the file is 032_configure_containerd, and a node stops at the step that fails — so a repair placed at a later weight never runs on the nodes that need it most. Measured: a cleanup configuration at weight 10 reached both nodes and cleared the label in 90 seconds.

  A node already stuck in the retry loop recovers on its own once the file is gone: measured, the configuration checksum returned in 75 seconds.

• D8RegistryNodeNotConverged CE S5
  Some nodes have not applied their registry configuration.

  The agent on one or more nodes has not applied the layout it was given. Those nodes keep pulling images the way they were configured before, so this is not an outage — but a change made to the registry configuration has not reached them, and the next one will not either.

  Which nodes, and what their agents say:

  d8 k get registrynodes -o custom-columns=\
  NODE:.metadata.name,GEN:.metadata.generation,APPLIED:.status.observedGeneration,OK:.status.reconciled
  d8 k -n kube-system logs -l component=registry-agent --tail=50
  

  Twenty minutes, because the agent re-reads its layout on a timer and a node that is merely slow or briefly unreachable resolves this on its own.

• D8RegistryNodeRunningFromDisk CE S6
  Some nodes are routing pulls from their on-disk copy of the configuration.

  The agent on one or more nodes read its layout from the API server and could not use it, so it is routing pulls from the copy it keeps on disk. Most often the credentials the layout references cannot be read; a read that failed while the status write after it went through looks the same. That is the copy working as designed — those nodes pull images normally, which is exactly why this is worth saying out loud: from every other angle it looks like success, and their configuration can drift arbitrarily far behind the cluster’s.

  An hour, because a node that falls back for a pass or two is not news; a node that has not come off the copy since is.

  d8 k get registrynodes -o json | jq -r '
    .items[] | select(.status.conditions[]? | select(.type=="Reconciled" and .reason=="AppliedFromCache"))
    | .metadata.name'
  d8 k -n kube-system logs -l component=registry-agent --tail=200 | grep 'cannot resolve the credentials'
  

  What this alert cannot see is an agent that cannot write its status at all, since that write is how it reports the fallback. A node cut off from the API server is the main case: the agent reaches the API server the way the kubelet does, so the node shows as NotReady and is reported as such. An agent whose own access is refused is the other, and while its node stays Ready nothing reports it.

• D8RegistryStaleCacheData CE S8
  A node still holds registry cache data that nothing uses.

  Node NODE_NAME holds data of the in-cluster registry cache while no cache is configured. Nothing reads it, and nothing will unless the cache is turned back on.

  It is kept on purpose. Turning the cache back on refills from what is already there rather than from scratch, and over a slow link that difference is hours — so the module does not delete it, because that would make the decision irreversible in the one direction that hurts. What it will not do is keep it quietly, which is what this alert is.

  Nothing is wrong and nothing needs doing urgently. To reclaim the space, remove the contents of the directory the store kept its blobs in: /var/lib/deckhouse/registry on a node of a NodeGroup with systemType: DeckhouseEngine, /opt/deckhouse/registry on other nodes. Keep the directory itself: the agent has it mounted, and once it is removed the agent no longer sees the node’s directory, so data a cache leaves there later goes unreported until the agent restarts. With the cache off there is no RegistryStorage left to ask, so the command below looks at both.

  Open a root shell on the node over SSH. A node of a NodeGroup with systemType: DeckhouseEngine has no SSH: start a debug pod on it and switch to the node’s root filesystem:

  d8 k debug node/NODE_NAME -it --image=ubuntu
  chroot /host
  

  Then remove the data, keeping the directories themselves:

  for dir in /opt/deckhouse/registry /var/lib/deckhouse/registry; do
    [ -d "$dir" ] && du -sh "$dir" && find "$dir" -mindepth 1 -delete
  done
  

  Twenty-four hours, so that turning the cache off and on again in one working session never raises it.

• D8RegistryStorageIncomplete CE S6
  Some registry cache replicas do not hold the whole image set.

  One or more replicas of the in-cluster cache are still filling, or have stopped short. While an upstream is configured this costs nothing at pull time — a node falls back to it — but it does mean the cluster could not survive losing that upstream, and it is what holds an air-gap transition back.

  Two hours, because filling a cache over a slow link legitimately takes that long.

  A replica that reports an error counts as not holding the set, whatever it holds: the error says why — a store that is not on the disk meant for it, for one. A replica whose store the registry cannot write to is left out: it is neither filling nor going to, and D8RegistryStoreWritesRefused reports it with what to do.

  d8 k get registrystorage registry -o json | jq '.status.replicas[] | {node, full, error}'
  

• D8RegistryStorageLeaderAbsent CE S4
  The registry cache has no leader.

  No ready replica of the in-cluster registry cache (registry-storage in d8-system) reports holding the storage lease.

  The leader is where the nodes pull from, and the only replica that writes. Without it nothing fills the cache, the followers have nobody to replicate from, and d8 mirror push has nowhere to go. The agents still route pulls to the replica named as leader in their layout and wait for it before anything else: with an upstream configured a pull is slower and comes from the upstream; in an air-gapped cluster an image a node does not already hold cannot be pulled at all.

  Which of three things it is decides what to do, so look first:

  d8 k -n d8-system get pods -l app=registry-storage -o wide
  d8 k -n d8-system get lease registry-storage -o jsonpath='{.spec.holderIdentity} {.spec.renewTime}{"\n"}'
  d8 k get registrystorage registry -o jsonpath='{.status.leader}{"\n"}'
  d8 k get registrystorage registry -o json | jq '.status.replicas[] | {node, role, full, error}'
  

  • The Lease is held and its renewTime keeps moving: the leader is alive, but its report is not counted. Look at the replica on the node named in holderIdentity:

    d8 k -n d8-system get pods -l app=registry-storage -o wide --field-selector spec.nodeName=holderIdentity
    d8 k -n d8-system describe pod the pod from the line above
    

    If it is not ready, Prometheus does not scrape it, since it scrapes only ready pods. syncer has no readiness probe; it is usually the distribution or auth container that is not ready, and a leader with either of them not ready serves pulls badly or not at all. Find out which one fails and why. If the pod is ready, its scrape fails while the others’ do not: look at its kube-rbac-proxy container’s log for errors.

  • The Lease’s renewTime has stopped while other replicas are ready: nobody leads. First rule out a master that has gone down: the Lease’s holderIdentity above is the node the last leader ran on.

    d8 k get nodes -l node-role.kubernetes.io/control-plane
    

    If a master is down and is not coming back, delete its Node object: the replica that ran on it then no longer counts in the election. Otherwise, or if the alert keeps firing after that, look at the error each replica reports in the list above and at the syncer logs of the replicas that are up:

    d8 k -n d8-system logs registry-storage-0 -c syncer --tail=100
    

  • No replica is ready, or none can be scraped. Then this is not about the election: the pods are down or not ready (where the extended-monitoring module is enabled, KubernetesStatefulSetReplicasUnavailable says so as well), or Prometheus cannot read them through kube-rbac-proxy, and the leader may well be alive. The Lease above tells it: a renewTime that keeps moving means somebody holds it.

• D8RegistryStorageNotReclaimed CE S6
  The registry cache has not reclaimed its disk for a week.

  No replica of the in-cluster cache has finished a garbage collection in the last week. Nothing is broken yet: the cache serves images normally, and it will keep doing so until the disk fills.

  It is worth attention because the collection is the only thing that ever removes anything. Every release adds a slice of the repository, so a store nobody reclaims grows in one direction — and a full store cannot be written to, which in an air-gapped cluster means it cannot be updated.

  Where to look:

  d8 k get registryoperation -o wide
  d8 k get registrystorage registry -o jsonpath='{.status.replicas}' | jq \
    'map({node, collectedAt, collectionError})'
  d8 k get registrystorage registry -o jsonpath='{.spec.garbageCollection}' | jq
  

  The likely reasons, in the order they are worth checking:

  • the collection is turned off in the registry ModuleConfig under storage.garbageCollection.enabled;
  • the schedule has not come round yet, or names an hour that never arrives — an unreadable expression is refused rather than guessed at — and the image set has not changed either;
  • the collections are created and fail: the RegistryOperation objects of type GarbageCollection say why in status.replicas, for a day after they finish;
  • no release is deployed, which is the one input that justifies deleting anything: every deletion is “older than what the cluster runs”, and without that there is nothing to compare against;
  • collectionError on a replica, which says what its own last attempt hit.

  One reading tells the first two apart from the rest. The controller logs every collection it creates and every firing of the schedule it skips, so their absence means none was ever started — a different problem from one that is started and failing:

  d8 k -n d8-system logs -l app=registry-controller -c registry-controller --tail=-1 | grep -iE "garbage collection|schedule fired"
  

  Collecting is not an outage — the replica keeps serving every image it holds — but it pauses filling the replica’s store and can fail a d8 mirror push started meanwhile, so the schedule belongs at a quiet hour, which is where it defaults to.

• D8RegistryStorageUpdateBlocked CE S6
  The in-cluster registry cache cannot be updated to its new revision.

  The registry-storage StatefulSet has a new revision, and the controller that replaces its replicas has refused to replace the next one for an hour. Existing replicas remain running, but the cache update cannot proceed. The reason label identifies what is blocking it.

  Reason: XXX

  • HolderNotReady — the cache has one replica and no upstream, so its new images must be on its master before it is replaced. The registry-storage-image-holder DaemonSet keeps them there and has not got all of them yet. If its pod on the master is in ImagePullBackOff, the cache does not hold an image: push the Deckhouse Platform release the cluster is updating to with d8 mirror push, and the holder pulls it on its next retry. If the pod is in CrashLoopBackOff or CreateContainerError, the holder has the image but cannot run it, and a push does not help: the cause is in the events of the pod. Otherwise it is still rolling or pulling, and there is nothing to do.
  • SingleReplicaWithoutHolder — the same, and the DaemonSet has no pod on that master at all. For a moment this is normal while the DaemonSet creates its pod there; lasting an hour, it means the DaemonSet does not schedule a pod there. Find out why with d8 k -n d8-system describe daemonset registry-storage-image-holder.
  • LeaderWithoutFullFollower — only the leader is left to replace, and no follower holds the whole image set to take over from it. If the leader holds the set, the followers are copying it and the update continues once one of them finishes. If the leader reports that it does not hold it, the followers cannot complete the set either, since they copy it from the leader: pull the Deckhouse Platform release and the modules the cluster uses with d8 mirror pull, as described in the air-gapped update guide, and push them into the cache with d8 mirror push, as in step 2 of Going air-gapped. If the leader has not reported yet, reports an error or cannot tell which set the cluster needs, pushing does not help: the cause is in the log of the syncer container of the leader’s pod.
  • LeaderNotServing — the storage lease names a master where no serving replica runs. The replicas move the lease on their own once it expires.
  • StorageUnknown — the RegistryStorage resource does not exist. The controller creates it; check the log of registry-controller.

  The event on the StatefulSet says which replica and what it is waiting for:

  d8 k -n d8-system get events --field-selector involvedObject.name=registry-storage,reason=UpdateBlocked
  

  And the two things it is waiting on:

  d8 k -n d8-system get pods -l app=registry-storage-image-holder -o wide
  d8 k get registrystorage registry -o jsonpath='{.status.replicas}' | jq
  

• D8RegistryStoreCorrupted CE S5
  The registry cache holds broken blobs that were not repaired.

  A replica of the in-cluster cache holds blobs whose content does not match their digest, and they have not been repaired for two hours. Every pull of an image that uses such a blob from this replica fails; while an image of the cluster’s set uses one, the replica is not full, and a leader hands the lease to a replica that holds the set whole.

  The module creates a RegistryOperation of type Repair for these blobs on its own, and repeats it four hours after one that did not fix them. Where to look:

  d8 k get registrystorage registry -o jsonpath='{.status.replicas}' | jq 'map({node, full, corruptedBlobs})'
  d8 k get registryoperation -o wide
  d8 k get registryimages -l registry.deckhouse.io/state=Corrupted
  

  status.replicas[].repair.digests[].message of the last Repair says why a blob could not be repaired. The usual reasons:

  • no whole copy anywhere: the cache has one replica and no upstream, or every replica holds the blob broken — as after a d8 mirror push of a broken bundle. Push the release again with d8 mirror push, then create a Repair for the digests by hand;
  • another operation held the queue, such as a long garbage collection: the repair runs once it finishes;
  • the other replicas could not be reached from this one.

  The disk under the store is worth checking too: blobs that go bad on their own are a sign of a failing disk.

• D8RegistryStoreNearlyFull CE S6
  The registry cache on node NODE_NAME holds over 85% of storage.size.

  The in-cluster registry cache on node NODE_NAME holds more than 85% of its budget, storage.size in the registry ModuleConfig. Nothing is refused yet. The next release may not fit: once the budget is reached, the cache stops accepting writes and D8RegistryStoreWritesRefused follows.

  d8 k get registrystorage registry -o json | jq '.status.replicas[] | {node, store}'
  

  Raise storage.size if the disk under the store has room — filesystemFreeBytes above reserveBytes is what is left — or check that the garbage collection runs: it is the only thing that reclaims space, and D8RegistryStorageNotReclaimed says when it has not.

• D8RegistryStoreUnbounded CE S7
  The registry cache has no storage.size, so only the node's disk limits it.

  The in-cluster registry cache is on, and storage.size is not set in the registry ModuleConfig. The node is still protected: the store never takes the space kept free for the kubelet. But that is the only limit, so the store may grow until it leaves the node exactly that and no more — and on a master that shares its disk with the store, everything else on it then works with the minimum.

  Set storage.size to what the store may take on each master: enough for three releases of the image set together with the modules and the module catalog, and no more than fits on the disk under the store above the reserve:

  d8 k get registrystorage registry -o json | jq '.status.replicas[] | {node, store}'
  

  An hour, so that turning the cache on and setting the size in two steps does not raise it.

• D8RegistryStoreWritesRefused CE S4
  The registry cache on node NODE_NAME does not accept writes (XXX).

  The in-cluster registry cache on node NODE_NAME has stopped accepting writes, because XXX. It keeps serving every image it holds. What stops is everything that ADDS to it: filling the next release, replicating from the leader, d8 mirror push. In an air-gapped cluster a release that is not in the cache cannot be installed.

  What the replica measured, and the error it reports:

  d8 k get registrystorage registry -o json | jq '.status.replicas[] | {node, store, error}'
  

  BudgetExhausted — the store holds as much as storage.size allows. The node is safe: this is the limit doing its job, and images the store cannot keep are streamed from the upstream while there is one. Either raise storage.size in the registry ModuleConfig, if the disk has room for it, or let the garbage collection reclaim what no deployed release needs any more: it runs on storage.garbageCollection.schedule and keeps the deployed release, the previous one and everything newer.

  ReserveExhausted — the filesystem under the store is down to the space kept free on it. The node is safe for the same reason, and images are streamed the same way. storage.size is not what stops it, the disk is. On a filesystem shared with the node (dedicatedFilesystem absent from the replica’s store) that is the kubelet’s eviction threshold plus a margin: free space on the node, or — which is what fixes it for good — give the store a disk of its own, sized for storage.size. On a disk of the store’s own (dedicatedFilesystem: true) it is the margin alone, and the disk itself is full: grow it — the reserve follows without a restart — or let the garbage collection reclaim what no deployed release needs, as for BudgetExhausted.

  NotWritable — the registry cannot create files in its tree in the store, whatever room the store has: docker/registry/v2 and the directories under it, or the store’s directory while that tree does not exist yet. It runs as root without capabilities, so those directories have to belong to root and let their owner write. Other causes are the immutable attribute, a read-only mount, or a security module such as SELinux or AppArmor. No limit is involved: neither the garbage collection nor a larger storage.size helps, and the collection does not start for it. Until it is fixed, a pull the cache would keep from the upstream fails with an error instead of being streamed. The replica’s error names the directory on the node, its owner and mode, and the command that fixes it:

  d8 k get registrystorage registry -o json | jq -r '.status.replicas[] | select(.store.reason == "NotWritable") | "\(.node): \(.error)"'
  

  NotWritable takes precedence over a limit the store is also at. Once it is fixed, the replica says so within a minute, and the alert clears or names the limit.

• D8RegistryUpstreamProbeFailing CE S4
  The configured registry upstream does not pass its preflight check.

  A change to the primary upstream was refused, and the cluster is still using the last one that worked. Nothing is degraded — that is what holding the previous upstream is for — but the change has not taken effect and will not until the check passes.

  The outcome label says which of three things is wrong, and they are not the same problem:

  • unreachable — the registry did not answer at all: network, firewall, or the registry itself.
  • auth — it answered and rejected the credentials. Usually an expired or wrong license key.
  • sentinel — it answered and accepted the credentials, but does not hold the Deckhouse images. Usually the wrong repository path.

  Current outcome: XXX

  d8 k get registryconfig registry -o jsonpath='{.status.conditions}' | jq
  

• D8RegistryUpstreamRejected CE S6
  An additional registry was refused.

  A RegistryUpstream resource was not accepted, so pulls for the registry it names are not intercepted on any node. Whatever asked for it — a module, or an operator — is not getting it.

  Reason: XXX

  d8 k get registryupstreams -o custom-columns=\
  NAME:.metadata.name,MATCH:.spec.match,ACCEPTED:.status.conditions[0].status,REASON:.status.conditions[0].reason
  

Module secret-copier

• D8SecretCopierDeprecatedLabels CE S9
  Obsolete antiopa_secret_copier=yes label detected.

  The secret-copier module has changed its labeling approach for original secrets in the default namespace.

  The label antiopa-secret-copier: "yes" is deprecated and will be removed soon.

  To resolve this issue, replace the label antiopa-secret-copier: "yes" with secret-copier.deckhouse.io/enabled: "" for all secrets used by the secret-copier module in the default namespace.

Module terraform-manager

• D8TerraformStateExporterClusterStateChanged CE S8
  Terraform-state-exporter cluster state change detected.

  The current Kubernetes cluster state is STATUS_REFERENCE compared to the Terraform state.

  It is important to reconcile the states.

  Troubleshooting steps:

  1. View the differences:

     dhctl terraform check
     

  2. Apply the necessary changes to bring the cluster in sync:

     dhctl converge
     

• D8TerraformStateExporterClusterStateError CE S8
  Terraform-state-exporter cluster state error.

  The terraform-state-exporter can’t check difference between the Kubernetes cluster state and the Terraform state.

  That was likely caused by terraform-state-exporter failing to run Terraform with the current state and configuration.

  Troubleshooting steps:

  1. View the differences:

     dhctl terraform check
     

  2. Apply the necessary changes to bring the cluster in sync:

     dhctl converge
     

• D8TerraformStateExporterHasErrors CE S8
  Terraform-state-exporter has encountered errors.

  Errors occurred during the operation of the terraform-state-exporter.

  To get more details, check the Pod logs:

  d8 k -n d8-system logs -l app=terraform-state-exporter -c exporter
  

• D8TerraformStateExporterNodeStateChanged CE S8
  Terraform-state-exporter node state change detected.

  The current state of node NODE_GROUP_NAME/NAME is STATUS_REFERENCE compared to the Terraform state.

  It is important to reconcile the states.

  Troubleshooting steps:

  1. View the differences:

     dhctl terraform check
     

  2. Apply the necessary changes to bring the cluster in sync:

     dhctl converge
     

• D8TerraformStateExporterNodeStateError CE S8
  Terraform-state-exporter node state error.

  The terraform-state-exporter can’t check the difference between the node NODE_GROUP_NAME/NAME state and the Terraform state.

  Probably, it occurred because terraform-manager had failed to run Terraform with the current state and configuration.

  Troubleshooting steps:

  1. View the differences:

     dhctl terraform check
     

  2. Apply the necessary changes to bring the cluster in sync:

     dhctl converge
     

• D8TerraformStateExporterNodeTemplateChanged CE S8
  Terraform-state-exporter node template change detected.

  The terraform-state-exporter has detected a mismatch between the node template in the cluster provider configuration and the one specified in the NodeGroup NAME`.

  Node template is STATUS_REFERENCE.

  Troubleshooting steps:

  1. View the differences:

     dhctl terraform check
     

  2. Adjust NodeGroup settings to fix the issue or bring the cluster in sync via the following command:

     dhctl converge
     

• D8TerraformStateExporterPodIsNotReady CE S8
  Terraform-state-exporter Pod is not Ready.

  The terraform-state-exporter cannot check the difference between the actual Kubernetes cluster state and the Terraform state.

  To resolve the issue, check the following:

  1. Deployment description:

     d8 k -n d8-system describe deployment terraform-state-exporter
     

  2. Pod status:

     d8 k -n d8-system describe pod -l app=terraform-state-exporter
     

• D8TerraformStateExporterPodIsNotRunning CE S8
  Terraform-state-exporter Pod is not Running.

  The terraform-state-exporter cannot check the difference between the actual Kubernetes cluster state and the Terraform state.

  To resolve the issue, check the following:

  1. Deployment description:

     d8 k -n d8-system describe deployment terraform-state-exporter
     

  2. Pod status:

     d8 k -n d8-system describe pod -l app=terraform-state-exporter
     

• D8TerraformStateExporterTargetAbsent CE S8
  Terraform-state-exporter target is missing in Prometheus.

  Prometheus cannot find the terraform-state-exporter target.

  To investigate the details:

  • Check the Pod status:

    d8 k -n d8-system get pod -l app=terraform-state-exporter
    

  • Check the container logs:

    d8 k -n d8-system logs -l app=terraform-state-exporter -c exporter
    

• D8TerraformStateExporterTargetDown CE S8
  Prometheus can't scrape terraform-state-exporter.

  Prometheus is unable to scrape metrics from the terraform-state-exporter.

  To investigate the details:

  • Check the Pod status:

    d8 k -n d8-system get pod -l app=terraform-state-exporter
    

  • Check the container logs:

    d8 k -n d8-system logs -l app=terraform-state-exporter -c exporter
    

Module user-authn

• D8DexAllTargetsDown CE S6
  Prometheus is unable to scrape Dex metrics.

  Prometheus cannot scrape any Dex pod in the d8-user-authn namespace. Users may be unable to sign in.

  Check the Dex pods and their logs:

  d8 k -n d8-user-authn get pods -l app=dex
  d8 k -n d8-user-authn logs deploy/dex
  

• D8DexAuthenticatorNamespaceUnusable CE S7
  DexAuthenticator NAMESPACE/NAME is left out of the release because its namespace accepts no objects.

  The NAMESPACE namespace is terminating, so the API server refuses every create in it. The user-authn module leaves the DexAuthenticator NAME out of its release: with it, the release would fail on the objects it cannot create and the Deckhouse main queue would stop behind the retrying module.

  A namespace does not finish terminating while something in it holds a finalizer, and the authenticator is usually that something. Check what keeps the namespace:

  d8 k get namespace NAMESPACE -o jsonpath='{.status.conditions}'
  d8 k -n NAMESPACE get dexauthenticator NAME -o jsonpath='{.metadata.finalizers}'
  

  Let the deletion finish: remove the finalizer that holds the authenticator, or delete the object that owns it. The authenticator is deleted with its namespace; to keep it, recreate it in a live namespace. The alert resolves once the authenticator is gone.

• D8DexAuthRequestsQuotaExhausted CE S3
  Dex AuthRequest objects ResourceQuota is exhausted in namespace "d8-user-authn".

  The ResourceQuota for authrequests.dex.coreos.com objects in namespace d8-user-authn has been fully exhausted.

  Dex is currently unable to create new AuthRequest custom resources, which will cause:

  • Dex health checks to fail immediately.
  • Authentication through external identity providers to stop working.
  • Users unable to log in via OAuth/OIDC flows.

  Immediate actions required:

  1. Check current AuthRequest objects and clean up stale ones:

     d8 k get authrequests.dex.coreos.com -n d8-user-authn --no-headers | wc -l
     d8 k delete authrequests.dex.coreos.com --all -n d8-user-authn
     

     Deleting AuthRequest objects interrupts sign-ins that are in progress.

  2. Temporarily increase the ResourceQuota limit:

     d8 k patch resourcequota authrequest-quota -n d8-user-authn --type='json' -p='[{"op": "replace", "path": "/spec/hard/count\/authrequests.dex.coreos.com", "value": "20000"}]'
     

     The module restores the quota on its next run.

  3. Investigate why Dex is not cleaning up AuthRequest objects automatically.

  4. AuthRequest objects expire after 10 minutes. A growing count usually means that a client or a bot repeatedly starts sign-ins. Find the source in the Dex logs.
• D8DexAuthRequestsQuotaHigh CE S4
  Dex AuthRequest objects quota usage is higher than 80% in namespace "d8-user-authn".

  The ResourceQuota for authrequests.dex.coreos.com objects in namespace d8-user-authn is approaching its limit.

  Current usage: XXX%

  When the quota is exhausted, Dex will be unable to create new AuthRequest custom resources, which may cause:

  • Dex health checks to fail.
  • Authentication through external identity providers (GitLab, GitHub, etc.) to stop working.

  Investigation steps:

  1. Check for stale AuthRequest objects that Dex should have cleaned up:

     d8 k get authrequests.dex.coreos.com -n d8-user-authn
     

  2. AuthRequest objects expire after 10 minutes. A growing count usually means that a client or a bot repeatedly starts sign-ins. Find the source in the Dex logs.

  3. Consider temporarily increasing the ResourceQuota limit if needed. The module restores the quota on its next run.
• D8DexProviderLDAPTLSConflict CE S8
  DexProvider NAME has a conflicting LDAP TLS configuration.

  DexProvider NAME combines mutually exclusive LDAP TLS settings: XXX. Dex silently ignores the redundant flag, which masks misconfiguration:

  • insecureNoSSL + startTLS: TLS is disabled (ldap://, port 389), so startTLS is ignored.
  • insecureNoSSL + insecureSkipVerify: TLS is disabled, there is no certificate to verify.
  • insecureNoSSL + rootCAData: TLS is disabled, the CA chain is ignored.

  The CRD schema rejects this combination on create/update; this DexProvider was created before the validation was added and remains active.

  Update the DexProvider to remove the conflict:

  • Plaintext LDAP (port 389): keep spec.ldap.insecureNoSSL: true, remove startTLS, insecureSkipVerify, and rootCAData.
  • STARTTLS on port 389: set spec.ldap.insecureNoSSL: false, spec.ldap.startTLS: true. Use insecureSkipVerify or rootCAData as needed.
  • ldaps:// on port 636: leave insecureNoSSL and startTLS at false. Use insecureSkipVerify or rootCAData as appropriate.

  Audit all LDAP DexProvider objects:

  d8 k get dexproviders.deckhouse.io -o yaml | yq '.items[] | select(.spec.type=="LDAP") | {name: .metadata.name, ldap: .spec.ldap | pick(["insecureNoSSL","startTLS","insecureSkipVerify","rootCAData"])}'
  

• D8UserAuthnControllerReconcileErrorsHigh CE S8
  JOB_NAME reconcile errors are high for CONTROLLER_NAME.

  Controller CONTROLLER_NAME of JOB_NAME has a reconcile error rate above 0.1 errors per second for more than 10 minutes.

  Check logs:

  d8 k -n d8-user-authn logs deploy/JOB_NAME --all-pods=true
  

• D8UserAuthnControllerTargetDown CE S6
  Prometheus is unable to scrape user-authn-controller.

  Prometheus cannot scrape metrics from user-authn-controller in namespace d8-user-authn.

  The controller is down, not Ready (informer cache has not synced), or the PodMonitor is missing. Static user login, UserAccount, expire, and provider checks will not run.

  Check:

  d8 k -n d8-user-authn get pods -l app=user-authn-controller
  d8 k -n d8-user-authn describe pods -l app=user-authn-controller
  

• D8UserAuthnControllerUnavailable CE S6
  user-authn-controller has unavailable replicas.

  Deckhouse has detected unavailable replicas in the user-authn-controller deployment in the d8-user-authn namespace.

  While this alert is active, static Users may not get Dex Password objects, UserAccount projections stall, expired Users are not deleted, and DexProviderCheck / UserOperation stay pending.

  Check deployment and pod states:

  d8 k -n d8-user-authn get deploy user-authn-controller
  d8 k -n d8-user-authn get pods -l app=user-authn-controller
  d8 k -n d8-user-authn logs deploy/user-authn-controller --all-pods=true
  

• D8UserAuthnDexClientControllerTargetDown CE S6
  Prometheus is unable to scrape dexclient-controller.

  Prometheus cannot scrape metrics from dexclient-controller in namespace d8-user-authn.

  The controller is down, not Ready (informer cache has not synced), or the PodMonitor is missing. The credentials of DexClients are not kept up to date.

  Check:

  d8 k -n d8-user-authn get pods -l app=dexclient-controller
  d8 k -n d8-user-authn describe pods -l app=dexclient-controller
  

• D8UserAuthnDexClientControllerUnavailable CE S6
  dexclient-controller has unavailable replicas.

  Deckhouse has detected unavailable replicas in the dexclient-controller deployment in the d8-user-authn namespace.

  While this alert is active, a new DexClient gets no credentials Secret and no Dex client, and changes of DexClients are not applied.

  Check deployment and pod states:

  d8 k -n d8-user-authn get deploy dexclient-controller
  d8 k -n d8-user-authn get pods -l app=dexclient-controller
  d8 k -n d8-user-authn logs deploy/dexclient-controller --all-pods=true
  

• D8UserAuthnDexProviderTokenExchangeDeprecated CE S8
  Tokens are exchanged through DexProvider XXX, which is deprecated.

  In the last 24 hours, Dex exchanged tokens (RFC 8693 token exchange) through the DexProvider XXX. Token exchange through a DexProvider is deprecated. A DexProvider is meant for sign-in, and its connector does not check the audience of the token it exchanges.

  Move the exchange to a TokenExchangeProvider:

  1. Create a TokenExchangeProvider with the issuer of the DexProvider, the audience the exchanged tokens carry, and the DexClient that exchanges them in spec.clients.
  2. Set spec.identity.username to the claim the DexProvider took the user name from, for example claim: email, with a prefix such as ci:; the prefix is required.
  3. Before the switch, add the prefixed subjects to the ClusterAuthorizationRule, AuthorizationRule and RoleBinding objects that grant rights to the old names.
  4. In the exchange requests, replace connector_id with status.connectorID of the new object.
  5. After the switch, remove the old subjects from these objects.
  6. Turn off token exchange through the DexProvider by setting spec.tokenExchange.enabled: false. It cannot be turned on again, and token exchange through a DexProvider will be removed after the next LTS release.

  Find the clients that exchange tokens in the Dex log:

  d8 k -n d8-user-authn logs deploy/dex -c dex --all-pods=true --since=24h | grep 'token exchange successful' | grep '"connector_id":"XXX"'
  

  The alert stops 24 hours after the last exchange through the DexProvider.

• D8UserAuthnIdentityMigrationIncomplete CE S6
  Dex user data is not fully copied into UserIdentity objects.

  For more than 30 minutes, user-authn-controller has not finished copying the 2FA (TOTP) secret and the lock of Dex users from OfflineSessions and Password objects into UserIdentity objects.

  Until the copy of a provider and user pair completes, its 2FA secret and lock are stored only in the OfflineSessions and Password objects.

  The controller retries failed pairs by itself. Find the reason in its logs:

  d8 k -n d8-user-authn logs deploy/user-authn-controller --all-pods=true | grep -E 'identitymigration|UserIdentity'
  

  Typical reasons are API server errors and missing rights on useridentities.dex.coreos.com. Pairs that cannot be copied at all, for example an OfflineSessions object that cannot be decoded, are logged as Not copying the pair into UserIdentity and do not raise this alert.

• D8UserAuthnIDTokenTTLTooLong CE S6
  user-authn idTokenTTL is XXX, which is too long.

  The ModuleConfig user-authn sets idTokenTTL: XXX.

  Dex rotates the key that signs ID tokens every 6 hours (signer.config.keysRotationPeriod: 6h) and keeps a previous key only while tokens signed with it can still be valid.

  idTokenTTL maps to Dex expiry.idTokens. If that lifetime is greater than or equal to the signing-key rotation period (6h), every rotation must retain an extra previous key until all tokens from that key expire. Over time the set of retained signing keys grows without bound and Dex persists that growing key material.

  That is why idTokenTTL must be strictly less than 6h. Lower it, for example:

  apiVersion: deckhouse.io/v1alpha1
  kind: ModuleConfig
  metadata:
    name: user-authn
  spec:
    version: 3
    settings:
      idTokenTTL: 1h
  

  Settings version 3 rejects idTokenTTL >= 6h. A ModuleConfig of an earlier settings version with such a value is still accepted and its value is converted to 5h59m, so Dex already runs with 5h59m; this alert keeps firing until the ModuleConfig itself is updated.

• D8UserAuthnRevocationFeedLagging CE S6
  The revocation feed of user-authn-controller is not delivering to dex-authenticator.

  For more than 5 minutes, a stream of the revocation feed of user-authn-controller has not written to its dex-authenticator for over 30 seconds. A stream writes at least a heartbeat every 10 seconds, so the stream is stuck.

  While the stream is stuck, a session that is revoked (user lock, password reset, logout) keeps working in that dex-authenticator until its next refresh in Dex: up to 30 seconds while Dex is reachable, and no longer than idTokenTTL otherwise.

  Check the logs of the controller and of the dex-authenticator pods:

  d8 k -n d8-user-authn logs deploy/user-authn-controller --all-pods=true | grep revocation-feed
  

  The controller closes a stream that falls behind, and the dex-authenticator reconnects by itself. A lag that does not go away usually means a network problem between the dex-authenticator pods and the controller pods.

• D8UserAuthnRevocationFeedNotConsumed CE S6
  The dex-authenticator of the Dex client XXX does not receive the revocation feed.

  For more than 15 minutes, no replica of user-authn-controller has had a stream of the revocation feed open for the Dex client XXX. The client belongs to a DexAuthenticator (the client ID is dex-authenticator-name@namespace).

  The dex-authenticator of this client works without the feed: a session revoked by a user lock, a password reset, or a logout stops working there only at its next refresh in Dex, within 30 seconds while Dex is reachable, and no later than idTokenTTL.

  What to check:

  • the logs of the dex-authenticator pods: errors connecting to https://revocation-feed.d8-user-authn/revocations and answers 401 or 429;
  • that the Secret of the DexAuthenticator holds the CA of the feed in the revocation-feed-ca.crt key;
  • network policies in the namespace of the DexAuthenticator: they must allow connections to the user-authn-controller pods in d8-user-authn on port 8443.

  d8 k -n d8-user-authn get endpoints revocation-feed
  d8 k -n d8-user-authn logs deploy/user-authn-controller --all-pods=true | grep revocation-feed
  

Module user-authz

• D8UserAuthzAuthorizationRuleInvalid CE S7
  The bindings of XXX XXXXXX/XXXNAME cannot be applied (XXX).

  user-authz-controller cannot apply the bindings of XXX XXXXXX/XXXNAME; the Ready condition of the object is False with the reason XXX.

  • InvalidSpec: the rule cannot be rendered into bindings (unsupported accessLevel, an accessLevel not allowed for a namespaced rule, an additionalRoles entry that is not a ClusterRole or has an invalid name). Fix the object.
  • ApplyError: the API server rejected a write. The message of the condition names the binding and the error; a binding whose roleRef was changed by hand has to be deleted so that the controller recreates it.

  d8 k XXX-n XXX XXXget XXX NAME -o jsonpath='{.status.conditions}'
  

• D8UserAuthzAuthorizationRulesInvalid CE S7
  VALUE XXX objects cannot be applied (XXX).

  user-authz-controller reports VALUE XXX objects whose bindings cannot be applied, reason XXX. Up to 50 of them are named by the D8UserAuthzAuthorizationRuleInvalid alerts; a large number with the reason ApplyError points at a cluster-wide cause (an admission webhook, API server errors) rather than at the objects.

  d8 k get XXX -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,READY:'.status.conditions[?(@.type=="Ready")].status',REASON:'.status.conditions[?(@.type=="Ready")].reason' | grep -v True
  d8 k -n d8-user-authz logs deploy/user-authz-controller --all-pods=true
  

• D8UserAuthzBindingsDrift CE S7
  user-authz-controller cannot converge the bindings of XXX.

  For more than 15 minutes some bindings of XXX objects differ from the state user-authz-controller wants (missing, superfluous, or with different subjects, roleRef, labels or owner). The controller reports drift only while its writes fail, so users may be missing access or keep access they should have lost.

  Find the objects and the reason:

  d8 k get clusterauthorizationrules -o custom-columns=NAME:.metadata.name,READY:'.status.conditions[?(@.type=="Ready")].status',REASON:'.status.conditions[?(@.type=="Ready")].reason',MESSAGE:'.status.conditions[?(@.type=="Ready")].message'
  d8 k get authorizationrules -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,READY:'.status.conditions[?(@.type=="Ready")].status',MESSAGE:'.status.conditions[?(@.type=="Ready")].message'
  d8 k -n d8-user-authz logs deploy/user-authz-controller --all-pods=true
  

• D8UserAuthzControllerReconcileErrorsHigh CE S8
  user-authz-controller reconcile errors are high for CONTROLLER_NAME.

  Controller CONTROLLER_NAME of user-authz-controller has a reconcile error rate above 0.1 errors per second for more than 10 minutes.

  Check logs:

  d8 k -n d8-user-authz logs deploy/user-authz-controller --all-pods=true
  

• D8UserAuthzControllerTargetDown CE S6
  Prometheus is unable to scrape user-authz-controller.

  Prometheus cannot scrape metrics from user-authz-controller in the d8-user-authz namespace.

  The controller is down, not Ready (its informer cache has not synced), or the PodMonitor is missing. Access rules are not reconciled while it is down.

  Check:

  d8 k -n d8-user-authz get pods -l app=user-authz-controller
  d8 k -n d8-user-authz describe pods -l app=user-authz-controller
  d8 k -n d8-monitoring get podmonitor user-authz-controller
  

• D8UserAuthzControllerUnavailable CE S6
  user-authz-controller has unavailable replicas.

  Deckhouse has detected unavailable replicas in the user-authz-controller deployment in the d8-user-authz namespace.

  While this alert is active, changes to ClusterAuthorizationRule and AuthorizationRule objects are not applied to their ClusterRoleBindings and RoleBindings, and the d8:dict bindings and the RoleBindings of system and subsystem roles in module namespaces are not updated. Existing bindings keep working.

  Check deployment and pod states:

  d8 k -n d8-user-authz get deploy user-authz-controller
  d8 k -n d8-user-authz get pods -l app=user-authz-controller
  d8 k -n d8-user-authz logs deploy/user-authz-controller --all-pods=true
  

• D8UserAuthzCustomRolesNotAggregated CE S6
  The aggregated ClusterRole of the XXX access level has no rules although custom roles for it exist.

  Custom ClusterRoles annotated with user-authz.deckhouse.io/access-level are granted to users through the aggregated ClusterRole user-authz:level:custom, which kube-controller-manager fills from the roles labelled with the level. For more than 10 minutes the aggregated role of the XXX level has no rules while custom roles it must aggregate exist, so users of that level (and every level above it) have none of the custom permissions.

  Check that the aggregated role exists and that the clusterrole-aggregation controller of kube-controller-manager is running:

  d8 k get clusterrole -l user-authz.deckhouse.io/access-level
  d8 k get clusterrole -o name | grep 'user-authz:.*:custom' | xargs -n1 d8 k get -o yaml
  d8 k -n kube-system logs -l component=kube-controller-manager --tail=200 | grep -i aggregat
  

• D8UserAuthzDeprecatedRBACv2CapabilityInUse CE S5
  A binding references a deprecated RBACv2 capability that no longer grants access.

  The XXX XXXXXX in namespace NAMESPACEXXX binds the deprecated RBACv2 capability XXX directly.

  Capabilities (d8:manage:permission:*, d8:use:capability:*) are aggregation building blocks — they are meant to be composed into the d8:{system,subsystem,namespace,project}:* roles, not bound directly. They were renamed and have NO compatibility alias, so this binding already grants nothing after the upgrade.

  No ClusterRole named XXX exists in the cluster. The alert also names a binding to an old role name that a module no longer ships, for example after the module moved to the new role names.

  Recreate the binding against the appropriate new role (or a custom role aggregating the new capability). Because roleRef is immutable, delete and recreate the binding:

  d8 k get XXX XXXXXX -n NAMESPACEXXX -o yaml
  

• D8UserAuthzDeprecatedRBACv2ModuleRoleInUse CE S7
  A binding references the old name of a role a module renamed.

  The XXX XXXXXX in namespace NAMESPACEXXX references XXX, the old name of a role that a module renamed to XXX.

  The module keeps the old name as a compatibility alias, so the binding keeps working. The alias may grant less than XXX, for example when the module moved into a wider subsystem and the alias kept the rights of the old role, so check what the subjects of the binding need before recreating it. The module removes the alias in one of its own releases; the Deckhouse release is not held back by this binding. See the documentation of the module for when.

  Recreate the binding on the new name. Because roleRef is immutable, delete and recreate the binding pointing at XXX:

  d8 k get XXX XXXXXX -n NAMESPACEXXX -o yaml
  

• D8UserAuthzDeprecatedRBACv2RoleInUse CE S7
  A binding still references a deprecated RBACv2 role.

  The XXX XXXXXX in namespace NAMESPACEXXX references the deprecated RBACv2 ClusterRole XXX.

  The RBACv2 role model was renamed: d8:manage:subsystem|all:level became d8:system:* / d8:subsystem:*:*, and d8:use:role:level became d8:namespace:level. This name still works via a compatibility alias kept for one release and will be removed in the next one — after that this binding will grant nothing. The release that removes the aliases is held back by the deprecatedRBACv2BindingsCount release requirement until this binding is recreated on the new name.

  Migrate the binding to the new role name. Because roleRef is immutable, delete and recreate the binding pointing at the new role:

  d8 k get XXX XXXXXX -n NAMESPACEXXX -o yaml
  # Recreate it with roleRef.name set to the new role (d8:system:*, d8:subsystem:*, or d8:namespace:*).
  

  XXXuser-authz-controller renders this binding from spec.additionalRoles of the ClusterAuthorizationRule XXX and creates it again when it is deleted. Replace XXX with the new role name in spec.additionalRoles of that ClusterAuthorizationRule instead.XXXIf a project template renders this binding, replace the role name in the ProjectTemplate instead, because multitenancy-manager renders the binding again from it.XXXXXXXXX

  In the new model the rights for user-authn and user-authzXXX and the right to assign rolesXXX come with d8:subsystem:iam:XXX, not with d8:subsystem:security:XXX. To keep them, replace this binding with two bindings with the same subjects, one to d8:subsystem:iam:XXX and one to d8:subsystem:security:XXX. d8:subsystem:iam:XXX also grants the rights for multitenancy-manager.XXX

  The list of subsystems changed as well. deckhouse, infrastructure and kubernetes were merged into cluster, networking became network, the user-authz and user-authn modules moved from security to iam, and multitenancy-manager moved from deckhouse to iam. The aliases of the old subsystem roles select the capabilities of the modules these roles covered, and the old subsystem labels that modules released on their own may still carry. The capabilities themselves changed in this release, so the rights of the binding are close to the ones it had before the upgrade, not the same. The replacement d8:subsystem:cluster:level covers the whole cluster subsystem and grants more, and so do the roles of the other subsystems that the description of the alias names, so check which roles the subjects of the binding need. The user-authz documentation lists the remaining differences.

  A ClusterRoleBinding to a deprecated d8:manage:* name also keeps the RoleBindings d8:use:use role:binding:name that the previous release created for it in the namespaces of the modules of its old subsystem (d8-cloud-provider-*, d8-monitoring and the rest). The use role is viewer for a viewer alias and user for a manager alias. When the binding is recreated on the new role name, they are replaced by RoleBindings to the namespace role of the new role in the namespaces of the modules of the new subsystem.

  Note: the new d8:namespace:admin no longer grants serviceaccounts/token creation or impersonation — these disaster-grade actions now require a superadmin role by design.

• D8UserAuthzForeignAggregationLabel CE S6
  ClusterRole NAME carries the XXX aggregation label without being a custom capability.

  The ClusterRole NAME carries the rbac.deckhouse.io/aggregate-to-XXX-as label but is not labelled rbac.deckhouse.io/kind: custom-capabilityXXX (its kind label is XXX)XXX. The aggregation controller reads nothing but that label, so it merges the rules of this role into XXXthe platform d8:XXX:* rolesXXXthe deprecated d8:manage:all:* and d8:use:role:* roles and every custom role that selects the labelXXXthe d8:subsystem:XXX:* roles, if a subsystem has this name, and into every custom role that selects the labelXXX. Every holder of these roles in the cluster gets the rules, outside the role model’s contract.

  For the aggregation label of the system scope, of the namespace or project scope, or of a built-in subsystem, the rbacv2-cluster-roles.deckhouse.io webhook refuses such an object, and every UPDATE of this role, including a GitOps re-apply, is denied until it is fixed. Choose one:

  • If the role is meant to extend the platform roles, replace it with a custom capability. The name of an object cannot be changed, so create d8:custom:scope-capability:name with the same rules and aggregation label, the rbac.deckhouse.io/kind: custom-capability and rbac.deckhouse.io/scope: scope labels, and no aggregationRule, and then delete this role. Deckhouse reads rbac.deckhouse.io/namespace only on the ClusterRoles it ships: if this role has that label, neither it nor its replacement gives RoleBinding objects in that namespace, so bind the subjects to a namespace role there with a RoleBinding. A capability of the old scheme with the rbac.deckhouse.io/kind: use label and rbac.deckhouse.io/aggregate-to-kubernetes-as: LEVEL or rbac.deckhouse.io/aggregate-to-all-as: LEVEL extended the namespace roles, so its replacement is d8:custom:namespace-capability:name with rbac.deckhouse.io/scope: namespace and rbac.deckhouse.io/aggregate-to-namespace-as: LEVEL instead. For the commands, refer to the corresponding section in the user-authz module FAQ.
  • Otherwise remove the rbac.deckhouse.io/aggregate-to-*-as labels from it.

  List every such role:

  d8 k get clusterroles -o json | jq -r '.items[] | select((.metadata.labels // {}) | to_entries | any(.key | startswith("rbac.deckhouse.io/aggregate-to-"))) | select((.metadata.labels.heritage // "") != "deckhouse") | select((.metadata.labels["rbac.deckhouse.io/kind"] // "") != "custom-capability") | .metadata.name'
  

• D8UserAuthzLegacyRBACv2CustomRoleFound CE S7
  The custom role NAME has lost access it gave through the labels of an earlier role model.

  The ClusterRole NAME has no heritage: deckhouse label and has an aggregationRule field, and one of the following holds:

  • A selector of its aggregationRule field requires the rbac.deckhouse.io/kind: manage or use label of the experimental role model. The built-in capabilities no longer carry these labels, so such a selector matches none of them. The Kubernetes aggregation controller keeps from it only the rules of the custom capabilities that still carry the old labels, and the subjects of the role have lost the other permissions it gave. Custom roles get no compatibility aliases.
  • The role has the rbac.deckhouse.io/use-role label without rbac.deckhouse.io/scope: system or subsystem. Deckhouse creates the RoleBinding objects for the subjects of a role in the namespaces of the modules only for roles with that scope, so it has deleted the ones it had created for this role, even if its selectors are already rewritten.
  • A selector of its aggregationRule field requires the aggregation label of a lineage that no built-in role collects, such as rbac.deckhouse.io/aggregate-to-all-as of the old scheme, the label of a subsystem the role model no longer has, or the label of a subsystem a module has left. The modules label their capabilities only with the lineages that the built-in roles collect, so such a selector matches none of the built-in capabilities either. A selector by a lineage that a custom capability carries is not counted, because it still gets the rules of that capability. A custom capability here is a ClusterRole without the heritage: deckhouse label and without an aggregationRule field. Neither is a selector by the lineage of the subsystem in the rbac.deckhouse.io/subsystem label of the role, because the custom capabilities of that subsystem carry it. Neither exception applies to the all, deckhouse, infrastructure, kubernetes and networking lineages, which do not count as custom ones. A compatibility alias of an old role, a ClusterRole with the rbac.deckhouse.io/deprecated: "true" label, does not count as a built-in role that collects a lineage, because it collects the lineage of an old subsystem only for the bindings to the old name.

  Repair the access in this order:

  1. Create a role of the new scheme next to this one. Give it the d8:custom: name prefix and the rbac.deckhouse.io/kind: custom-role label, and select the capabilities by the labels of the new scheme, without rbac.deckhouse.io/kind. Select the capabilities that a selector by a lineage no built-in role collects was meant for by their rbac.deckhouse.io/capability labels, or by rbac.deckhouse.io/scope: system and the module label of their module. If this role has the rbac.deckhouse.io/use-role label, keep the label on the new role and add rbac.deckhouse.io/scope: subsystem with rbac.deckhouse.io/subsystem: SUBSYSTEM, or rbac.deckhouse.io/scope: system, so that Deckhouse creates the RoleBinding objects in the namespaces of the modules again.
  2. Create capabilities of the new scheme in place of the custom capabilities of the old scheme that this role aggregates. The D8UserAuthzForeignAggregationLabel alert names them.
  3. For every RoleBinding and ClusterRoleBinding object to this role, create one with the same subjects to the new role. If this role has the rbac.deckhouse.io/aggregate-to-all-as: LEVEL label, what it extended depends on its kind label. With rbac.deckhouse.io/kind: manage, its permissions also reached the subjects of d8:manage:all:LEVEL. A custom role cannot be aggregated into d8:system:LEVEL, so bind the new role to those subjects with a ClusterRoleBinding. With rbac.deckhouse.io/kind: use, they reached the subjects of d8:use:role:LEVEL in every namespace, so move the permissions into a namespace capability with rbac.deckhouse.io/aggregate-to-namespace-as: LEVEL, which extends d8:namespace:LEVEL.
  4. Check the permissions of the subjects with d8 k auth can-i VERB RESOURCE --as=USER_NAME.
  5. Delete the old bindings, this role and the old capabilities.

  For the commands, refer to the corresponding section in the user-authz module FAQ.

  List all such roles:

  d8 k get clusterroles -o json | jq -r '
    def lineages: [((.matchLabels // {}) | keys[]), (.matchExpressions[]? | select(.operator == "In" or .operator == "Exists") | .key)]
      | map(capture("^rbac\\.deckhouse\\.io/aggregate-to-(?lineage.+)-as$").lineage);
    ([.items[] | select(.metadata.labels.heritage == "deckhouse" and .metadata.labels["rbac.deckhouse.io/deprecated"] != "true")
          | .aggregationRule.clusterRoleSelectors[]? | lineages[]] + ["system", "namespace", "project"]) as $collected
    | ([.items[] | select(.metadata.labels.heritage != "deckhouse" and .aggregationRule == null) | (.metadata.labels // {}) | keys[]
          | capture("^rbac\\.deckhouse\\.io/aggregate-to-(?lineage.+)-as$").lineage]) as $carried
    | .items[]
    | select(.metadata.labels.heritage != "deckhouse" and .aggregationRule != null)
    | (.metadata.labels["rbac.deckhouse.io/subsystem"] // "") as $own
    | select(
        (
          ((.metadata.labels["rbac.deckhouse.io/use-role"] // "") != "")
          and (((.metadata.labels["rbac.deckhouse.io/scope"] // "") | IN("system", "subsystem")) | not)
        )
        or (
          [
            .aggregationRule.clusterRoleSelectors[]?
            | (.matchLabels["rbac.deckhouse.io/kind"] // empty),
              (.matchExpressions[]? | select(.key == "rbac.deckhouse.io/kind" and .operator == "In") | .values[])
          ]
          | any(IN("manage", "use"))
        )
        or (
          [
            .aggregationRule.clusterRoleSelectors[]? | lineages[]
            | select((IN($collected[]) | not)
                and (IN("all", "deckhouse", "infrastructure", "kubernetes", "networking") or (. != $own and (IN($carried[]) | not))))
          ]
          | length > 0
        )
      )
    | .metadata.name'
  

• D8UserAuthzLegacyUseCapability CE S7
  ClusterRole NAME carries the use labels of the previous scheme, which the namespace and project roles do not collect.

  The ClusterRole NAMEXXX of the MODULE_NAME moduleXXX carries the rbac.deckhouse.io/kind: use and rbac.deckhouse.io/aggregate-to-XXX-as: XXX labelsXXX of the use roles that a module with a subsystem of its own shipped in the previous schemeXXX of a use capability of the previous schemeXXX. rbac.deckhouse.io/kind: use is the label of a use capability of the previous schemeXXX. XXXThe moduleXXXThe capabilityXXX is still on that scheme.

  XXXThe d8:namespace:* and d8:project:* roles do not collect this label. Only the deprecated d8:use:role:* aliases still collect it, and they will be removed in the next release. A RoleBinding moved from d8:use:role:XXX to d8:namespace:XXX and a project role therefore do not get the rules of this ClusterRole.XXXNo role collects a ClusterRole with rbac.deckhouse.io/kind: use and this label. The d8:system:* and d8:subsystem:* roles and the deprecated d8:manage:* aliases leave out rbac.deckhouse.io/kind: use, and the d8:namespace:*, d8:project:* and d8:use:role:* roles do not select this label. Only a binding to this ClusterRole itself gives its rules.XXX

  Choose one:

  • If the ClusterRole belongs to a module, update the module to a release that labels its namespace capabilities with rbac.deckhouse.io/kind: capability and rbac.deckhouse.io/aggregate-to-namespace-as. Until then, give the subjects that need these rules a RoleBinding to NAME in their namespace, next to their RoleBinding to the d8:namespace:* role of their level. Neither role is a deprecated alias, so these RoleBinding objects do not hold the next release. In a project namespace, the cluster-objects-grants-validator.multitenancy.deckhouse.io webhook admits a RoleBinding to NAME only if the ClusterRole has the rbac.deckhouse.io/delegatable: "true" labelXXX, which the use capabilities of the previous scheme do not haveXXX.XXX Where the webhook refuses it, keep the RoleBinding to the d8:use:role:* alias: it keeps these rules, but the D8UserAuthzDeprecatedRBACv2RoleInUse alert reports it, and the next release, which removes the aliases, stays pending until the module is updated and the RoleBinding is moved to the d8:namespace:* role.XXX Where the webhook refuses it, the subjects get these rules only once the module is updated.XXX
  • If it is a custom capability, replace it with d8:custom:namespace-capability:NAME with the same rules and the rbac.deckhouse.io/kind: custom-capability, rbac.deckhouse.io/scope: namespace and rbac.deckhouse.io/aggregate-to-namespace-as: XXX labels. For the commands, refer to the corresponding section in the user-authz module FAQ.

  List every such ClusterRole:

  d8 k get clusterroles -l rbac.deckhouse.io/kind=use -o json | jq -r '.items[] | select(.metadata.labels | keys | map(capture("^rbac\\.deckhouse\\.io/aggregate-to-(?lineage.+)-as$").lineage) | any(IN("kubernetes", "all", "system", "iam", "security", "cluster", "delivery", "network", "storage", "observability", "managed-services", "deckhouse", "infrastructure", "networking"))) | .metadata.name'
  

• D8UserAuthzPermissionBrowserUnavailable CE S6
  Permission Browser API server has unavailable replicas.

  Deckhouse has detected unavailable replicas in the permission-browser-apiserver deployment in the d8-user-authz namespace. Permission Browser requests can fail while this alert is active.

  Check deployment and pod states:

  d8 k -n d8-user-authz get deploy permission-browser-apiserver
  d8 k -n d8-user-authz get pods -l app=permission-browser-apiserver
  

• D8UserAuthzReleaseSlow CE S6
  The user-authz module release repeatedly takes more than 5 minutes.

  Applying the user-authz Helm release took more than 5 minutes on average over the last hour, and it was applied at least twice in that hour, so this is not the single slow pass of a cluster bootstrap or of the one-time migration to user-authz-controller. A release that does not finish within the module release timeout (20 minutes) fails, changes to ClusterAuthorizationRule and AuthorizationRule objects stop being applied and platform updates stall (the D8DeckhouseCouldNotRunModule alert fires in that case).

  The release itself no longer grows with the number of rules: bindings are managed by user-authz-controller, and custom ClusterRoles are aggregated. A slow release therefore points outside the module: an API server that answers slowly, an object stuck with a finalizer, a Deckhouse Pod short of CPU, or the release engine itself. Check what the release is waiting on:

  d8 s queue list
  d8 k -n d8-system logs deploy/deckhouse | grep '"module":"user-authz"'
  

  If the cluster has just been updated to the release that moved the bindings to the controller, one slow pass is expected and does not raise this alert by itself; if it fires anyway, the pass is being retried. It is proportional to the number of bindings and can exceed the timeout on clusters with more than 5000 of them; the module FAQ describes what to do in that case.

• D8UserAuthzRenderSlow CE S7
  Rendering the user-authz module templates repeatedly takes more than 30 seconds.

  Rendering the templates of the user-authz module took more than 30 seconds on average over the last 30 minutes, across at least three renders; normally one render takes about a second. Rendering is CPU work inside the Deckhouse Pod and does not depend on the number of ClusterAuthorizationRule or AuthorizationRule objects, so the cause is the node, not the module: most often the Deckhouse Pod is short of CPU (CPU steal on a virtual machine, a node running at its limits, a bootstrap that starts every module at once).

  A single slow render during a cluster bootstrap does not raise this alert by itself. If it fires, compare the module with the others and look at the node:

  topk(10, increase(deckhouse_helm_operation_seconds_sum{operation="template"}[30m]) / increase(deckhouse_helm_operation_seconds_count{operation="template"}[30m]))
  rate(node_cpu_seconds_total{mode="steal"}[5m])
  

  d8 k -n d8-system logs deploy/deckhouse | grep '"module":"user-authz"'
  

• D8UserAuthzRetiredAggregationLabel CE S7
  ClusterRole NAME aggregates by the label of the retired XXX subsystem.

  The ClusterRole NAMEXXX of the MODULE_NAME moduleXXX carries the rbac.deckhouse.io/aggregate-to-XXX-as label.

  The role model has no XXX subsystem any more. deckhouse, infrastructure and kubernetes were merged into cluster, and networking was renamed to network. No d8:system:* or d8:subsystem:* role collects this label, so the holders of these roles do not get the rules of this ClusterRole. Only the deprecated d8:manage:* aliases still collect it, and they will be removed in the next release.

  Choose one:

  • If the ClusterRole belongs to a module, update the module to a release that labels its capabilities with the lineage of its new subsystem. For most modules of the old subsystem that is rbac.deckhouse.io/aggregate-to-XXX-as. The subsystem of each module is listed in the user-authz documentation.
  • If it is a custom capability, replace the label with rbac.deckhouse.io/aggregate-to-XXX-as or with the label of the subsystem the capability belongs to now.

  Until then, a subject bound to a new role gets the rules of such a capability of a module only through a custom role. Create a d8:custom:system:NAME ClusterRole with the rbac.deckhouse.io/kind: custom-role and rbac.deckhouse.io/scope: system labels and bind it to the subjects with a ClusterRoleBinding. A capability of the previous scheme has no rbac.deckhouse.io/capability label, so the role selects the capabilities of the module by the module: XXXMODULE_NAMEXXXMODULEXXX and rbac.deckhouse.io/kind: manage labels. Such a selector takes the capabilities of every level, so add rbac.deckhouse.io/aggregate-to-XXX-as: viewer to it for a role that only views. The D8UserAuthzLegacyRBACv2CustomRoleFound alert names this role, because it selects by a label of the previous scheme. Once the module carries the new labels, select its capabilities by their rbac.deckhouse.io/capability label instead. A capability that already carries rbac.deckhouse.io/kind: capability is selected by its rbac.deckhouse.io/capability label right away.

  List every such ClusterRole:

  d8 k get clusterroles -o json | jq -r '.items[] | select((.metadata.labels["rbac.deckhouse.io/kind"] // "") != "use") | select((.metadata.labels // {}) | keys | any(IN("rbac.deckhouse.io/aggregate-to-deckhouse-as", "rbac.deckhouse.io/aggregate-to-infrastructure-as", "rbac.deckhouse.io/aggregate-to-kubernetes-as", "rbac.deckhouse.io/aggregate-to-networking-as"))) | .metadata.name'
  

• D8UserAuthzRevocationCheckNotSynced CE S6
  The revocation check of the user-authz webhook has not loaded the revocations.

  For more than 10 minutes the user-authz-webhook instance POD_NAME has not loaded the Revocation objects of the user-authn module (the objects in d8-user-authn that record revoked user credentials): it cannot list them, or cannot find out whether the API server serves them.

  Until it loads them, the instance has no opinion about revoked credentials: on the master node of this instance, kubectl with the token of a locked or logged-out user keeps working until the token expires. Other authorization decisions are not affected.

  Check the webhook’s access to the API and its logs:

  d8 k -n d8-user-authz logs POD_NAME -c webhook --tail=100 | grep -i revocation
  d8 k auth can-i list revocations.dex.coreos.com -n d8-user-authn --as=system:serviceaccount:d8-user-authz:webhook
  

• D8UserAuthzRevocationCheckStale CE S6
  The revocation check of the user-authz webhook no longer sees new revocations.

  For more than 15 minutes the user-authz-webhook instance POD_NAME has been unable to watch the Revocation objects of the user-authn module (the objects in d8-user-authn that record revoked user credentials).

  The instance keeps refusing the Dex tokens it already knows to be revoked, but a revocation made since its watch broke does not reach it: on the master node of this instance, kubectl with the token of a locked or logged-out user keeps working until the token expires.

  Check the webhook’s access to the API and its logs:

  d8 k -n d8-user-authz logs POD_NAME -c webhook --tail=100 | grep -i revocation
  d8 k auth can-i watch revocations.dex.coreos.com -n d8-user-authn --as=system:serviceaccount:d8-user-authz:webhook
  

• D8UserAuthzRuleNeedsMultiTenancy CE S6
  A ClusterAuthorizationRule asks for namespace limits that nothing enforces.

  The ClusterAuthorizationRule NAME sets XXX, but the enableMultiTenancy setting of the user-authz module is off, so the authorization webhook that enforces those options is not deployed.

  The rule is not partially applied. It is applied without its limits: the subjects named in it hold its access level in every namespace of the cluster, the system ones included.

  Either enable multi-tenancy, or remove the options from the rule:

  d8 k get clusterauthorizationrule NAME -o yaml
  d8 k get moduleconfig user-authz -o jsonpath='{.spec.settings.enableMultiTenancy}'
  

• D8UserAuthzRulePropagationLag CE S6
  One user-authz webhook instance stopped picking up rule changes.

  One instance of user-authz-webhook has not updated its rules for more than an hour while another instance has. The rules are changing in the cluster and this instance does not see the changes.

  Find the instance and check its log:

  d8 k -n d8-user-authz get pods -l app=user-authz-webhook -o wide
  d8 k -n d8-user-authz logs -l app=user-authz-webhook -c webhook --tail=100
  

  An instance whose watch keeps failing becomes not ready, so check the readiness of the Pod as well.

• D8UserAuthzRulesNeedMultiTenancy CE S6
  More ClusterAuthorizationRules ask for namespace limits than the per-rule alert names.

  VALUE ClusterAuthorizationRules set limitNamespaces, namespaceSelector or allowAccessToSystemNamespaces while the enableMultiTenancy setting of the user-authz module is off. D8UserAuthzRuleNeedsMultiTenancy names the first 50 of them; this alert accounts for the rest.

  None of those rules is partially applied: the subjects named in each of them hold its access level in every namespace of the cluster, the system ones included.

  Check whether multi-tenancy was meant to be enabled, and list the affected rules:

  d8 k get moduleconfig user-authz -o jsonpath='{.spec.settings.enableMultiTenancy}'
  d8 k get clusterauthorizationrule -o json | jq -r '.items[] | select((.spec.limitNamespaces // [] | length > 0) or (.spec.namespaceSelector != null) or (.spec.allowAccessToSystemNamespaces == true)) | .metadata.name'
  

• D8UserAuthzWebhookDirectoryDiverged CE S6
  The masters disagree about the multi-tenancy rules.

  Instances of user-authz-webhook have been using different sets of ClusterAuthorizationRules for 10 minutes.

  Each master node authorizes the requests that reach it, so the same request is answered differently depending on the node: a user can be inside their namespace limits on one node and outside them on another.

  Find the instance that is behind and look at why:

  d8 k -n d8-user-authz get pods -l app=user-authz-webhook -o wide
  d8 k -n d8-user-authz logs -l app=user-authz-webhook -c webhook --tail=100 | grep -i 'list/watch'
  

  A growing user_authz_webhook_rules_watch_errors_total on one instance points at its watch; a user_authz_webhook_rules_directory_updated_timestamp_seconds that has stopped moving points at rules that are no longer updated. Deleting the lagging Pod is the last resort: the DaemonSet recreates it, and on a cluster with a single master node every request that the authorization webhook has to answer is denied while the Pod restarts. Control plane components, kubelets, and the ServiceAccounts of the kube-system and d8-* namespaces keep working, because the webhook does not check them.

• D8UserAuthzWebhookRulesQuarantined CE S7
  A ClusterAuthorizationRule cannot be compiled by the webhook.

  For more than 10 minutes the user-authz-webhook has quarantined at least one ClusterAuthorizationRule: its limitNamespaces pattern or namespaceSelector does not compile, or the rule cannot be read at all.

  A rule with a broken filter is applied without that filter, so its subjects get a narrower scope than written and are denied where the filter was meant to allow. A rule that cannot be read is not applied at all, and its subjects are denied every namespace.

  Find the rule:

  d8 k get clusterauthorizationrules -o custom-columns=NAME:.metadata.name,LIMIT:.spec.limitNamespaces,SELECTOR:.spec.namespaceSelector
  d8 k -n d8-user-authz logs -l app=user-authz-webhook -c webhook --tail=100 | grep -i quarantin
  

• D8UserAuthzWebhookRulesWatchErrors CE S7
  The user-authz webhook cannot watch the ClusterAuthorizationRules.

  For more than 10 minutes the user-authz-webhook rules informer has been failing to watch the ClusterAuthorizationRules (a non-zero rate of watch errors). New or changed rules may not be reaching the webhook, so multi-tenancy decisions can lag behind what users configure.

  Check the webhook’s access to the API and its logs:

  d8 k -n d8-user-authz logs -l app=user-authz-webhook -c webhook --tail=100
  d8 k auth can-i watch clusterauthorizationrules.deckhouse.io --as=system:serviceaccount:d8-user-authz:webhook
  

• D8UserAuthzWebhookTargetDown CE S6
  Prometheus is unable to scrape the user-authz webhook.

  Prometheus cannot collect metrics from at least one instance of user-authz-webhook in the d8-user-authz namespace. One instance without metrics is enough for the alert to fire.

  If the instance is running and only its metrics are unreachable, it keeps answering authorization requests, but the other alerts of the module cannot see its state. If it is not running or has not read the rules yet, it answers 503 to every authorization request, and the API server denies the request, because the webhook is configured with failurePolicy: Deny.

  Check the DaemonSet, its metrics sidecar and the scrape target:

  d8 k -n d8-user-authz get pods -l app=user-authz-webhook -o wide
  d8 k -n d8-user-authz logs -l app=user-authz-webhook -c kube-rbac-proxy --tail=50
  

Module vertical-pod-autoscaler

• VerticalPodAutoscalerDeprecatedUpdateModeAuto CE S9
  VerticalPodAutoscaler objects use the deprecated Auto update mode.

  There are VALUE VerticalPodAutoscaler object(s) in the cluster configured with the deprecated update mode (spec.updatePolicy.updateMode: Auto).

  The Auto update mode is deprecated. It currently behaves like Recreate (the VPA updater evicts pods to apply new resource requests) and its semantics are reserved to change in future versions, which makes cluster behavior ambiguous and potentially disruptive.

  Set the affected VPA object(s) to one of the supported update modes (spec.updatePolicy.updateMode):

  • InPlaceOrRecreate: Apply recommendations in place when possible, fall back to recreation (recommended).
  • Recreate: Evict and recreate pods to apply recommendations.
  • Initial: Apply only when pods are (re)created.
  • Off: Recommendations only, no changes to running pods.

  To list the affected objects, open the VPA Deprecated Update Mode Grafana dashboard, or run:

  kube_verticalpodautoscaler_spec_updatepolicy_updatemode{update_mode="Auto"} == 1