Channel/version:

Reference
The list of alerts

Starting from version 1.76, Deckhouse Kubernetes Platform has been renamed to Deckhouse Platform.

The platform now offers more features, and editions have changed. We are already updating the documentation, but if you have any questions, please contact your account manager or technical support.

The list of alerts

The page displays a list of all alerts of monitoring in the Deckhouse Platform.

The list does not contain alerts of modules from a source.

Alerts are grouped by modules. To the right of the alert name, there are icons indicating the minimum DP edition in which the alert is available, and the alert severity.

For each alert, a summary is provided, and if available, the detailed alert description can be viewed by expanding it.

Alert severity

Alert descriptions contain the Severity (S) parameter, which indicates the level of criticality. Its value varies from S1 to S9 and can be interpreted as follows:

S1 — maximum level, critical failure/crash (immediate action required);
S2 — high level, close to maximum, possible accident (rapid response required);
S3 — medium level, potentially serious problem (verification required);
S4-S9 — low level. There is a problem, but overall performance is not impaired.

Module admission-policy-engine

D8AdmissionPolicyEngineAuditUnavailable CE S4
The audit controller of admission-policy-engine has no available replicas.
The gatekeeper-audit Deployment in the d8-admission-policy-engine namespace has no available replicas.

Admission of new objects is not affected, but the module stops reporting and stops picking up new policies:
- Violations among the objects that already run are not collected, so the PodSecurityStandardsViolation, OperationPolicyViolation and SecurityPolicyViolation alerts go silent instead of reporting that all is well.
- The list of resources the webhook intercepts is not updated, so a policy added while the audit is down is not enforced on admission.
Find out why the replica is gone:
d8 k -n d8-admission-policy-engine get pods -l app=gatekeeper,control-plane=audit-controller d8 k -n d8-admission-policy-engine describe pods -l app=gatekeeper,control-plane=audit-controller d8 k -n d8-admission-policy-engine logs deploy/gatekeeper-audit -c manager --tail=200 d8 k -n d8-admission-policy-engine logs deploy/gatekeeper-audit -c constraint-exporter --tail=200
D8AdmissionPolicyEngineNotBootstrapped CE S7
Admission-policy-engine module hasn't been bootstrapped for 10 minutes.
The admission-policy-engine module couldn’t bootstrap.

Steps to troubleshoot:
1. Verify that the module’s components are up and running:
  
  d8 k get pods -n d8-admission-policy-engine
2. Check logs for issues, such as missing constraint templates or incomplete CRD creation:
  
  d8 k logs -n d8-system -lapp=deckhouse --tail=1000 | grep admission-policy-engine
D8AdmissionPolicyEngineWebhookDegraded CE S3
The admission webhook of admission-policy-engine has unavailable replicas.
The gatekeeper-controller-manager Deployment in the d8-admission-policy-engine namespace has unavailable replicas, and the remaining ones still serve admission requests.

The webhook has no redundancy left. Losing the last replica makes the API server reject every request the webhook intercepts, because the webhook has failurePolicy: Fail.

A rollout of the Deployment raises this alert for a short time and needs no action. If the alert holds, find out why the replica does not start:
d8 k -n d8-admission-policy-engine get pods -l app=gatekeeper,control-plane=controller-manager -o wide d8 k -n d8-admission-policy-engine describe pods -l app=gatekeeper,control-plane=controller-manager d8 k -n d8-admission-policy-engine get events --sort-by=.lastTimestamp | tail -30
D8AdmissionPolicyEngineWebhookNotServing CE S2
The API server cannot call the admission webhook of admission-policy-engine.
The API server is rejecting requests because it fails to call a webhook of the admission-policy-engine module. The webhook has failurePolicy: Fail, so the cluster is blocked for everything the webhook intercepts.

The alert is independent of the replica counters: it fires while the pods are Ready as well. A replica that is running but unreachable to the API server produces the same outage, and only this alert reports it. Typical causes are a gatekeeper-webhook-server-cert that no longer matches the caBundle of the ValidatingWebhookConfiguration, an empty gatekeeper-webhook-service endpoint list, and a network policy that blocks the API server.

The API server counts a webhook that exceeds its timeoutSeconds of 15 the same way it counts one it cannot reach, so a few slow evaluations of a large object raise the metric as well. They do not raise the alert: for: 15m requires the failures to continue, while a burst leaves the rate elevated for little more than its own duration plus the five-minute window.

Read the reason the API server reports, then check the certificate and the endpoints:
d8 k -n d8-admission-policy-engine logs deploy/gatekeeper-controller-manager -c manager --all-pods=true --tail=200 d8 k -n d8-admission-policy-engine get endpoints gatekeeper-webhook-service d8 k get validatingwebhookconfiguration d8-admission-policy-engine-config -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d | openssl x509 -noout -subject -dates d8 k -n d8-admission-policy-engine get secret gatekeeper-webhook-server-cert -o jsonpath='{.data.ca\.crt}' | base64 -d | openssl x509 -noout -subject -dates
The two certificates must match. Deckhouse Platform reissues them through the module’s hook, so deleting the secret makes it regenerate the pair.
D8AdmissionPolicyEngineWebhookUnavailable CE S2
The admission webhook of admission-policy-engine has no available replicas.
The gatekeeper-controller-manager Deployment in the d8-admission-policy-engine namespace has no available replicas.

The Deployment serves the admission webhook of the module, and the webhook has failurePolicy: Fail. While no replica is available, the API server rejects every request the webhook intercepts, cluster-wide:
- Pods and the controllers that create them are not created or updated.
- Deletion of those resources is rejected as well.
- Roles, RoleBindings and Gatekeeper constraints are not applied.
- The d8 k exec, d8 k attach and d8 k debug requests intercepted by the webhooks are rejected. This includes d8 k exec and d8 k attach in the d8-* and kube-* namespaces.
Find out why the replicas are gone:
d8 k -n d8-admission-policy-engine get deploy gatekeeper-controller-manager d8 k -n d8-admission-policy-engine get pods -l app=gatekeeper,control-plane=controller-manager -o wide d8 k -n d8-admission-policy-engine describe pods -l app=gatekeeper,control-plane=controller-manager d8 k -n d8-admission-policy-engine logs deploy/gatekeeper-controller-manager -c manager --all-pods=true --tail=200
If a pod stays Pending, look at the events of the namespace and at the nodes the replicas can run on:
d8 k -n d8-admission-policy-engine get events --sort-by=.lastTimestamp | tail -30 d8 k get nodes -l node-role.deckhouse.io/system d8 k get nodes -l node-role.kubernetes.io/control-plane
OperationPolicyViolation CE S3
At least one object violates the configured cluster operation policies.
You have configured operation policies for the cluster, and one or more existing objects are violating these policies.

To identify violating objects:
- Run the following Prometheus query:
  
  count by (violating_namespace, violating_kind, violating_name, violation_msg) ( d8_gatekeeper_exporter_constraint_violations{ violation_enforcement="deny", source_type="OperationPolicy" } )
- Alternatively, check the admission-policy-engine Grafana dashboard.
PodSecurityStandardsDefaultPolicyNotSet CE S5
The default Pod Security Standards policy must be set explicitly.
The default policy isn’t set in the admission-policy-engine ModuleConfig, so Deckhouse derives it from the version the cluster was bootstrapped with and applies Privileged to non-system namespaces. This happens when the cluster was bootstrapped with a Deckhouse version lower than v1.55, or when the bootstrap version can’t be determined: the d8-system/install-data ConfigMap is missing or can’t be read, or its version field is empty or isn’t a valid version.

The update to Deckhouse 1.78 is blocked until you set the default policy explicitly. Starting from 1.78, Baseline is the default policy for all clusters, so without an explicit value the cluster would switch from Privileged to Baseline on update.

To set the default policy, add the settings.podSecurityStandards.defaultPolicy parameter to the admission-policy-engine ModuleConfig.

If the ModuleConfig already exists, patch it so that its other settings are preserved:
d8 k patch mc admission-policy-engine --type=merge -p '{"spec":{"settings":{"podSecurityStandards":{"defaultPolicy":"Privileged"}}}}'
If the ModuleConfig doesn’t exist, create it:
apiVersion: deckhouse.io/v1alpha1 kind: ModuleConfig metadata: name: admission-policy-engine spec: version: 1 settings: podSecurityStandards: defaultPolicy: Privileged
Choose the value that suits your cluster: Privileged, Baseline, or Restricted. Before switching to Baseline or Restricted, make sure that the workloads in non-system namespaces comply with the chosen policy. Otherwise, their pods are denied when recreated if the enforcementAction parameter is set to Deny.

The alert is resolved within 15 minutes after the parameter is set.
PodSecurityStandardsViolation CE S3
At least one pod violates the configured cluster pod security standards.
You have configured Pod Security Standards, and one or more running pods are violating these standards.

To identify violating pods:
- Run the following Prometheus query:
  
  count by (violating_namespace, violating_name, violation_msg) ( d8_gatekeeper_exporter_constraint_violations{ violation_enforcement="deny", violating_namespace=~".*", violating_kind="Pod", source_type="PSS" } )
- Alternatively, check the admission-policy-engine Grafana dashboard.
SecurityPolicyViolation CE S3
At least one object violates the configured cluster security policies.
You have configured security policies for the cluster, and one or more existing objects are violating these policies.

To identify violating objects:
- Run the following Prometheus query:
  
  count by (violating_namespace, violating_kind, violating_name, violation_msg) ( d8_gatekeeper_exporter_constraint_violations{ violation_enforcement="deny", source_type="SecurityPolicy" } )
- Alternatively, check the admission-policy-engine Grafana dashboard.

Module cert-manager

CertmanagerCertificateChallengePending CE S4
ACME Challenge for domain "XXX" is pending for more than 1 hour.
The ACME Challenge NAMESPACE/NAME for domain XXX has been in pending state for more than 1 hour and requires investigation.

Common causes:
- DNS misconfiguration (for DNS-01 challenges).
- Ingress or firewall issues blocking HTTP-01 validation.
- ACME provider rate limiting.
- Network connectivity issues.
Resolution steps:
1. Check the Challenge status and reason:
  d8 k -n NAMESPACE describe challenge NAME
2. Check cert-manager logs for errors:
  d8 k -n d8-cert-manager logs -l app=cert-manager -c cert-manager --tail=100
3. If the challenge is stuck, delete the Order to trigger a retry:
  d8 k -n NAMESPACE get orders d8 k -n NAMESPACE delete order ORDER_NAME
CertmanagerCertificateChallengePending CE S6
ACME Challenge for domain "XXX" is pending for more than 10 minutes.
The ACME Challenge NAMESPACE/NAME for domain XXX has been in pending state for more than 10 minutes.

This may indicate issues with the ACME challenge validation process.

Common causes:
- DNS propagation delays (for DNS-01 challenges).
- Ingress or firewall issues blocking HTTP-01 validation.
- ACME provider rate limiting or temporary unavailability.
Diagnostic steps:
1. Check the Challenge status:
  d8 k -n NAMESPACE describe challenge NAME
2. Check cert-manager logs for errors:
  d8 k -n d8-cert-manager logs -l app=cert-manager -c cert-manager --tail=100
CertmanagerCertificateChallengeStuck CE S4
ACME Challenge for domain "XXX" is stuck in "STATUS_REFERENCE" state.
The ACME Challenge NAMESPACE/NAME for domain XXX is in STATUS_REFERENCE state and requires manual intervention.

Reason: XXX

The certificate will not be issued or renewed until the issue is resolved.

Common causes:
- DNS misconfiguration (for DNS-01 challenges).
- Ingress or firewall issues (for HTTP-01 challenges).
- Rate limiting by the ACME provider.
- Authorization expired before challenge completion.
Resolution steps:
1. Check the Challenge status and reason:
  d8 k -n NAMESPACE describe challenge NAME
2. Delete the failed Order to trigger a new certificate request:
  d8 k -n NAMESPACE get orders d8 k -n NAMESPACE delete order ORDER_NAME
CertmanagerCertificateExpired CE S4
Certificate NAMESPACE/NAME is not provided.
Certificate is not provided.

To check the certificate details, run the following command:
d8 k -n NAMESPACE describe certificate NAME
CertmanagerCertificateExpiredSoon CE S4
Certificate will expire soon.
The certificate NAMESPACE/NAME will expire in less than two weeks.

To check the certificate details, run the following command:
d8 k -n NAMESPACE describe certificate CERTIFICATE-NAME
CertmanagerCertificateOrderErrors CE S5
Cert-manager couldn't order a certificate.
Cert-manager received responses with the status code STATUS_REFERENCE when requesting SCHEME://HOST_NAMEPATH.

This can affect certificate ordering and prolongation in the future. For details, check the cert-manager logs using the following command:
d8 k -n d8-cert-manager logs -l app=cert-manager -c cert-manager

Module chrony

NodeTimeOutOfSync CE S5
Clock on the node NODE_NAME is drifting.
Time on the node NODE_NAME is out of sync and drifts apart from the NTP server clock by VALUE seconds.

To resolve the time synchronization issues:
- Fix network errors:
  
  Ensure the upstream time synchronization servers defined in the chrony configuration are available.
  
  Eliminate large packet loss and excessive latency to upstream time synchronization servers.
- Modify the NTP servers list defined in the chrony configuration.
NTPDaemonOnNodeDoesNotSynchronizeTime CE S5
NTP daemon on the node NODE_NAME haven't synchronized time for too long.
Steps to troubleshoot:
1. Check if the chrony pod is running on the node by executing the following command:
  
  d8 k -n d8-chrony --field-selector spec.nodeName="NODE_NAME" get pods
2. Verify the chrony daemon’s status by executing the following command:
  
  d8 k -n d8-chrony exec POD_NAME -- /opt/chrony-static/bin/chronyc sources
3. Resolve the time synchronization issues:
  
  Fix network errors:
  
  Ensure the upstream time synchronization servers defined in the chrony configuration are available.
  
  Eliminate large packet loss and excessive latency to upstream time synchronization servers.
  
  Modify the NTP servers list defined in the chrony configuration.

Module cloud-provider-dvp

D8CloudProviderDVPMigrationPending CE S6
The cloud-provider-dvp module is waiting for migration resources to be applied.
The cloud-provider-dvp module has detected the legacy DVPProviderClusterConfiguration secret in the cluster and is waiting for the new migration resources (ModuleConfig v2, CredentialSecret, NodeGroups, InstanceClasses) to be applied before completing the migration.

To complete the migration, apply the resources from the d8-migration-resources Secret:
d8 k -n d8-cloud-provider-dvp get secret d8-migration-resources -o jsonpath='{.data.resources\.yaml}' | base64 -d | d8 k apply -f -
After applying the resources, the alert will disappear automatically once the module detects that all required resources are in place.

Module cloud-provider-huaweicloud

D8HuaweiCloudMachineStuckInDeleting EE S6
HuaweiCloudMachine NAME is stuck in Deleting for more than 24h.
The HuaweiCloudMachine NAME in namespace NAMESPACE has been stuck in Deleting state for more than 24 hours.

This usually means the deletion job in HuaweiCloud is failing or the API is unavailable.

To inspect the machine status, run:
d8 k get huaweicloudmachine -n NAMESPACE NAME -o yaml
Check the controller logs for errors:
d8 k logs -n d8-cloud-provider-huaweicloud -l app=caphc-controller-manager

Module cloud-provider-yandex

D8YandexNatInstanceConnectionsQuotaUtilization CE S4

Connection quota utilization of the Yandex NAT instance exceeds 85% over the last 5 minutes.

The connection quota for the Yandex NAT instance has exceeded 85% utilization over the past 5 minutes.

To prevent potential issues, contact Yandex technical support and request an increase in the connection quota.

NATInstanceWithDeprecatedAvailabilityZone CE S9

NAT instance NAME is in a deprecated availability zone.

The NAT instance NAME is located in the availability zone ru-central1-c, which has been deprecated by Yandex Cloud. To resolve this issue, migrate the NAT instance to either ru-central1-a or ru-central1-b by following the instructions below.

The migration process involves irreversible changes and may result in a significant downtime. The duration typically depends on Yandex Cloud’s response time and can last several tens of minutes.

Migrate the NAT instance. To get providerClusterConfiguration.withNATInstance, run the following command:
```
d8 k -n d8-system exec -ti svc/deckhouse-leader -c deckhouse -- deckhouse-controller module values -g cloud-provider-yandex -o json | jq -c | jq '.cloudProviderYandex.internal.providerClusterConfiguration.withNATInstance'
```
- If you specified withNATInstance.natInstanceInternalAddress and/or withNATInstance.internalSubnetID in providerClusterConfiguration, remove them using the following command:
```
d8 system edit provider-cluster-configuration
```
- If you specified withNATInstance.externalSubnetID and/or withNATInstance.natInstanceExternalAddress in providerClusterConfiguration, change them to the appropriate values.
  
  To get the address and subnet ID, use the Yandex Cloud Console or CLI.
  
  To change withNATInstance.externalSubnetID and withNATInstance.natInstanceExternalAddress, run the following command:
```
d8 system edit provider-cluster-configuration
```

Run the appropriate edition and version of the Deckhouse installer container on the local machine. You may have to change the container registry address to do that. After that, perform the converge.

Get the appropriate edition and version of Deckhouse:

DH_VERSION=$(d8 k -n d8-system get deployment deckhouse -o jsonpath='{.metadata.annotations.core\.deckhouse\.io\/version}')
DH_EDITION=$(d8 k -n d8-system get deployment deckhouse -o jsonpath='{.metadata.annotations.core\.deckhouse\.io\/edition}' | tr '[:upper:]' '[:lower:]')
echo "DH_VERSION=$DH_VERSION DH_EDITION=$DH_EDITION"

Run the installer:

docker run --pull=always -it -v "$HOME/.ssh/:/tmp/.ssh/" registry.deckhouse.io/deckhouse/${DH_EDITION}/install:${DH_VERSION} bash

Perform the converge:

dhctl converge --ssh-agent-private-keys=/tmp/.ssh/SSH_KEY_FILENAME --ssh-user=USERNAME --ssh-host MASTER-NODE-0-HOST

Update the route table.

Get the route table name:

d8 k -n d8-system exec -ti svc/deckhouse-leader -c deckhouse -- deckhouse-controller module values -g cloud-provider-yandex -o json | jq -c | jq '.global.clusterConfiguration.cloud.prefix'

Get the NAT instance name:

d8 k -n d8-system exec -ti svc/deckhouse-leader -c deckhouse -- deckhouse-controller module values -g cloud-provider-yandex -o json | jq -c | jq '.cloudProviderYandex.internal.providerDiscoveryData.natInstanceName'

Get the NAT instance internal IP address:

yc compute instance list | grep -e "INTERNAL IP" -e NAT_INSTANCE_NAME_FROM_PREVIOUS_STEP

Update the route:

yc vpc route-table update --name ROUTE_TABLE_NAME_FROM_PREVIOUS_STEP --route "destination=0.0.0.0/0,next-hop=NAT_INSTANCE_INTERNAL_IP_FROM_PREVIOUS_STEP"

NodeGroupNodeWithDeprecatedAvailabilityZone CE S9
NodeGroup NODE_GROUP_NAME contains nodes in a deprecated availability zone.
Certain nodes in the node group NODE_GROUP_NAME are located in the availability zone ru-central1-c, which has been deprecated by Yandex Cloud.

Steps to troubleshoot:
1. Identify the nodes that need to be migrated by running the following command:
  
  d8 k get node -l "topology.kubernetes.io/zone=ru-central1-c"
2. Migrate your nodes, disks, and load balancers to one of the supported zones: ru-central1-a, ru-central1-b, or ru-central1-d. Refer to the Yandex migration guide for detailed instructions.
  
  You can’t migrate public IP addresses between zones. For details, refer to the migration guide.

Module cni-cilium

CiliumAgentEndpointsNotReady CE S4
Over 50% of all known endpoints aren't ready in agent NAMESPACE/POD_NAME.
For details, refer to the logs of the agent:
d8 k -n NAMESPACE logs POD_NAME
CiliumAgentMapPressureCritical CE S4

eBPF map MAP_NAME exceeds 90% utilization in agent NAMESPACE/POD_NAME.

The eBPF map resource utilization limit has almost been reached.

Check with the vendor for potential remediation steps.
CiliumAgentMetricNotFound CE S4
Agent NAMESPACE/POD_NAME isn't sending some metrics.
Steps to troubleshoot:
1. Check the logs of the agent:
  
  d8 k -n NAMESPACE logs POD_NAME
2. Verify the agent’s health status:
  
  d8 k -n NAMESPACE exec -ti POD_NAME cilium-health status
3. Compare the metrics with those of a neighboring agent.
Note that the absence of metrics can indirectly indicate that new pods can’t be created on the node due to connectivity issues with the agent.
CiliumAgentPolicyImportErrors CE S4
Agent NAMESPACE/POD_NAME fails to import policies.
For details, refer to the logs of the agent:
d8 k -n NAMESPACE logs POD_NAME
CiliumAgentUnreachableHealthEndpoints CE S4
Agent NAMESPACE/POD_NAME can't reach some of the node health endpoints.
For details, query Cilium health status API:
d8 k -n NAMESPACE exec POD_NAME -- cilium-health status
You can also refer to the agent logs:
d8 k -n NAMESPACE logs POD_NAME
CniCiliumNonStandardVXLANPortFound CE S4

Cilium configuration uses a non-standard VXLAN port.

The Cilium configuration specifies a non-standard VXLAN port XXX. The recommended port number is XXX.

To resolve this issue, update the tunnel-port parameter in the cilium-configmap ConfigMap located in the d8-cni-cilium namespace to match the recommended port.

If you configured the non-standard port on purpose, ignore this alert.
CniCiliumOrphanEgressGatewayPolicyFound SE-PLUS S4

Orphaned EgressGatewayPolicy with an irrelevant EgressGateway name has been found.

The cluster contains an orphaned EgressGatewayPolicy named NAME with an irrelevant EgressGateway name.

To resolve this issue, verify the EgressGateway name specified in the EgressGatewayPolicy resource EGRESSGATEWAY and update it as needed.
CNICiliumPolicyWithRequiresFieldsFound CE S4
The XXX XXXNAMESPACE/XXXNAME uses the fromRequires or toRequires fields.
The XXX NAMEXXX in the NAMESPACE namespaceXXX uses the fromRequires or toRequires fields.

Cilium 1.19 and later doesn’t support these fields. The Cilium agent of these versions can’t read such policies and fails on all nodes. Deckhouse won’t deploy a release with such a Cilium version until all these policies are rewritten.

To resolve this issue, rewrite each fromRequires constraint as an ingressDeny rule with an inverted selector. Don’t move the selector into the deny rule as is: this inverts the policy. A deny rule wins over allow rules of any policy, so it keeps the original scope: fromRequires applied to all policies selecting the endpoint.

For example, fromRequires with matchLabels: {env: prod} becomes:
ingressDeny: - fromEndpoints: - matchExpressions: - {key: env, operator: NotIn, values: [prod]} XXX - matchExpressions: - {key: k8s:io.kubernetes.pod.namespace, operator: Exists} - {key: k8s:io.kubernetes.pod.namespace, operator: NotIn, values: [NAMESPACE]} XXX
- Add a separate NotIn selector for each label of the constraint. XXX
- Keep the second selector: in a CiliumNetworkPolicy, fromRequires also implied the policy namespace, so sources from other namespaces were rejected. The Exists condition is required, otherwise the rule also blocks the host, the world, and other sources without a namespace. XXX
- Rewrite toRequires the same way, as an egressDeny rule with toEndpoints.
- Add the deny rule first, check the traffic, and only then remove fromRequires and toRequires.
To list all affected policies, use the d8_cni_cilium_policy_with_requires_fields == 1 expression in Prometheus.
D8IPv4OptionsGuardAuditMode CE S9
The CVE-2026-98432 filter only counts packets on VALUE nodes of the cluster.
The ipv4OptionsGuard.mode of the CNI module is Audit on VALUE nodes: packets with IP options are only counted, not dropped, and MPTCP stays enabled. The nodes remain exposed to CVE-2026-98432.

Clusters that existed before the mitigation get Audit by default. Once the D8IPv4OptionsGuardMatchedTraffic alert shows that no workload relies on IP options, switch to Enforce in the ModuleConfig of the CNI module (cni-cilium, cni-flannel or cni-simple-bridge):
spec: settings: ipv4OptionsGuard: mode: Enforce
D8IPv4OptionsGuardBackendMismatch CE S6
The CVE-2026-98432 filter on the node NODE_NAME uses a different iptables mode than iptables-wrapper selected.
The node NODE_NAME is protected, but the filter uses XXX: the kernel modules of the mode selected by iptables-wrapper are not loaded, while those of the other mode are.

The details are in the logs of the ipv4-options-guard container of the CNI agent pod on that node.

The filter and the other Deckhouse components use different packet filtering frameworks on this node. Such a security configuration is harder to audit and troubleshoot.

Possible actions:
- Find out why the kernel modules of the mode selected by iptables-wrapper are not loaded on the node.
- If loading these kernel modules is acceptable, set ipv4OptionsGuard.allowKernelModuleLoad: true in the CNI module settings.
D8IPv4OptionsGuardFilterBypassed CE S6
Packets can skip the CVE-2026-98432 filter in VALUE network namespaces on the node NODE_NAME.
The node NODE_NAME uses legacy iptables. In VALUE network namespaces, a rule in the PREROUTING chain of the raw table ahead of the jump to D8-IPV4OPT-GUARD can end the traversal of the table (for example, -j ACCEPT). Packets accepted by that rule never reach the filter.

The guard puts its jump first when it installs the filter, but does not move it later, so as not to fight with other software on the node that manages its own rules. The affected network namespaces are named in the logs of the ipv4-options-guard container of the CNI agent pod on that node.

Possible actions:
- Find the software that inserted the rule (iptables-legacy -t raw -S PREROUTING in the network namespace) and make it append the rule instead, or use a target that does not end the traversal.
- Or put the jump first by hand: iptables-legacy -t raw -I PREROUTING 1 -j D8-IPV4OPT-GUARD. The guard removes the old jump on its next pass.
D8IPv4OptionsGuardMatchedTraffic CE S7

IPv4 packets with IP options were seen on the node NODE_NAME in the Audit mode.

Over the last hour, the ipv4-options-guard sidecar matched VALUE IPv4 packets with IP options (rule XXX) on the node NODE_NAME.

In the Audit mode they are only counted. In the Enforce mode they would be dropped, and this alert is not raised.

Legitimate TCP traffic almost never carries IP options. If a workload relies on them (for example, CIPSO labeled networking), exclude the node with the ipv4-options-guard.deckhouse.io/disabled: "true" label before switching ipv4OptionsGuard.mode to Enforce. Otherwise, this may be an attempt to exploit CVE-2026-98432.
D8IPv4OptionsGuardMetricsAbsent CE S4

The CVE-2026-98432 filter state is unknown on the node NODE_NAME.

The CNI agent pod runs on the node NODE_NAME, but Prometheus gets no metrics from its ipv4-options-guard container.

Without these metrics, the other D8IPv4OptionsGuard* alerts cannot fire for this node, so it may be unprotected unnoticed.

Check the ipv4-options-guard container of the CNI agent pod on that node and the scraping of its metrics.
D8IPv4OptionsGuardNodeUnprotected CE S4
The CVE-2026-98432 filter cannot be installed on the node NODE_NAME.
The ipv4-options-guard sidecar of the CNI agent could not protect the node NODE_NAME (reason: XXX).
- iptables_wrapper_failed — iptables-wrapper could not determine the iptables mode of the node.
- kernel_modules_not_loaded — the kernel modules of the mode selected by iptables-wrapper are not loaded, and loading them is not allowed.
The details are in the logs of the ipv4-options-guard container of the CNI agent pod on that node.

The node remains exposed to CVE-2026-98432 until its kernel is updated.

Possible actions:
- Update the node kernel to a version with the fix and reboot the node.
- If the node uses legacy iptables and loading the kernel modules named in the reason is acceptable, set ipv4OptionsGuard.allowKernelModuleLoad: true in the CNI module settings.
D8IPv4OptionsGuardReconcileFailed CE S6

The CVE-2026-98432 filter failed in VALUE network namespaces on the node NODE_NAME.

Some network namespaces on the node NODE_NAME are not protected.

Check the logs of the ipv4-options-guard container of the CNI agent pod on that node.
D8NodeHasUnmetKernelRequirements CE S4
Nodes have unmet kernel requirements.
Deckhouse has detected that some nodes don’t meet the required kernel constraints. As a result, certain modules can’t run on these nodes.

Kernel requirements for each module:
- Cilium module: Kernel version must be >= 5.8.
To list all affected nodes, use the d8_node_kernel_does_not_satisfy_requirements == 1 expression in Prometheus.

Module cni-flannel

D8IPv4OptionsGuardAuditMode CE S9
The CVE-2026-98432 filter only counts packets on VALUE nodes of the cluster.
The ipv4OptionsGuard.mode of the CNI module is Audit on VALUE nodes: packets with IP options are only counted, not dropped, and MPTCP stays enabled. The nodes remain exposed to CVE-2026-98432.

Clusters that existed before the mitigation get Audit by default. Once the D8IPv4OptionsGuardMatchedTraffic alert shows that no workload relies on IP options, switch to Enforce in the ModuleConfig of the CNI module (cni-cilium, cni-flannel or cni-simple-bridge):
spec: settings: ipv4OptionsGuard: mode: Enforce
D8IPv4OptionsGuardBackendMismatch CE S6
The CVE-2026-98432 filter on the node NODE_NAME uses a different iptables mode than iptables-wrapper selected.
The node NODE_NAME is protected, but the filter uses XXX: the kernel modules of the mode selected by iptables-wrapper are not loaded, while those of the other mode are.

The details are in the logs of the ipv4-options-guard container of the CNI agent pod on that node.

The filter and the other Deckhouse components use different packet filtering frameworks on this node. Such a security configuration is harder to audit and troubleshoot.

Possible actions:
- Find out why the kernel modules of the mode selected by iptables-wrapper are not loaded on the node.
- If loading these kernel modules is acceptable, set ipv4OptionsGuard.allowKernelModuleLoad: true in the CNI module settings.
D8IPv4OptionsGuardFilterBypassed CE S6
Packets can skip the CVE-2026-98432 filter in VALUE network namespaces on the node NODE_NAME.
The node NODE_NAME uses legacy iptables. In VALUE network namespaces, a rule in the PREROUTING chain of the raw table ahead of the jump to D8-IPV4OPT-GUARD can end the traversal of the table (for example, -j ACCEPT). Packets accepted by that rule never reach the filter.

The guard puts its jump first when it installs the filter, but does not move it later, so as not to fight with other software on the node that manages its own rules. The affected network namespaces are named in the logs of the ipv4-options-guard container of the CNI agent pod on that node.

Possible actions:
- Find the software that inserted the rule (iptables-legacy -t raw -S PREROUTING in the network namespace) and make it append the rule instead, or use a target that does not end the traversal.
- Or put the jump first by hand: iptables-legacy -t raw -I PREROUTING 1 -j D8-IPV4OPT-GUARD. The guard removes the old jump on its next pass.
D8IPv4OptionsGuardMatchedTraffic CE S7

IPv4 packets with IP options were seen on the node NODE_NAME in the Audit mode.

Over the last hour, the ipv4-options-guard sidecar matched VALUE IPv4 packets with IP options (rule XXX) on the node NODE_NAME.

In the Audit mode they are only counted. In the Enforce mode they would be dropped, and this alert is not raised.

Legitimate TCP traffic almost never carries IP options. If a workload relies on them (for example, CIPSO labeled networking), exclude the node with the ipv4-options-guard.deckhouse.io/disabled: "true" label before switching ipv4OptionsGuard.mode to Enforce. Otherwise, this may be an attempt to exploit CVE-2026-98432.
D8IPv4OptionsGuardMetricsAbsent CE S4

The CVE-2026-98432 filter state is unknown on the node NODE_NAME.

The CNI agent pod runs on the node NODE_NAME, but Prometheus gets no metrics from its ipv4-options-guard container.

Without these metrics, the other D8IPv4OptionsGuard* alerts cannot fire for this node, so it may be unprotected unnoticed.

Check the ipv4-options-guard container of the CNI agent pod on that node and the scraping of its metrics.
D8IPv4OptionsGuardNodeUnprotected CE S4
The CVE-2026-98432 filter cannot be installed on the node NODE_NAME.
The ipv4-options-guard sidecar of the CNI agent could not protect the node NODE_NAME (reason: XXX).
- iptables_wrapper_failed — iptables-wrapper could not determine the iptables mode of the node.
- kernel_modules_not_loaded — the kernel modules of the mode selected by iptables-wrapper are not loaded, and loading them is not allowed.
The details are in the logs of the ipv4-options-guard container of the CNI agent pod on that node.

The node remains exposed to CVE-2026-98432 until its kernel is updated.

Possible actions:
- Update the node kernel to a version with the fix and reboot the node.
- If the node uses legacy iptables and loading the kernel modules named in the reason is acceptable, set ipv4OptionsGuard.allowKernelModuleLoad: true in the CNI module settings.
D8IPv4OptionsGuardReconcileFailed CE S6

The CVE-2026-98432 filter failed in VALUE network namespaces on the node NODE_NAME.

Some network namespaces on the node NODE_NAME are not protected.

Check the logs of the ipv4-options-guard container of the CNI agent pod on that node.

Module cni-simple-bridge

D8IPv4OptionsGuardAuditMode CE S9
The CVE-2026-98432 filter only counts packets on VALUE nodes of the cluster.
The ipv4OptionsGuard.mode of the CNI module is Audit on VALUE nodes: packets with IP options are only counted, not dropped, and MPTCP stays enabled. The nodes remain exposed to CVE-2026-98432.

Clusters that existed before the mitigation get Audit by default. Once the D8IPv4OptionsGuardMatchedTraffic alert shows that no workload relies on IP options, switch to Enforce in the ModuleConfig of the CNI module (cni-cilium, cni-flannel or cni-simple-bridge):
spec: settings: ipv4OptionsGuard: mode: Enforce
D8IPv4OptionsGuardBackendMismatch CE S6
The CVE-2026-98432 filter on the node NODE_NAME uses a different iptables mode than iptables-wrapper selected.
The node NODE_NAME is protected, but the filter uses XXX: the kernel modules of the mode selected by iptables-wrapper are not loaded, while those of the other mode are.

The details are in the logs of the ipv4-options-guard container of the CNI agent pod on that node.

The filter and the other Deckhouse components use different packet filtering frameworks on this node. Such a security configuration is harder to audit and troubleshoot.

Possible actions:
- Find out why the kernel modules of the mode selected by iptables-wrapper are not loaded on the node.
- If loading these kernel modules is acceptable, set ipv4OptionsGuard.allowKernelModuleLoad: true in the CNI module settings.
D8IPv4OptionsGuardFilterBypassed CE S6
Packets can skip the CVE-2026-98432 filter in VALUE network namespaces on the node NODE_NAME.
The node NODE_NAME uses legacy iptables. In VALUE network namespaces, a rule in the PREROUTING chain of the raw table ahead of the jump to D8-IPV4OPT-GUARD can end the traversal of the table (for example, -j ACCEPT). Packets accepted by that rule never reach the filter.

The guard puts its jump first when it installs the filter, but does not move it later, so as not to fight with other software on the node that manages its own rules. The affected network namespaces are named in the logs of the ipv4-options-guard container of the CNI agent pod on that node.

Possible actions:
- Find the software that inserted the rule (iptables-legacy -t raw -S PREROUTING in the network namespace) and make it append the rule instead, or use a target that does not end the traversal.
- Or put the jump first by hand: iptables-legacy -t raw -I PREROUTING 1 -j D8-IPV4OPT-GUARD. The guard removes the old jump on its next pass.
D8IPv4OptionsGuardMatchedTraffic CE S7

IPv4 packets with IP options were seen on the node NODE_NAME in the Audit mode.

Over the last hour, the ipv4-options-guard sidecar matched VALUE IPv4 packets with IP options (rule XXX) on the node NODE_NAME.

In the Audit mode they are only counted. In the Enforce mode they would be dropped, and this alert is not raised.

Legitimate TCP traffic almost never carries IP options. If a workload relies on them (for example, CIPSO labeled networking), exclude the node with the ipv4-options-guard.deckhouse.io/disabled: "true" label before switching ipv4OptionsGuard.mode to Enforce. Otherwise, this may be an attempt to exploit CVE-2026-98432.
D8IPv4OptionsGuardMetricsAbsent CE S4

The CVE-2026-98432 filter state is unknown on the node NODE_NAME.

The CNI agent pod runs on the node NODE_NAME, but Prometheus gets no metrics from its ipv4-options-guard container.

Without these metrics, the other D8IPv4OptionsGuard* alerts cannot fire for this node, so it may be unprotected unnoticed.

Check the ipv4-options-guard container of the CNI agent pod on that node and the scraping of its metrics.
D8IPv4OptionsGuardNodeUnprotected CE S4
The CVE-2026-98432 filter cannot be installed on the node NODE_NAME.
The ipv4-options-guard sidecar of the CNI agent could not protect the node NODE_NAME (reason: XXX).
- iptables_wrapper_failed — iptables-wrapper could not determine the iptables mode of the node.
- kernel_modules_not_loaded — the kernel modules of the mode selected by iptables-wrapper are not loaded, and loading them is not allowed.
The details are in the logs of the ipv4-options-guard container of the CNI agent pod on that node.

The node remains exposed to CVE-2026-98432 until its kernel is updated.

Possible actions:
- Update the node kernel to a version with the fix and reboot the node.
- If the node uses legacy iptables and loading the kernel modules named in the reason is acceptable, set ipv4OptionsGuard.allowKernelModuleLoad: true in the CNI module settings.
D8IPv4OptionsGuardReconcileFailed CE S6

The CVE-2026-98432 filter failed in VALUE network namespaces on the node NODE_NAME.

Some network namespaces on the node NODE_NAME are not protected.

Check the logs of the ipv4-options-guard container of the CNI agent pod on that node.

Module control-plane-manager

D8ControlPlaneManagerObsoleteMonitoringKubernetesControlPlaneModuleConfig CE S9
Obsolete monitoring-kubernetes-control-plane ModuleConfig should be removed.
The monitoring-kubernetes-control-plane module was merged into the control-plane-manager module, but its ModuleConfig still exists in the cluster and is no longer used.

Remove it manually:
d8 k delete moduleconfig monitoring-kubernetes-control-plane
If the ModuleConfig is deployed from an external source (e.g. a Git repository), make sure it is removed from there so it is not created again.
D8ControlPlaneManagerPodNotRunning CE S6
Controller pod isn't running on node NODE_NAME.
The d8-control-plane-manager pod is either failing or hasn’t been scheduled on node NODE_NAME.

To resolve this issue, check the status of the kube-system/d8-control-plane-manager DaemonSet and its pods by running the following command:
d8 k -n kube-system get daemonset,pod --selector=app=d8-control-plane-manager
D8ControlPlaneNodeMaintenanceModeEnabled CE S4
Maintenance mode is enabled for control-plane node NODE_NAME.
Control-plane node NODE_NAME has maintenance mode enabled with label control-plane-manager.deckhouse.io/maintenance. While this label is present, the control plane is not managed by control-plane-manager.

If maintenance is complete, remove the label:
d8 k label cpn NODE_NAME control-plane-manager.deckhouse.io/maintenance-
D8ControlPlaneOperationRunningTooLong CE S4
Control-plane operation XXX for COMPONENT_NAME on node NODE_NAME runs or retries longer than 10 minutes.
Control-plane operation XXX for component COMPONENT_NAME on node NODE_NAME has been running or retrying for more than 10 minutes.

Check the corresponding ControlPlaneOperation and logs of the d8-control-plane-manager Pod on that node:
d8 k get cpo XXX -o yaml d8 k -n kube-system logs -l app=d8-control-plane-manager --field-selector spec.nodeName=NODE_NAME
D8EtcdDatabaseHighFragmentationRatio CE S7
etcd database size in use is less than 50% of the allocated storage.
The etcd database size in use on instance INSTANCE_NAME is less than 50% of the allocated disk space, indicating potential fragmentation. Additionally, the total storage size exceeds 75% of the configured quota.

To resolve this issue, defragment the etcd database by running the following command:
d8 k -n kube-system exec -ti etcd-NODE_NAME -- /usr/bin/etcdctl \ --cacert /etc/kubernetes/pki/etcd/ca.crt \ --cert /etc/kubernetes/pki/etcd/ca.crt \ --key /etc/kubernetes/pki/etcd/ca.key \ --endpoints https://127.0.0.1:2379/ defrag --command-timeout=30s
D8EtcdExcessiveDatabaseGrowth CE S4

etcd cluster database is growing rapidly.

Based on the growth rate observed over the last six hours, Deckhouse predicts that the etcd database will run out of disk space within one day on instance INSTANCE_NAME.

To prevent disruptions, investigate the cause and take necessary action.
D8EtcdHighRequestsLatency CE S5
Etcd request latency (p99) has exceeded 800 ms multiple times in the past hour.
The 99th percentile of etcd request duration on INSTANCE_NAME has exceeded 800 ms more than twice in the last hour.

High etcd latency can cause API server timeouts, slow admission webhooks, and general cluster instability.

Investigate the following:
- Node disk I/O: Etcd is extremely sensitive to disk performance. Check disk utilization and I/O wait on the control plane node.
- Network-attached storage: If the etcd data directory resides on NFS, Ceph, or another network storage, check connectivity and latency to the storage backend. Etcd requires a consistently low-latency local disk (SSD recommended).
- Control plane summary: Review the Kubernetes Cluster → Control Plane Status Grafana dashboard for an overall picture of API server and etcd health, request rates, and error rates.
- Node base performance: Check node CPU and RAM utilization and load average. If the node is CPU-bound (load average significantly exceeds the number of CPU cores), consider scaling up the control plane node.
D8InvalidEtcdDefragCronSchedule CE S6
Invalid etcd defragmentation cron schedule in ModuleConfig.
The controlPlaneManager.etcd.defrag.cronSchedule value XXX is not a valid cron expression.

Deckhouse has fallen back to the default schedule (0 1 * * *) to keep defragmentation running.

To fix this, set a valid 5-field cron expression:
d8 k patch moduleconfig control-plane-manager --type merge \ -p '{"spec":{"settings":{"etcd":{"defrag":{"cronSchedule":"0 1 * * *"}}}}}'
Valid examples:
- 0 1 * * * — daily at 01:00 UTC
- 0 */6 * * * — every 6 hours
- 30 2 * * 0 — weekly on Sunday at 02:30 UTC

D8KubeEtcdDatabaseSizeCloseToTheLimit CE S3

etcd database size is approaching the limit.

The etcd database size on NODE_NAME is nearing its size limit. This may be caused by a high number of events, such as pod evictions or the recent creation of numerous resources in the cluster.

Possible solutions:

Defragment the etcd database by running the following command:

d8 k -n kube-system exec -ti etcd-NODE_NAME -- /usr/bin/etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/ca.crt \
  --key /etc/kubernetes/pki/etcd/ca.key \
  --endpoints https://127.0.0.1:2379/ defrag --command-timeout=30s

Increase node memory. Starting from 24 GB, quota-backend-bytes will increase by 1 GB for every extra 8 GB of memory.

Example:

Node memory	quota-backend-bytes
16 GB	2147483648 (2 GB)
24 GB	3221225472 (3 GB)
32 GB	4294967296 (4 GB)
40 GB	5368709120 (5 GB)
48 GB	6442450944 (6 GB)
56 GB	7516192768 (7 GB)
64 GB	8589934592 (8 GB)
72 GB	8589934592 (8 GB)
…	…

D8KubernetesStaleTokensDetected CE S8
Stale service account tokens detected for XXX/XXX.
Service account XXX/XXX is using a stale token. This issue may occur if an application reads the token only at startup and does not reload it periodically. As a result, an outdated token might be used, leading to security breach and authentication failures.

Recommended actions:
- Ensure your application is configured to periodically reload the token from the file system.
- Verify that you are using an up-to-date client library that supports automatic token rotation.
Note that currently these tokens are not blocked because the --service-account-extend-token-expiration flag is enabled by default (Default: true). With this flag enabled, admission-injected tokens are extended up to 1 year during token generation to facilitate a safe transition from legacy tokens to the bound service account token feature, ignoring the value of service-account-max-token-expiration.

For further investigation: You can find the problematic pods using the following command:
d8 k get pods -A -o json | jq -r '.items[] | select(.spec.serviceAccountName=="XXX" and .metadata.namespace=="XXX") | .metadata.name'
D8KubernetesVersionIsDeprecated CE S7

Kubernetes version VERSION_NUMBER is deprecated.

The current Kubernetes version VERSION_NUMBER has been deprecated, and support for it will be removed in upcoming releases.

Please migrate to the next kubernetes version (at least 1.32)

Read documentation about how to update the Kubernetes version in the cluster.

D8NeedDecreaseEtcdQuotaBackendBytes CE S6

Deckhouse suggests reducing quota-backend-bytes.

When the control plane node memory is reduced, Deckhouse may suggest reducing quota-backend-bytes. While Deckhouse is capable of automatically increasing this value, reducing it must be done manually.

To modify quota-backend-bytes, set the controlPlaneManager.etcd.maxDbSize parameter. Before setting a new value, check the current database usage on every control plane node by running:

for pod in $(d8 k get pod -n kube-system -l component=etcd,tier=control-plane -o name); do
  d8 k -n kube-system exec -ti "$pod" -- /usr/bin/etcdctl \
    --cacert /etc/kubernetes/pki/etcd/ca.crt \
    --cert /etc/kubernetes/pki/etcd/ca.crt \
    --key /etc/kubernetes/pki/etcd/ca.key \
    endpoint status -w json | jq --arg a "$pod" -r \
    '.[0].Status.dbSize / 1024 / 1024 | tostring | $a + ": " + . + " MB"';
done

Things to note:

The maximum value for controlPlaneManager.etcd.maxDbSize is 8 GB.
If control plane nodes have less than 24 GB, set controlPlaneManager.etcd.maxDbSize to 2 GB.

Starting from 24 GB, quota-backend-bytes will increase by 1 GB for every extra 8 GB of memory.

Example:

Node memory	quota-backend-bytes
16 GB	2147483648 (2 GB)
24 GB	3221225472 (3 GB)
32 GB	4294967296 (4 GB)
40 GB	5368709120 (5 GB)
48 GB	6442450944 (6 GB)
56 GB	7516192768 (7 GB)
64 GB	8589934592 (8 GB)
72 GB	8589934592 (8 GB)
…	…

D8ObsoleteEncryptionAlgorithmInClusterConfiguration CE S6
Obsolete encryptionAlgorithm in ClusterConfiguration needs to be migrated to control-plane-manager ModuleConfig.
The encryptionAlgorithm field is set to a non-default value in ClusterConfiguration but has not been migrated to ModuleConfig control-plane-manager yet.

Until the migration is complete, the value from ClusterConfiguration is used to avoid unintended certificate rotation.

To resolve this alert, migrate the setting:
1. If ModuleConfig control-plane-manager does not exist, create it first:
  
  d8 s module enable control-plane-manager
2. Migrate the value from ClusterConfiguration to ModuleConfig in one command:
  
  d8 k patch moduleconfig control-plane-manager --type merge -p "$(cat <<EOF {"spec": {"version": 3, "settings": {"encryptionAlgorithm": "$(d8 k -n kube-system get secret d8-cluster-configuration -o jsonpath='{.data.cluster-configuration\.yaml}' | base64 -d | grep encryptionAlgorithm | awk '{print $2}')"}}} EOF )"
Once ModuleConfig is updated, this alert will be resolved automatically.
1. Remove encryptionAlgorithm from ClusterConfiguration:
  
  d8 system edit cluster-configuration
  
  Delete the encryptionAlgorithm field and save.
D8ObsoleteGlobalControlPlaneResourcesRequests CE S6
Obsolete global control-plane resource requests need to be migrated.
Control-plane resource requests are being migrated from global settings to moduleconfig control-plane-manager.

The obsolete settings are still used as a fallback, but they should be moved to the control-plane-manager module settings. The migration does not change the meaning of the values: cpu and memory remain the total resource requests for control-plane components on each master node.

To complete the migration, follow the steps below:
1. Review the current obsolete settings in moduleconfig global:
  
  d8 k get mc global -o jsonpath='{.spec.settings.modules.resourcesRequests.controlPlane}'; echo
  
  Example of obsolete settings:
  
  spec: settings: modules: resourcesRequests: controlPlane: cpu: 1000m memory: 500Mi
2. Make sure the moduleconfig control-plane-manager resource exists and uses settings version 3.
  
  If the resource does not exist, create it:
  
  d8 s module enable control-plane-manager
  
  Set the settings version:
  
  d8 k patch mc control-plane-manager --type merge \ -p '{"spec":{"version":3}}'
3. Copy the obsolete values from moduleconfig global to moduleconfig control-plane-manager.
  
  The following command preserves the original JSON value types, including numeric CPU values:
  
  d8 k patch mc control-plane-manager --type merge -p "$( d8 k get mc global -o json | jq -c \ '{"spec":{"version":3,"settings":{"resourcesRequests": .spec.settings.modules.resourcesRequests.controlPlane}}}' )"
  
  After migration, the moduleconfig control-plane-manager resource should contain:
  
  spec: version: 3 settings: resourcesRequests: cpu: 1000m memory: 500Mi
4. Check that the values were applied:
  
  d8 k get mc control-plane-manager -o jsonpath='{.spec.settings.resourcesRequests}'; echo
5. Remove the obsolete settings from moduleconfig global:
  
  d8 k patch mc global --type merge \ -p '{"spec":{"settings":{"modules":{"resourcesRequests":null}}}}'
D8ObsoletePublishAPIinUserAuthn CE S6
Obsolete Publish API settings in user-authn need to be migrated.
Publish API ingress settings are being migrated from mc user-authn to mc control-plane-manager.

To complete the migration, follow the steps below:
1. Review current settings for publishAPI in the moduleconfig user-authn resource.
  
  Note that legacy user-authn spec.version: 1 uses publishAPI.enable instead of publishAPI.enabled. No manual rewrite is required: the command in step 4 uses the normalized payload from the temporary ConfigMap.
2. If the moduleconfig control-plane-manager resource exists, update its spec to spec.version: 3. If settings.apiserver.loadBalancer is set, migrate it to settings.apiserver.publishAPI.loadBalancer. If the resource does not exist, proceed to step 3.
3. If moduleconfig control-plane-manager does not exist, create it via d8:
  
  d8 s module enable control-plane-manager
  
  And set the settings version:
  
  d8 k patch moduleconfig control-plane-manager --type merge \ -p '{"spec":{"version":3}}'
4. Apply the migrated settings into moduleconfig control-plane-manager from temporary configMap by executing the following script:
  
  d8 k patch moduleconfig control-plane-manager --type merge -p "$(cat <<EOF {"spec": {"settings": {"apiserver": {"publishAPI": {"ingress": \ $(d8 k -n kube-system get cm d8-publishapi-config-migration -ojsonpath='{.data.config}') }}}}} EOF )"
5. Remove obsolete spec.settings.publishAPI section from moduleconfig user-authn.
D8ProblematicFeatureGateInUse CE S9
Feature gate XXX is STATUS_REFERENCE in current Kubernetes version XXX and should be removed from the configuration. Is not using in the cluster control-plane.
The feature gate XXX is currently enabled in the control-plane-manager ModuleConfig, but it is STATUS_REFERENCE in the current Kubernetes version XXX.

XXX This feature gate does not exist in any component (apiserver, kube-controller-manager, kube-scheduler, kubelet) and is not being applied to the cluster. XXX

Recommended actions:
1. Remove XXX from the spec.settings.enabledFeatureGates list in the control-plane-manager ModuleConfig:
d8 k edit mc control-plane-manager
For more information about Kubernetes feature gates, refer to the official documentation.
D8UsingFeatureGateWillBeDeprecated CE S9
Feature gate XXX will be deprecated in Kubernetes version XXX.
The feature gate XXX is currently enabled in the control-plane-manager ModuleConfig, but it will be deprecated in Kubernetes version XXX.

Current Kubernetes version: XXX

Recommended actions:
1. Review the usage of this feature gate in your cluster.
2. Plan migration away from this feature gate before upgrading to version XXX.
3. Remove XXX from the spec.settings.enabledFeatureGates list in the control-plane-manager ModuleConfig:
d8 k edit mc control-plane-manager
For more information about Kubernetes feature gates, refer to the official documentation.
HelmReleasesHasResourcesWithDeprecatedVersions CE S5
At least one Helm release contains resources with deprecated apiVersion, which will be removed in Kubernetes version VERSION_NUMBER.
To list all affected resources, run the following Prometheus query:
max by (helm_release_namespace, helm_release_name, helm_version, resource_namespace, resource_name, api_version, kind, k8s_version) (resource_versions_compatibility) == 1
For more details on upgrading deprecated resources, refer to the Kubernetes deprecation guide available at https://kubernetes.io/docs/reference/using-api/deprecation-guide/#vXXX.

Note that the check runs once per hour, so this alert should resolve within an hour after migrating deprecated resources.
HelmReleasesHasResourcesWithUnsupportedVersions CE S4
At least one Helm release contains resources with unsupported apiVersion for Kubernetes version VERSION_NUMBER.
To list all affected resources, run the following Prometheus query:
max by (helm_release_namespace, helm_release_name, helm_version, resource_namespace, resource_name, api_version, kind, k8s_version) (resource_versions_compatibility) == 2
For more details on migrating deprecated resources, refer to the Kubernetes deprecation guide available at https://kubernetes.io/docs/reference/using-api/deprecation-guide/#vXXX.

Note that the check runs once per hour, so this alert should resolve within an hour after migrating deprecated resources.
K8SApiserverDown CE S3

API servers can't be reached.

No API servers are reachable, or they have all disappeared from service discovery.
K8sCertificateExpiration CE S5
Kubernetes has API clients with soon-to-expire certificates.
Some clients are connecting to COMPONENT_NAME with certificates that will expire in less than a day on node COMPONENT_NAME.

To check control plane certificates, use kubeadm:
1. Install kubeadm using the following command:
  
  apt install kubeadm=1.24.*
2. Check certificates:
  
  kubeadm alpha certs check-expiration
To check kubelet certificates, do the following on each node:
1. Check kubelet configuration:
  
  ps aux \ | grep "/usr/bin/kubelet" \ | grep -o -e "--kubeconfig=\S*" \ | cut -f2 -d"=" \ | xargs cat
2. Locate the client-certificate or client-certificate-data field.
3. Check certificate expiration using OpenSSL.
Note that there are no tools to find other stale kubeconfig files. Consider enabling the control-plane-manager module for advanced debugging.
K8sCertificateExpiration CE S6
Kubernetes has API clients with soon-to-expire certificates.
Some clients are connecting to COMPONENT_NAME with certificates that will expire in less than 7 days on node NODE_NAME.

To check control plane certificates, use kubeadm:
1. Install kubeadm using the following command:
  
  apt install kubeadm=1.24.*
2. Check certificates:
  
  kubeadm alpha certs check-expiration
To check kubelet certificates, do the following on each node:
1. Check kubelet configuration:
  
  ps aux \ | grep "/usr/bin/kubelet" \ | grep -o -e "--kubeconfig=\S*" \ | cut -f2 -d"=" \ | xargs cat
2. Locate the client-certificate or client-certificate-data field.
3. Check certificate expiration using OpenSSL.
Note that there are no tools to find other stale kubeconfig files. Consider enabling the control-plane-manager module for advanced debugging.
K8SControllerManagerTargetDown CE S3

Controller manager is down.

There is no running kube-controller-manager. As a result, deployments and replication controllers are not progressing.
K8SSchedulerTargetDown CE S3

Scheduler is down.

The Kubernetes scheduler is not running. As a result, new pods are not being assigned to nodes.
KubeEtcdHighFsyncDurations CE S7
Syncing (fsync) WAL files to disk is slow.
In the last 15 minutes, the 99th percentile of the fsync duration for WAL files exceeded 0.5 seconds: VALUE.

Possible causes:
- High disk latency where etcd data is stored.
- High CPU usage on the node.
KubeEtcdHighNumberOfLeaderChanges CE S5
The etcd cluster is re-electing the leader too frequently.
There have been VALUE leader re-elections for the etcd cluster member running on node NODE_NAME in the last 10 minutes.

Possible causes:
- High disk latency where etcd data is stored.
- High CPU usage on the node.
- Degradation of network connectivity between cluster members in the multi-master mode.
KubeEtcdInsufficientMembers CE S4

Insufficient members in the etcd cluster.

The etcd cluster has too few members, increasing the risk of failure if another member becomes unavailable. To resolve this issue, check the status of etcd Pods: bash d8 k -n kube-system get pod -l component=etcd
KubeEtcdNoLeader CE S4

The etcd cluster member running on node NODE_NAME has lost the leader.

To resolve this issue, check the status of the etcd Pods: bash d8 k -n kube-system get pod -l component=etcd | grep NODE_NAME
KubeEtcdTargetAbsent CE S5
There is no etcd target in Prometheus.
Steps to troubleshoot:
1. Check the status of the etcd Pods:
  
  d8 k -n kube-system get pod -l component=etcd
2. Review Prometheus logs:
  
  d8 k -n d8-monitoring logs -l app.kubernetes.io/name=prometheus -c prometheus
KubeEtcdTargetDown CE S5
Prometheus is unable to scrape etcd metrics.
Steps to troubleshoot:
1. Check the status of the etcd Pods:
  
  d8 k -n kube-system get pod -l component=etcd
2. Review Prometheus logs:
  
  d8 k -n d8-monitoring logs -l app.kubernetes.io/name=prometheus -c prometheus

Module deckhouse

ApplicationIsInMaintenanceMode CE S6
Application NAME in namespace NAMESPACE is running in maintenance mode.
Application NAME in namespace NAMESPACE is running in maintenance mode. In this mode, its state is not reconciled, which prevents configuration or hook changes from being applied automatically.

To switch the application back to normal mode, edit the corresponding Application resource with the following command:
d8 k patch application NAME -n NAMESPACE --type=json -p='[{"op": "remove", "path": "/spec/maintenance"}]'
D8CNIEnabledMoreThanOne CE S2

More than one CNI is enabled in the cluster.

Deckhouse has detected that multiple CNIs are enabled in the cluster. For the cluster to work correctly, only one CNI must be enabled.

To resolve this issue, disable any unnecessary CNI.
D8CNIMisconfigured CE S5
The parameters specified in ModuleConfig of CNI_NAME do not match the ones that are actually being used in the cluster.
This happened because there were several sources for configuring CNI parameters in the cluster, and ModuleConfig did not have the highest priority previously.

To resolve this issue, the following steps should be taken:
1. Find the ConfigMap d8-system/desired-cni-moduleconfig in the cluster, which contains the actual ModuleConfig settings.
  
  d8 k -n d8-system get configmap desired-cni-moduleconfig -o yaml
2. Apply this prepared ModuleConfig to the cluster. This will not cause any actual reconfigurations in the cluster and is completely safe.
3. Once the parameters in “ModuleConfig” match those used in the cluster:
  
  this alert will be resolved,
  
  and the priority of the sources for configuring CNI will change, and ModuleConfig will become the main source of truth.
4. After that, you can add the desired parameters to the ModuleConfig CNI and they will be applied immediately.
D8DeckhouseConfigInvalid CE S5
Deckhouse configuration is invalid.
Deckhouse configuration contains errors.

Steps to troubleshoot:
1. Check Deckhouse logs by running the following command:
  
  d8 k -n d8-system logs -f -l app=deckhouse
2. Edit the Deckhouse global configuration:
  
  d8 k edit mc global
  
  Or edit configuration of a specific module:
  
  d8 k edit mc MODULE_NAME
D8DeckhouseCouldNotDeleteModule CE S4
Deckhouse is unable to delete the MODULE_NAME module.
To investigate the issue, check the logs by running the following command:
d8 k -n d8-system logs -f -l app=deckhouse
D8DeckhouseCouldNotDiscoverModules CE S4
Deckhouse is unable to discover modules.
To investigate the issue, check the logs by running the following command:
d8 k -n d8-system logs -f -l app=deckhouse
D8DeckhouseCouldNotRunGlobalHook CE S5
Deckhouse is unable to run the HOOK_NAME global hook.
To investigate the issue, check the logs by running the following command:
d8 k -n d8-system logs -f -l app=deckhouse
D8DeckhouseCouldNotRunModule CE S4
Deckhouse is unable to start the MODULE_NAME module.
To investigate the issue, check the logs by running the following command:
d8 k -n d8-system logs -f -l app=deckhouse
D8DeckhouseCouldNotRunModuleHook CE S7
Deckhouse is unable to run the MODULE_NAME/HOOK_NAME module hook.
To investigate the issue, check the logs by running the following command:
d8 k -n d8-system logs -f -l app=deckhouse
D8DeckhouseCustomTargetDown CE S4

Prometheus is unable to scrape custom metrics generated by Deckhouse hooks.
D8DeckhouseDeprecatedConfigmapManagedByArgoCD CE S4

Deprecated Deckhouse ConfigMap managed by Argo CD.

The Deckhouse ConfigMap is no longer used.

To resolve this issue, remove the d8-system/deckhouse ConfigMap from Argo CD.
D8DeckhouseGlobalHookFailsTooOften CE S9
The HOOK_NAME Deckhouse global hook is crashing too frequently.
The HOOK_NAME hook has failed multiple times in the last __SCRAPE_INTERVAL_X_4__.

To investigate the issue, check the logs by running the following command:
d8 k -n d8-system logs -f -l app=deckhouse
D8DeckhouseHasNoAccessToRegistry CE S7
Deckhouse is unable to connect to the registry.
Deckhouse can’t connect to the registry to check for new releases. With a release channel set, it checks the channel every 3 minutes; without one, it checks the digest of its own image every 15 seconds. Without access to the registry, automatic updates are unavailable.

Module images come from the same registry, so pods that need a new image fail to start as well, sometimes with errors that do not mention the registry. The usual causes are expired or changed credentials, a registry certificate that has expired or was replaced with one signed by an unknown authority, and a network or proxy change.

To investigate the issue, follow these steps:
1. Read the registry error in the logs. UNAUTHORIZED points at the credentials, x509 at the certificate, and a timeout or a refused connection at the network:
  
  d8 k -n d8-system logs -l app=deckhouse --tail=500 | grep -iE 'check deckhouse release|registry'
2. Check which registry address the cluster is configured to use:
  
  d8 k -n d8-system get secret deckhouse-registry -o jsonpath='{.data.address}' | base64 -d
3. If the address is correct, the credentials are the next suspect. Check when the secret was last updated:
  
  d8 k -n d8-system get secret deckhouse-registry
4. If the error mentions x509, check who issued the registry certificate and when it expires. Run the command on a master node and substitute the host from the registry address, and its port if it is not 443:
  
  openssl s_client -connect REGISTRY_HOST:443 -servername REGISTRY_HOST </dev/null 2>/dev/null | openssl x509 -noout -issuer -enddate
5. Check that the node running Deckhouse can reach the registry at all, for example through a proxy or a firewall rule that changed.
D8DeckhouseIsHung CE S4

Deckhouse is down.

Deckhouse is probably down, since the deckhouse_live_ticks metric in Prometheus has stopped increasing. This metric is expected to increment every 10 seconds.
D8DeckhouseIsNotOnReleaseChannel CE S9
Deckhouse isn't subscribed to any regular release channels.
Deckhouse in this cluster isn’t subscribed to any of the regular release channels: Alpha, Beta, EarlyAccess, Stable, or RockSolid.

To resolve this issue, follow these steps:
1. Check the current release channel used in the cluster:
  
  d8 k -n d8-system get deploy deckhouse -o json | jq '.spec.template.spec.containers[0].image' -r
2. Subscribe to one of the regular release channels by adjusting the deckhouse module configuration.
D8DeckhouseModuleHookFailsTooOften CE S9
The MODULE_NAME/HOOK_NAME Deckhouse hook is crashing too frequently.
The HOOK_NAME hook of the MODULE_NAME module has failed multiple times in the last __SCRAPE_INTERVAL_X_4__.

To investigate the issue, check the logs by running the following command:
d8 k -n d8-system logs -f -l app=deckhouse
D8DeckhouseModuleUpdatePolicyNotFound CE S5
Module update policy not found for MODULE_RELEASE.
The module update policy for MODULE_RELEASE is missing.

To resolve this issue, remove the label from the module release using the following command:
d8 k label mr MODULE_RELEASE modules.deckhouse.io/update-policy-
A new suitable policy will be detected automatically.
D8DeckhouseModuleValidationError CE S5
Module configuration failed for module MODULE_NAME.
Initial config for module MODULE_NAME is not valid.

You can get more details via
d8 k get mr -l module=MODULE_NAME
Provided error: XXX
D8DeckhouseObsoleteMonitoringDeckhouseModuleConfig CE S9
Obsolete monitoring-deckhouse ModuleConfig should be removed.
The monitoring-deckhouse module was merged into the deckhouse module, but its ModuleConfig still exists in the cluster and is no longer used.

Remove it manually:
d8 k delete moduleconfig monitoring-deckhouse
If the ModuleConfig is deployed from an external source (e.g. a Git repository), make sure it is removed from there so it is not created again.
D8DeckhousePodIsNotReady CE S4

The Deckhouse Pod is NOT Ready.
D8DeckhousePodIsNotRunning CE S4

The Deckhouse Pod is NOT Running.
D8DeckhousePodIsRestartingTooOften CE S9
Excessive Deckhouse restarts detected.
Number of restarts in the last hour: VALUE.

Excessive Deckhouse restarts indicate a potential issue. Normally, Deckhouse should be up and running continuously.

To investigate the issue, check the logs by running the following command:
d8 k -n d8-system logs -f -l app=deckhouse
D8DeckhouseQueueIsHung CE S6
Module queue MODULE_NAME (type XXX, hook HOOK_NAME) is hung.
The module queue head has remained unchanged for more than 10 minutes.

Affected queue:
- Module: MODULE_NAME
- Task type: XXX
- Hook: HOOK_NAME
To investigate the issue, check the logs of the deckhouse controller pods:
d8 k -n d8-system logs -f -l app=deckhouse
D8DeckhouseQueueIsHungCritical CE S4
Critical module queue MODULE_NAME (type XXX, hook HOOK_NAME) is hung.
The critical module queue head has remained unchanged for more than 10 minutes.

Affected queue:
- Module: MODULE_NAME
- Task type: XXX
- Hook: HOOK_NAME
To investigate the issue, check the logs of the deckhouse controller pods:
d8 k -n d8-system logs -f -l app=deckhouse
D8DeckhouseQueueIsHungGlobal CE S4
Global/Parallel task XXX (hook HOOK_NAME) is hung.
The global/parallel task head has remained unchanged for more than 10 minutes.

Affected task:
- Task type: XXX
- Hook: HOOK_NAME
To investigate the issue, check the logs of the deckhouse controller pods:
d8 k -n d8-system logs -f -l app=deckhouse
D8DeckhouseSelfTargetAbsent CE S4

There is no Deckhouse target in Prometheus.
D8DeckhouseSelfTargetDown CE S4

Prometheus is unable to scrape Deckhouse metrics.
D8DeckhouseWatchErrorOccurred CE S5
Possible API server connection error in the client-go informer.
Deckhouse has detected an error in the client-go informer, possibly due to connection issues with the API server.

Steps to investigate:
1. Check Deckhouse logs for more information by running:
  
  d8 k -n d8-system logs deploy/deckhouse | grep error | grep -i watch
2. This alert attempts to detect a correlation between the faulty snapshot invalidation and API server connection errors, specifically for the handle-node-template hook in the node-manager module.
  
  To compare the snapshot with the actual node objects for this hook, run the following command:
  
  diff -u <(d8 k get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'|sort) <(d8 k -n d8-system exec svc/deckhouse-leader -c deckhouse -- deckhouse-controller module snapshots node-manager -o json | jq '."040-node-manager/hooks/handle_node_templates.go"' | jq '.nodes.snapshot[] | .filterResult.Name' -r | sort)
D8HasModuleConfigAllowedToDisable CE S4
The ModuleConfig annotation for allowing module disabling is set.
The ModuleConfig is pending module disabling.

It is recommended that you keep your module configurations clean by removing unnecessary approval annotations.

If you ignore this alert and do not clear the annotation, the module may be accidentally removed from the cluster, potentially leading to irreversible consequences.

To resolve this issue and stop the alert, run the following command:
d8 k annotate moduleconfig MODULE_NAME modules.deckhouse.io/allow-disabling-
D8ModuleOutdatedByMajorVersion CE S9
Module MODULE_NAME is running XXX major versions behind.
Module MODULE_NAME is running XXX major versions behind on major version releases. Major version updates may contain breaking changes and require immediate attention and careful planning.

To check the module releases status, run the following command:
d8 k get mr -l module=MODULE_NAME ```XXXXXX This major release has from-to version constraints that must be met before applying the update. To check the update instructions (from-to versions), run the following command: ```bash d8 k get mr NAME -o jsonpath='{.spec.update.versions[*]}'
This will show you the update paths including the from-to versions defined in the release.XXX

To approve the module major release update, run the following command:

bash d8 k annotate mr NAME modules.deckhouse.io/approved="true"XXX
D8NodeHasDeprecatedOSVersion CE S4
Nodes with deprecated OS versions detected.
Deckhouse has detected nodes running deprecated OS versions.

Steps to troubleshoot:
1. Get a list of affected nodes by running the following Prometheus query:
  
  kube_node_info{os_image=~"Ubuntu 18.04.*|Debian GNU/Linux 10.*|CentOS Linux 7.*"}
2. Update the affected nodes to a supported OS version.
DeckhouseEditionNotFound CE S6
Incorrect Deckhouse Platform edition.
Specify a Deckhouse Platform edition in MODULE_NAME ModuleConfig.
d8 k patch moduleconfig MODULE_NAME --type merge --patch '{"spec": {"settings": {"license": {"edition": "DKP_EDITION"}}}}'
Where DKP_EDITION is the edition you want to use: CE, BE, EE, SE, or SE-plus.

DeckhouseHighMemoryUsage CE S3

Deckhouse memory usage is too high.

Deckhouse pod POD_NAME on node NODE_NAME has high memory usage.

Please, if possible, get the dumps for debugging purposes (it will take around 30 seconds):

curl -s "http://$(d8 k -n d8-system get pod POD_NAME -o jsonpath='{.status.hostIP}'):4222/debug/pprof/heap" -o /tmp/heap.pprof
curl -s "http://$(d8 k -n d8-system get pod POD_NAME -o jsonpath='{.status.hostIP}'):4222/debug/pprof/goroutine" -o /tmp/goroutine.pprof
curl -s "http://$(d8 k -n d8-system get pod POD_NAME -o jsonpath='{.status.hostIP}'):4222/debug/pprof/profile?seconds=30" -o /tmp/cpu.pprof

and send these files (/tmp/heap.pprof, /tmp/cpu.pprof and /tmp/goroutine.pprof) to the support team.

DeckhouseMigratedModuleNotDownloaded CE S6
Migrated module MODULE_NAME has not been downloaded yet.
Module MODULE_NAME specified in DeckhouseRelease was moved to an external source and is available in a ModuleSource registry, but it has not been downloaded yet (no deployed ModuleRelease exists). DeckhouseRelease will be locked until the module is downloaded.

This alert is usually temporary while the module source controller downloads the module from the registry. If the alert persists, this may indicate:
- The module’s update policy or ModuleSource is misconfigured, preventing the module from being downloaded.
- A network issue is preventing the module from being downloaded.
To investigate the issue, do the following:
- Check the ModuleRelease objects for the module:
  
  d8 k get mr -l module=MODULE_NAME
- Check the configured ModuleSources:
  
  d8 k get modulesources
- Check the deckhouse logs for more details:
  
  d8 k -n d8-system logs -f -l app=deckhouse
DeckhouseMigratedModuleNotFound CE S6
Migrated module MODULE_NAME was not found in any ModuleSource registry.
Module MODULE_NAME specified in DeckhouseRelease was moved to an external source and was not found in any available ModuleSource registry. DeckhouseRelease will be locked until the module becomes available in one of the configured ModuleSource registries.

This may indicate:
- A ModuleSource registry is misconfigured.
- The module has not been published to the registry yet.
- A network issue is preventing access to the registry.
To investigate the issue, do the following:
- Check the configured ModuleSources:
  
  d8 k get modulesources
- List the available module sources:
  
  d8 k -n d8-system exec -it deployment/deckhouse -c deckhouse -- deckhouse-controller registry get sources
- List the modules available in a specific source:
  
  d8 k -n d8-system exec -it deployment/deckhouse -c deckhouse -- deckhouse-controller registry get modules source_name
- Check the deckhouse logs for more details:
  
  d8 k -n d8-system logs -f -l app=deckhouse
DeckhouseModuleUpdatingFailedBrokenSequence CE S4
Module update has failed. Updating sequence is broken.
The ‘MODULE_NAME’ module update has failed.

Current version: XXX. Desired version: VERSION_NUMBER.

Attempt to download interim releases failed.

Possible reasons:
- The necessary versions of the module image is not available in the registry.
To resolve this issue, ensure that all minor versions of the ‘MODULE_NAME’ module image between XXX and VERSION_NUMBER is available in the registry ‘XXX’.
DeckhouseModuleUpdatingFailedModuleIsNotValid CE S4
Module update has failed. Module is not valid.
The ‘MODULE_NAME’ module update has failed.

Desired version: VERSION_NUMBER.

Possible reasons:
- The module image is corrupted.
To resolve this issue, ensure that the VERSION_NUMBER image in the registry ‘XXX’ is valid.

Check which versions of the ‘MODULE_NAME’ module are available in the registry:
d8 cr ls XXX/MODULE_NAME/release
For a private registry, pass the credentials from the corresponding ModuleSource:
d8 cr ls -u USERNAME -p PASSWORD XXX/MODULE_NAME/release
To get the exact error, check Deckhouse logs: d8 k -n d8-system logs deploy/deckhouse | grep "failed to download module"
DeckhouseReleaseDisruptionApprovalRequired CE S4
Deckhouse release disruption approval required.
The new Deckhouse release includes a disruptive update that requires manual approval.

To check the details, run the following command:
d8 k describe DeckhouseRelease NAME
To approve the disruptive update, run the following command:
d8 k annotate DeckhouseRelease NAME release.deckhouse.io/disruption-approved=true
DeckhouseReleaseIsBlocked CE S5
Deckhouse release requirements haven't been met.
The requirements for the Deckhouse release haven’t been met.

To check the details, run the following command:
d8 k describe DeckhouseRelease NAME
DeckhouseReleaseIsWaitingManualApproval CE S3
A new Deckhouse release is awaiting manual approval.
A new Deckhouse release is available but requires manual approval before it can be applied.

To approve the release, run the following command:
d8 k patch DeckhouseRelease NAME --type=merge -p='{"approved": true}'
DeckhouseReleaseIsWaitingManualApproval CE S6
A new Deckhouse release is awaiting manual approval.
A new Deckhouse release is available but requires manual approval before it can be applied.

To approve the release, run the following command:
d8 k patch DeckhouseRelease NAME --type=merge -p='{"approved": true}'
DeckhouseReleaseIsWaitingManualApproval CE S9
A new Deckhouse release is awaiting manual approval.
A new Deckhouse release is available but requires manual approval before it can be applied.

To approve the release, run the following command:
d8 k patch DeckhouseRelease NAME --type=merge -p='{"approved": true}'
DeckhouseReleaseNotificationNotSent CE S4
Deckhouse release notification webhook hasn't been sent.
The Deckhouse release notification webhook failed to send.

To check the notification webhook address, run the following command:
d8 k get mc deckhouse -o yaml
DeckhouseUpdating CE S4

Deckhouse is being updated to XXX.
DeckhouseUpdatingFailed CE S4
Deckhouse update has failed.
The Deckhouse update has failed.

Possible reasons:
- The next minor or patch version of the Deckhouse image is not available in the registry.
- The Deckhouse image is corrupted.
Current version: VERSION_NUMBER.

To resolve this issue, ensure that the next version of the Deckhouse image is available in the registry.
ModuleAtConflict CE S4

Conflict detected for module MODULE_NAME.

Deckhouse has detected conflicting sources for the MODULE_NAME module.

To resolve this issue, specify the correct source in the module configuration.
ModuleConfigObsoleteVersion CE S4

ModuleConfig NAME is outdated.

Deckhouse has detected that ModuleConfig NAME is outdated.

To resolve this issue, update ModuleConfig NAME to the latest version.
ModuleHasDeprecatedUpdatePolicy CE S4
Module MODULE_NAME is matched by the deprecated update policy UPDATE_POLICY_NAME.
Deckhouse has detected that the module MODULE_NAME is using the deprecated update policy UPDATE_POLICY_NAME. The v1alpha1 policy has a selector that no longer functions.

To specify the update policy in the module configuration, run the following command:
d8 k patch moduleconfig MODULE_NAME --type='json' -p='[{"op": "add", "path": "/spec/updatePolicy", "value": "UPDATE_POLICY_NAME"}]'
After resolving all alerts related to the update policy UPDATE_POLICY_NAME, clear the selector by running the following command:
d8 k patch moduleupdatepolicies.v1alpha1.deckhouse.io UPDATE_POLICY_NAME --type='json' -p='[{"op": "replace", "path": "/spec/moduleReleaseSelector/labelSelector/matchLabels", "value": {"": ""}}]'
ModuleIsDeprecated CE S9

Module MODULE_NAME is deprecated.

The module MODULE_NAME is deprecated and will not receive new updates. The module may be removed in the future, so please disable it in a controlled manner.
ModuleIsInMaintenanceMode CE S6
Module MODULE_NAME is running in maintenance mode.
Module MODULE_NAME is running in maintenance mode. In this mode, its state is not reconciled, which prevents configuration or hook changes from being applied automatically.

To switch the module back to normal mode, edit the corresponding ModuleConfig resource with the following command:
d8 k patch moduleconfig MODULE_NAME --type=json -p='[{"op": "remove", "path": "/spec/maintenance"}]'
ModuleReleaseIsBlockedByRequirements CE S6
A new release for module MODULE_NAME has been blocked due to unmet requirements.
A new release for module MODULE_NAME has been blocked because it doesn’t meet the required conditions.

To check the requirements, run the following command:
d8 k get mr NAME -o json | jq .spec.requirements
ModuleReleaseIsOutdated CE S5
Module MODULE_NAME is XXX or more minor versions behind the latest available release.
Module MODULE_NAME is running XXX or more minor versions behind the latest available release. This module is significantly outdated and may have compatibility issues.

To approve the module release update, run the following command:
d8 k annotate mr NAME modules.deckhouse.io/approved="true"
ModuleReleaseIsOutdated CE S7
Module MODULE_NAME is 2 minor versions behind the latest available release.
Module MODULE_NAME is running 2 minor versions behind the latest available release. Updating is recommended to maintain compatibility and access to latest features.

To approve the module release update, run the following command:
d8 k annotate mr NAME modules.deckhouse.io/approved="true"
ModuleReleaseIsOutdated CE S9
Module MODULE_NAME is 1 minor version behind the latest available release.
Module MODULE_NAME is running 1 minor version behind the latest available release. Consider updating to maintain compatibility and security.

To approve the module release update, run the following command:
d8 k annotate mr NAME modules.deckhouse.io/approved="true"
ModuleReleaseIsWaitingManualApproval CE S6
A new release for module MODULE_NAME is awaiting manual approval.
A new release for module MODULE_NAME is available but requires manual approval before it can be applied.

To approve the module release, run the following command:
d8 k annotate mr NAME modules.deckhouse.io/approved="true"

Module descheduler

Module documentation

ModuleConfigDeprecated CE S9
Deprecated deckhouse-web ModuleConfig detected.
The deckhouse-web module has been renamed to documentation, and a new documentation ModuleConfig is generated automatically.

Steps to troubleshoot:
1. Remove the deprecated deckhouse-web ModuleConfig from the CI deployment process.
2. Delete it using the following command:
  
  d8 k delete mc deckhouse-web
ModuleDocumentationRenderError CE S6
Documentation rendering failed for module MODULE_NAME.
An error occurred while rendering documentation for the MODULE_NAME module. The broken module was removed from the documentation site during Hugo build.

Check the ModuleDocumentation resource status for details:
d8 k get moduledocumentations MODULE_NAME -o yaml
Check docs-builder logs for more information:
d8 k -n d8-system logs -l app=docs-builder | grep -i "removed broken module"

Module istio

D8IstioActiveWaypointsWithAmbientDisabled EE S6
Active Istio waypoint exists while ambient mesh is disabled.
Deckhouse has detected an active waypoint Deployment DEPLOYMENT_NAME in namespace NAMESPACE, created for WaypointInstance XXX, while ambient mesh is disabled.

This usually happens if ambient mesh was enabled, a WaypointInstance was created, and then ambient mesh was disabled. In this state the waypoint-controller is not running, so it cannot reconcile or clean up WaypointInstance resources.

To inspect affected resources, run:
d8 k -n NAMESPACE get waypointinstance XXX -o yaml d8 k -n NAMESPACE get deploy,svc,sa,gateway,pdb,hpa,vpa -l istio.deckhouse.io/waypoint-instance=XXX
Recommended remediation:
1. Temporarily enable ambient mesh.
2. Wait until waypoint-controller starts.
3. Delete the affected WaypointInstance or change configuration as needed.
4. Wait until managed waypoint resources are reconciled as needed or removed.
5. Disable ambient mesh again.
If ambient mesh cannot be re-enabled, manually remove the managed waypoint resources and then remove the WaypointInstance finalizer only after verifying that no managed resources remain.
D8IstioActualDataPlaneVersionNotEqualDesired EE S8
There are pods with Istio data plane version VERSION_NUMBER, but desired version is XXX.
There are pods in the NAMESPACE namespace with Istio data plane version VERSION_NUMBER, while the desired version is XXX. As a result, the Istio version will be changed after the pod is restarted.

To resolve the issue, use the following cheat sheet:
### Namespace-wide configuration # `istio.io/rev=vXYZ`: Use a specific revision. # `istio-injection=enabled`: Use the global revision. d8 k get ns NAMESPACE --show-labels ### Pod-wide configuration d8 k -n NAMESPACE get pods -l istio.io/rev=DESIRED_VISION
D8IstioActualVersionIsNotInstalled EE S4
The control plane version for pods with injected sidecars isn't installed.
There are pods in the NAMESPACE namespace with injected sidecars of version VERSION_NUMBER (revision REVISION_NUMBER), but the corresponding control plane version is not installed. As a result, these pods have lost synchronization with the state in Kubernetes.

To resolve this issue, install the required control plane version. Alternatively, update the namespace or pod configuration to match an installed control plane version.

To identify orphaned pods, run the following command:
d8 k -n NAMESPACE get pods -l 'service.istio.io/canonical-name' -o json | jq --arg revision REVISION_NUMBER '.items[] | select(.metadata.annotations."sidecar.istio.io/status" // "{}" | fromjson | .revision == $revision) | .metadata.name'
D8IstioAdditionalControlplaneDoesntWork CE S4
Additional control plane isn't working.
Deckhouse has detected that the additional Istio control plane ISTIO_REVISION_LABEL isn’t working.

As a result, sidecar injection for pods with ISTIO_REVISION_LABEL isn’t working as well.

To check the status of the control plane pods, run the following command:
d8 k get pods -n d8-istio -l istio.io/rev=ISTIO_REVISION_LABEL
D8IstioAllianceDataPlaneNotConnected EE S4
Data-plane connectivity check failed for XXX NAME.
The data-plane health check of XXX NAME with uuid XXX has been failing for more than 5 minutes.

This means that actual service mesh traffic between clusters is not flowing properly, even though the control plane metadata exchange may be working.

Possible causes:
- The remote cluster’s alliance-healthcheck pod is not running.
- Istio sidecar injection is not working properly on the healthcheck pods.
- The ingress gateway between clusters is misconfigured or unreachable.
- mTLS or trust domain configuration issues between clusters.
To check the status:
d8 k get XXX NAME -o json | jq '.status.conditions[] | select(.type=="DataplaneConnectionReady")'
D8IstioAllianceDiscoveryMetadataEndpointRequestedViaDeprecatedSubdomain EE S5
A peer cluster requests private metadata using a deprecated subdomain.
Remote XXX NAME, (UUID: XXX) requests private metadata from hostname XXX, that hostname uses the deprecated istio subdomain.

Update the remote cluster XXX resource: set spec.metadataEndpoint to (publicDomainTemplate + istio-metadata)/metadata/.

Migration steps on the remote cluster:
1. Ensure DNS resolves the new target (publicDomainTemplate + istio-metadata).
2. Update spec.metadataEndpoint to use the new domain name. Keep the path /metadata/ unchanged.
3. Verify remote metadata fetch and alliance health after the change.
D8IstioDataPlaneVersionMismatch EE S8
There are pods with data plane version different from the control plane version.
There are pods in the NAMESPACE namespace with Istio data plane version VERSION_NUMBER, which is different from the control plane version DESIRED_VERSION.

Steps to resolve the issue:
1. Restart affected pods and use the following PromQL query to get a full list:
  
  max by (namespace, dataplane_pod) (d8_istio_dataplane_metadata{full_version="VERSION_NUMBER"})
2. Use the automatic Istio data plane upgrade.
D8IstioDataPlaneWithoutIstioInjectionConfigured EE S4
Detected pods with Istio sidecars but istio-injection isn't configured.
There are pods in the NAMESPACE namespace with Istio sidecars, but istio-injection isn’t configured. As a result, these pods will lose their Istio sidecars after being recreated.

To identify the affected pods, run the following command:
d8 k -n NAMESPACE get pods -o json | jq -r --arg revision REVISION_NUMBER '.items[] | select(.metadata.annotations."sidecar.istio.io/status" // "{}" | fromjson | .revision == $revision) | .metadata.name'
D8IstioDeprecatedIstioVersionInstalled CE

The installed Istio version has been deprecated.

Deckhouse has detected that a deprecated Istio version VERSION_NUMBER is installed.

Support for this version will be removed in upcoming Deckhouse releases. The higher the alert severity, the greater the probability of support being discontinued.

To learn how to upgrade Istio, refer to the upgrade guide.
D8IstioDesiredVersionIsNotInstalled EE S6
Desired control plane version isn't installed.
There is a desired Istio control plane version XXX (revision REVISION_NUMBER) configured for pods in the NAMESPACE namespace, but that version isn’t installed. As a result, pods can’t be recreated in the NAMESPACE namespace.

To resolve this issue, install the desired control plane version. Alternatively, update the namespace or pod configuration to match an installed control plane version.

Use the following cheat sheet:
### Namespace-wide configuration # `istio.io/rev=vXYZ`: Use a specific revision. # `istio-injection=enabled`: Use the global revision. d8 k get ns NAMESPACE --show-labels ### Pod-wide configuration d8 k -n NAMESPACE get pods -l istio.io/rev=REVISION_NUMBER

D8IstioFederationMetadataEndpointDoesntWork EE S6

Federation metadata endpoint failed.

The metadata endpoint ENDPOINT_NAME for IstioFederation FEDERATION_NAME has failed to fetch via the Deckhouse hook.

To reproduce the request to the public endpoint, run the following command:

curl ENDPOINT_NAME

To reproduce the request to private endpoints (run from the Deckhouse pod), run the following:

KEY="$(deckhouse-controller module values istio -o json | jq -r .internal.remoteAuthnKeypair.priv)"
LOCAL_CLUSTER_UUID="$(deckhouse-controller module values -g istio -o json | jq -r .global.discovery.clusterUUID)"
REMOTE_CLUSTER_UUID="$(d8 k get istiofederation FEDERATION_NAME -o json | jq -r .status.metadataCache.public.clusterUUID)"
TOKEN="$(deckhouse-controller helper gen-jwt --private-key-path <(echo "$KEY") --claim iss=d8-istio --claim sub=$LOCAL_CLUSTER_UUID --claim aud=$REMOTE_CLUSTER_UUID --claim scope=private-federation --ttl 1h)"
curl -H "Authorization: Bearer $TOKEN" ENDPOINT_NAME

D8IstioGlobalControlplaneDoesntWork CE S4
Global control plane isn't working.
Deckhouse has detected that the global Istio control plane ISTIO_REVISION_LABEL isn’t working.

As a result, sidecar injection for pods with global revision isn’t working as well, and the validating webhook for Istio resources is absent.

To check the status of the control plane pods, run the following command:
d8 k get pods -n d8-istio -l istio.io/rev=ISTIO_REVISION_LABEL

D8IstioMulticlusterMetadataEndpointDoesntWork EE S6

Multicluster metadata endpoint failed.

The metadata endpoint ENDPOINT_NAME for IstioMulticluster MULTICLUSTER_NAME has failed to fetch via the Deckhouse hook.

To reproduce the request to the public endpoint, run the following command:

curl ENDPOINT_NAME

To reproduce the request to private endpoints (run from the d8-system/deckhouse pod), run the following:

KEY="$(deckhouse-controller module values istio -o json | jq -r .internal.remoteAuthnKeypair.priv)"
LOCAL_CLUSTER_UUID="$(deckhouse-controller module values -g istio -o json | jq -r .global.discovery.clusterUUID)"
REMOTE_CLUSTER_UUID="$(d8 k get istiomulticluster MULTICLUSTER_NAME -o json | jq -r .status.metadataCache.public.clusterUUID)"
TOKEN="$(deckhouse-controller helper gen-jwt --private-key-path <(echo "$KEY") --claim iss=d8-istio --claim sub=$LOCAL_CLUSTER_UUID --claim aud=$REMOTE_CLUSTER_UUID --claim scope=private-multicluster --ttl 1h)"
curl -H "Authorization: Bearer $TOKEN" ENDPOINT_NAME

D8IstioMulticlusterRemoteAPIHostDoesntWork EE S6
Multicluster remote API host health check failed.
The remote API host API_HOST for IstioMulticluster MULTICLUSTER_NAME has failed the health check performed by the Deckhouse monitoring hook.

To reproduce the request (run from the d8-system/deckhouse pod), run the following:
TOKEN="$(deckhouse-controller module values istio -o json | jq -r --arg ah API_HOST '.internal.multiclusters[]| select(.apiHost == $ah)| .apiJWT ')" curl -H "Authorization: Bearer $TOKEN" https://API_HOST/version
D8IstioOperatorReconcileError CE S5
The istio-operator is unable to reconcile Istio control plane setup.
Deckhouse has detected an error in the istio-operator reconciliation loop.

To investigate the issue, check the operator logs:
d8 k -n d8-istio logs -l app=operator,revision=REVISION_NUMBER
D8IstioPodsWithoutIstioSidecar EE S4
Detected pods without Istio sidecars but with istio-injection configured.
There is a pod POD_NAME in the NAMESPACE namespace without Istio sidecars, but with istio-injection configured.

To identify the affected pods, run the following command:
d8 k -n NAMESPACE get pods -l '!service.istio.io/canonical-name' -o json | jq -r '.items[] | select(.metadata.annotations."sidecar.istio.io/inject" != "false") | .metadata.name'
D8IstioRemoteClusterNotSynced EE S4
Istio remote cluster CLUSTER_ID is not synchronized.
The Istio control plane instance INSTANCE_NAME cannot synchronize with the remote cluster CLUSTER_ID.

Possible causes:
- The remote cluster is offline.
- The remote API endpoint is not reachable.
- The remote ServiceAccount token is invalid or expired.
- There is a TLS or certificate issue between the clusters.
- Remote and local Istio versions mismatch.
D8IstioReservedUIDConflict CE S4

Application container is running as UID 1337 (reserved by Istio).

Container CONTAINER_NAME in pod POD_NAME in namespace NAMESPACE is running as UID 1337.

UID 1337 is reserved by Istio for the istio-proxy sidecar container. When application containers run with this UID, their traffic bypasses Istio completely — no routing rules, mTLS, or telemetry will be applied.

To fix the issue, change the runAsUser in the container’s or pod’s securityContext to a different UID.
D8IstioVersionIsIncompatibleWithK8sVersion CE S3

The installed Istio version is incompatible with the Kubernetes version.

The installed Istio version VERSION_NUMBER may not work properly with the current Kubernetes version VERSION_NUMBER because it’s not supported officially.

To resolve the issue, upgrade Istio following the guide.
IstioIrrelevantExternalServiceFound CE S5

External service found with irrelevant ports specifications.

A service NAME in the NAMESPACE namespace has an irrelevant port specification.

The .spec.ports[] field isn’t applicable for services of the ExternalName type. However, Istio renders port listeners for external services as 0.0.0.0:port, which captures all traffic to the specified port. This can cause problems for services that aren’t registered in the Istio registry.

To resolve the issue, remove the .spec.ports section from the service configuration. It is safe.

Module kube-dns

KubernetesCoreDNSHasCriticalErrors CE S5
Critical errors found in CoreDNS.
Deckhouse has detected at least one critical error in the CoreDNS pod POD_NAME.

To resolve the issue, review the container logs:
d8 k -n kube-system logs POD_NAME

Module metallb

D8MetallbBGPSessionDown EE S4
MetalLB BGP session is down.
JOB_NAME, MetalLB CONTAINER_NAME on POD_NAME has BGP session PEER down.

Check the logs for details:
d8 k -n d8-metallb logs daemonset/speaker -c speaker
D8MetallbConfigNotLoaded EE S4
The MetalLB configuration hasn't been loaded.
JOB_NAME, MetalLB CONTAINER_NAME on POD_NAME hasn’t been loaded.

To find the cause of the issue, review the controller logs:
d8 k -n d8-metallb logs deploy/controller -c controller
D8MetallbConfigStale EE S4
MetalLB is running on a stale configuration.
JOB_NAME, MetalLB CONTAINER_NAME on POD_NAME is running on a stale configuration because the latest configuration failed to load.

To find the cause of the issue, review the controller logs:
d8 k -n d8-metallb logs deploy/controller -c controller
D8MetallbNotSupportedServiceAnnotationsDetected SE S4
The annotation 'ANNOTATION_NAME' has been deprecated for the service 'NAME' in the 'NAMESPACE' namespace.
The annotation ‘ANNOTATION_NAME’ has been deprecated for the service ‘NAME’ in the ‘NAMESPACE’ namespace.

The following service annotations are no longer effective:
- metallb.universe.tf/ip-allocated-from-pool: Remove this annotation.
- metallb.universe.tf/address-pool: Replace it with the .spec.loadBalancerClass parameter or use the network.deckhouse.io/metal-load-balancer-class annotation, referencing the appropriate MetalLoadBalancerClass.
- metallb.universe.tf/loadBalancerIPs: Replace it with network.deckhouse.io/load-balancer-ips: IP.
- metallb.universe.tf/allow-shared-ip: Replace it with network.deckhouse.io/load-balancer-shared-ip-key.
- metallb.io/ip-allocated-from-pool: Remove this annotation.
- metallb.io/address-pool: Replace it with the .spec.loadBalancerClass parameter or use the network.deckhouse.io/metal-load-balancer-class annotation, referencing the appropriate MetalLoadBalancerClass.
- metallb.io/loadBalancerIPs: Replace it with network.deckhouse.io/load-balancer-ips: IP.
- metallb.io/allow-shared-ip: Replace it with network.deckhouse.io/load-balancer-shared-ip-key.
Please note. Existing LoadBalancer services of Deckhouse have been migrated automatically, but the new ones will not be.
D8MetallbObsoleteLayer2PoolsAreUsed SE S7

The metallb module has obsolete layer2 pools configured.

In ModuleConfig version 2, addressPool ‘NAME’ of type “layer2” are ignored. They should be removed from the configuration.
D8MetallbUpdateMCVersionRequired SE S5

The metallb ModuleConfig settings are outdated.

D8 MetalLB settings are outdated.

To resolve this issue, increase version in the ModuleConfig metallb.
L2LoadBalancerOrphanServiceFound SE S4

Orphaned service with an irrelevant L2LoadBalancer name has been found.

The cluster contains an orphaned service NAME in the NAMESPACE namespace with an irrelevant L2LoadBalancer name.

To resolve this issue, verify the L2LoadBalancer name specified in the annotation network.deckhouse.io/l2-load-balancer-name.

Module multitenancy-manager

ClusterResourceGrantPolicyViolation CE S6

One or more resources not allowed by ClusterResourceGrantPolicy are in use.

One or more resources that are not allowed by ClusterResourceGrantPolicy are in use within this cluster’s projects.

Check the “Security > Cluster Resource Grant Violations” dashboard for detailed overview of violating objects.
MultitenancyManagerProjectTemplateNameReserved CE S6
The user ProjectTemplate XXX has a name reserved for a built-in template. The Deckhouse update will overwrite it with the built-in one.
The cluster has a user ProjectTemplate named XXX. The next Deckhouse release adds a built-in template with this name that creates only the project namespace.

The controller keeps built-in templates in sync with its own copies. After the update, it overwrites the spec, labels and annotations of the existing template named XXX with those of the built-in template. The user template is lost, including its resourcesTemplate and parametersSchema.

The projects that use the template then switch from the Helm text of the user template to the structured fields of the built-in one. The AuthorizationRule objects with the Admin access level of the administrators are replaced by the d8-administrators ProjectRoleBinding. A ResourceQuota such as all-pods is replaced by the d8-project-quota ResourceQuota when .spec.quota of the project sets each of its limits to the same value or a lower one. The replaced objects are deleted only once their replacements work, so the administrators keep their access without a gap and the namespace keeps its quota. From then on, the administrators have the d8:project:admin role instead of the Admin access level, and the two do not grant the same permissions.

Every other object that the Helm text created, apart from the project namespace, stays in place and keeps working, but the project no longer manages it. Such an object gets the projects.deckhouse.io/kept-from-helm-template label and is listed in .status.keptObjects of the project, and the MultitenancyManagerProjectKeepsHelmTemplateObjects alert names the project.

The project namespace stays managed by the project. The labels and annotations that the Helm text put on it are removed, except those that the namespace parameter of the built-in template sets. The pod security, monitoring and scanning labels (security.deckhouse.io/pod-policy, extended-monitoring.deckhouse.io/enabled and security-scanning.deckhouse.io/enabled) cannot be set that way, so the projects that need them use a renamed copy of the template or another template.

The built-in template declares only the boolean requiredRequests parameter and the namespace parameter with the labels and annotations of the project namespace. It does not check the administrators and resourceQuota parameters. The update moves them to .spec.administrators and .spec.quota where it can, and a value it cannot move stays in the parameters without effect. A project with any other parameter, or with a requiredRequests or namespace value of another form, goes to the Error state. It is not rendered until its parameters are changed, and its objects stay as they are until then.

To keep the template, move it to a name of its own before updating Deckhouse:
1. Create a copy of the template under a new name, for example, my-simple:
  
  d8 k get projecttemplates XXX -o json \ | jq '{apiVersion, kind, metadata: {name: "my-simple"}, spec}' \ | d8 k create -f -
  
  If the template is applied from a Git repository or by CI, rename it there as well.
2. Switch every project that uses the template to the copy, and make the same change in the Git repository or CI process that applies the projects:
  
  d8 k get projects -l projects.deckhouse.io/project-template=XXX d8 k patch project PROJECT_NAME --type=merge -p '{"spec":{"projectTemplateName":"my-simple"}}'
  
  The copy renders the same objects, so the objects in the project namespaces stay in place. The exception is a Helm text that puts the template name (.projectTemplateName) into object names. Such objects are recreated under the new name.
3. When no project uses the template XXX any more, delete it:
  
  d8 k delete projecttemplates XXX
If the projects are to switch to the built-in template instead, save the user template before updating Deckhouse, for example with d8 k get projecttemplates XXX -o yaml > XXX.yaml, and check the parameters of the projects that use it.

Module node-local-dns

D8NodeLocalDNSCacheDenialEvictionsHigh BE S6

node-local-dns denial cache is being frequently evicted.

The denial (NXDOMAIN) cache in node-local-dns is evicting entries frequently.

This might indicate that there are Services that are being recreated too often in cluster. So, denial cache size is not enough (denial 65536 is currently configured).

Affected node: NODE_NAME
D8NodeLocalDNSCacheSuccessEvictionsHigh BE S6

node-local-dns success cache is being frequently evicted.

The success cache in node-local-dns is evicting entries frequently (>100 in 5 minutes).

This could mean there are too many different DNS records application have to resolve (to many Services?). So, node-local-dns cache capacity is too small (success 120000 is currently configured), or traffic patterns changed.

Affected node: NODE_NAME
D8NodeLocalDNSForwardMaxConcurrentRejects BE S5

The forward plugin is rejecting DNS queries due to max concurrency limit.

More than 20 queries were rejected within the last 5 minutes on node NODE_NAME because the max concurrent query limit was reached.
D8NodeLocalDNSKubeforwardRequestLatencyP95High BE S4
High DNS request latency in kubeforward (p95 > 300ms under load).
The 95th percentile DNS request latency for A-records through kubeforward exceeded 300 milliseconds over the last 5 minutes while handling more than 0.1 QPS.

This may indicate slow upstreams, network issues, or DNS overload on this node.

Affected node: NODE_NAME

For more detailed diagnostics, inspect Top 5 upstreams by slow kubeforward requests in Grafana.

If the alert fires only on one node, run the checks below from node NODE_NAME (for example, from a debug pod on this node). If the alert fires on many nodes, treat it as a potential systemic issue first.

Get kubeforward upstream IPs from a master host:
d8 k -n kube-system get po -l k8s-app=kube-dns -o jsonpath='{.items[*].status.podIP}'
Then run latency checks on the affected node NODE_NAME (or from a debug pod on that node):
UPSTREAMS="paste kube-dns pod IPs here" for ip in $UPSTREAMS; do echo "== upstream ${ip} ==" for i in $(seq 1 20); do dig @"$ip" -p 5353 kubernetes.default.svc.cluster.local A +tries=1 +time=1 +stats | grep "Query time" done done
To identify slow requests, enable module logs (enableLogs) and inspect slow query messages:
pod=$(kubectl -n kube-system get po -l k8s-app=node-local-dns --field-selector spec.nodeName=NODE_NAME -o jsonpath='{.items[0].metadata.name}') d8 k -n kube-system logs "$pod" | grep '\[kubeforward\] slow query'
Example log line: [kubeforward] slow query A kubernetes.default.svc.cluster.local. took 425ms (rcode=NOERROR, upstream=10.96.0.10:53)

Interpretation:
- If most upstreams are slow and many nodes are affected, suspect systemic network/CoreDNS overload.
- If one upstream is slow or only one node is affected, suspect a localized issue.
D8NodeLocalDNSUpstreamHealthcheckFailed BE S5

Too many upstream health check failures in forward plugin.

The health check system of the forward plugin failed more than 10 times within 5 minutes on node NODE_NAME. As a result, all upstreams were considered unhealthy.

Module node-manager

CapsInstanceUnavailable CE S8
There are unavailable instances in the MACHINE_DEPLOYMENT_NAME MachineDeployment.
The MachineDeployment MACHINE_DEPLOYMENT_NAME has VALUE unavailable instances.

Check the status of the instances in the cluster with the following command:
d8 k get instance -l node.deckhouse.io/group=MACHINE_DEPLOYMENT_NAME
ClusterHasOrphanedDisks CE S6

Cloud data discoverer found orphaned disks in the cloud.

The cloud data discoverer has found disks in the cloud that do not have a corresponding PersistentVolume in the cluster.

You can safely delete these disks manually from your cloud provider:

ID: ID, Name: NAME
D8BashibleApiserverLocked CE S6
Bashible-apiserver has been locked for too long.
Bashible-apiserver has been locked for an extended period.

To resolve the issue, check if the bashible-apiserver Pods are up-to-date and running:
d8 k -n d8-cloud-instance-manager get pods -l app=bashible-apiserver
D8CloudDataDiscovererCloudRequestError CE S6
Cloud data discoverer cannot get data from the cloud.
Refer to the cloud data discoverer’s logs for details:
d8 k -n NAMESPACE logs deploy/cloud-data-discoverer
D8CloudDataDiscovererSaveError CE S6
Cloud data discoverer cannot save data to a Kubernetes resource.
Refer to the cloud data discoverer’s logs for details:
d8 k -n NAMESPACE logs deploy/cloud-data-discoverer
D8ClusterAutoscalerManagerPodIsNotReady CE S8
The POD_NAME Pod is NOT Ready.
To check the Pod’s status, run the following command:
d8 k -n NAMESPACE logs POD_NAME
D8ClusterAutoscalerPodIsNotRunning CE S8
The cluster-autoscaler Pod is NOT Running.
The POD_NAME Pod is STATUS.

To check the Pod’s status, run the following command:
d8 k -n NAMESPACE get pods POD_NAME -o json | jq .status
D8ClusterAutoscalerPodIsRestartingTooOften CE S9
Too many cluster-autoscaler restarts detected.
The cluster-autoscaler has restarted VALUE times in the past hour.

Frequent restarts may indicate a problem. The cluster-autoscaler is expected to run continuously without interruption.

Check the logs for details:
d8 k -n d8-cloud-instance-manager logs -f -l app=cluster-autoscaler -c cluster-autoscaler
D8ClusterAutoscalerTargetAbsent CE S8
Cluster-autoscaler target is missing in Prometheus.
The cluster-autoscaler automatically scales nodes in the cluster. If it’s unavailable, it will be impossible to add new nodes when there’s not enough resources for scheduling Pods. It may also lead to unnecessary cloud costs due to unused but still provisioned cloud instances.

To resolve the issue, follow these steps:
1. Check availability and status of cluster-autoscaler Pods:
  
  d8 k -n d8-cloud-instance-manager get pods -l 'app in (cluster-autoscaler,cluster-autoscaler-mcm)'
2. Verify that the cluster-autoscaler Deployment exists:
  
  d8 k -n d8-cloud-instance-manager get deploy -l 'app in (cluster-autoscaler,cluster-autoscaler-mcm)'
3. Check the Deployment’s status:
  
  d8 k -n d8-cloud-instance-manager describe deploy deployment-name
D8ClusterAutoscalerTargetDown CE S8

Prometheus is unable to scrape cluster-autoscaler’s metrics.
D8ClusterAutoscalerTooManyErrors CE S8
Cluster-autoscaler is issuing too many errors.
The cluster-autoscaler encountered multiple errors from the cloud provider when attempting to scale the cluster.

Check the logs for details:
d8 k -n d8-cloud-instance-manager logs -f -l app=cluster-autoscaler -c cluster-autoscaler
D8MachineControllerManagerPodIsNotReady CE S8

The POD_NAME Pod is NOT Ready.
D8MachineControllerManagerPodIsNotRunning CE S8
The machine-controller-manager Pod is NOT Running.
The POD_NAME Pod is STATUS.

To check the Pod’s status, run the following command:
d8 k -n NAMESPACE get pods POD_NAME -o json | jq .status
D8MachineControllerManagerPodIsRestartingTooOften CE S9
Too many machine-controller-manager restarts detected.
The machine-controller-manager has restarted VALUE times in the past hour.

Frequent restarts may indicate a problem. The machine-controller-manager is expected to run continuously without interruption.

Check the logs for details:
d8 k -n d8-cloud-instance-manager logs -f -l app=machine-controller-manager -c controller
D8MachineControllerManagerTargetAbsent CE S8
Machine-controller-manager target is missing in Prometheus.
Machine-controller-manager controls ephemeral nodes in the cluster. If it becomes unavailable, it will be impossible to create or delete nodes in the cluster.

To resolve the issue, follow these steps:
1. Check availability and status of machine-controller-manager Pods:
  
  d8 k -n d8-cloud-instance-manager get pods -l app=machine-controller-manager
2. Verify availability of the machine-controller-manager Deployment:
  
  d8 k -n d8-cloud-instance-manager get deployment machine-controller-manager
3. Check the Deployment’s status:
  
  d8 k -n d8-cloud-instance-manager describe deployment machine-controller-manager
D8MachineControllerManagerTargetDown CE S8

Prometheus is unable to scrape the machine-controller-manager’s metrics.
D8NeedMigrateStateToOpenTofu CE S4
Terraform state must be migrated to OpenTofu.
This likely means the automatic migration to OpenTofu was canceled due to destructive changes detected in the cluster state.

Verify the following:
- Current Terraform version:
  
  d8 k -n d8-system exec -it deployments/terraform-state-exporter -c exporter -- terraform version
- Version in the Terraform state:
  
  d8 k -n d8-system get secret d8-cluster-terraform-state -o json | jq -r '.data["cluster-tf-state.json"] | @base64d | fromjson | .terraform_version'
- Check for destructive changes:
  
  d8 k exec -it deploy/terraform-state-exporter -n d8-system -- dhctl terraform check
To resolve the issue and migrate to OpenTofu manually, follow these steps:
1. Using the install container with the previous major Deckhouse release (for example, 1.69.X, while the cluster is now on 1.70.X), run:
  
  dhctl converge
  
  This should resolve all destructive changes in the cluster.
2. After that, the terraform-auto-converger Pod should complete the migration automatically.
If the automatic migration doesn’t happen, use the install container with the current Deckhouse release and run the following command to enforce migration of the Terraform state to OpenTofu:
dhctl converge-migration
If this alert appears in a cluster created by Deckhouse Commander, it means the cluster has been updated to a new Deckhouse version without approving state changes (for example, destructive changes).

To resolve this, follow these steps:
1. In Deckhouse Commander, click Delete manually to temporarily remove the cluster from control.
2. In the cluster, remove the commander-agent module using the following command:
  
  d8 k delete mc commander-agent
3. Using the install container with the previous major Deckhouse release (for example, 1.69.X, while the cluster is now on 1.70.X), run the following command to resolve all destructive changes in the cluster:
  
  dhctl converge
4. In Deckhouse Commander, click Attach to reattach the cluster.
D8NodeCgroupV2NotSupported CE S8

Node NODE_NAME is using deprecated cgroup v1 – migration to v2 required.

Node NODE_NAME in NodeGroup NODE_GROUP_NAME does not support cgroup v2. Starting from Kubernetes 1.35, cgroup v1 is deprecated. To migrate to cgroup v2, please upgrade the kernel to version 5.8 or newer. For more information, please refer to the documentation.
D8NodeContainerdV2NotSupported CE S8

Node NODE_NAME is using deprecated containerd v1 – migration to v2 required.

Node NODE_NAME in NodeGroup NODE_GROUP_NAME does not support containerd v2. Starting from Kubernetes 1.37, containerd v1.y will no longer be supported. Please check the requirements for containerd v2 and schedule your migration plan.

The exact reason is listed in the node.deckhouse.io/containerd-v2-err annotation on the node.

If the reason is erofs and the node runs Ubuntu in AWS, the module is shipped in a separate linux-modules-extra package. DKP installs it automatically, but for some kernels, mostly outdated ones, the package is no longer published in the APT repository. In that case, the installation is skipped and a warning is written to the node logs. The problem has been confirmed for the 5.15.0-1028-aws, 6.8.0-1024-aws, and 6.8.0-1029-aws kernels. Switch the node group to a newer AMI or kernel version that includes erofs.
D8NodeControllerReconcileErrorsHigh CE S8
Node-controller reconcile errors are high for CONTROLLER_NAME.
Controller CONTROLLER_NAME has a reconcile error rate above 0.1 errors per second for more than 10 minutes.

Check node-controller logs:
kubectl -n d8-cloud-instance-manager logs deploy/node-controller --all-pods=true
D8NodeControllerReconcileP99High CE S8
Node-controller reconcile p99 is high for CONTROLLER_NAME.
The p99 reconcile duration for controller CONTROLLER_NAME has been above 5 seconds for more than 10 minutes.

This may indicate slow reconciliation, Kubernetes API latency, or degraded controller performance.

To inspect node-controller logs:
kubectl -n d8-cloud-instance-manager logs deploy/node-controller --all-pods=true
D8NodeControllerWorkqueueDepthHigh CE S8
Node-controller workqueue NAME has a high backlog.
The node-controller workqueue NAME has more than 50 queued objects for more than 15 minutes.

This may indicate that the controller cannot process reconciliation events fast enough.

To inspect node-controller logs:
kubectl -n d8-cloud-instance-manager logs deploy/node-controller --all-pods=true
D8NodeGroupIsNotUpdating CE S8

NodeGroup NODE_GROUP_NAME is not handling the update correctly.

There is a new update available for Nodes in the NODE_GROUP_NAME NodeGroup. Although Nodes have detected the update, none of them have received approval to start the update process.

Most likely, there is a problem with the update_approval hook of the node-manager module.
D8NodeIsNotUpdating CE S7
Node NODE_NAME cannot complete the update.
Node NODE_NAME in NodeGroup NODE_GROUP_NAME detected a new update, requested and received approval, started the update, and encountered a step that could cause downtime. The update manager (the update_approval hook of the node-group module) granted downtime approval, but no success message was received, which indicates that the update has not completed.

To investigate the details, view Bashible logs on the Node:
journalctl -fu bashible
D8NodeIsNotUpdating CE S8
Node NODE_NAME cannot complete the update.
Node NODE_NAME in NodeGroup NODE_GROUP_NAME has detected a new update, requested and received approval, but failed to complete the update.

To investigate the details, view Bashible logs on the Node:
journalctl -fu bashible
D8NodeIsNotUpdating CE S9
Node NODE_NAME is not updating.
Node NODE_NAME in NodeGroup NODE_GROUP_NAME has a pending update but it’s neither receiving it nor attempting to.

Most likely, Bashible is not handling the update correctly. It should annotate the Node with update.node.deckhouse.io/waiting-for-approval before the update can proceed.

Options to investigate the details:
- Check the expected configuration checksum for the NodeGroup:
  
  d8 k -n d8-cloud-instance-manager get secret configuration-checksums -o jsonpath={.data.NODE_GROUP_NAME} | base64 -d
- Check the current configuration checksum on the Node:
  
  d8 k get node NODE_NAME -o jsonpath='{.metadata.annotations.node\.deckhouse\.io/configuration-checksum}'
- View Bashible logs on the Node:
  
  journalctl -fu bashible
D8NodeIsUnmanaged CE S9

The NODE_NAME Node is not managed by the node-manager module.

The NODE_NAME Node is not managed by the node-manager module.

To resolve this issue, follow the instructions how to clean up the node and add it to the cluster.
D8NodeUpdateStuckWaitingForDisruptionApproval CE S8

Node NODE_NAME cannot obtain disruption approval.

Node NODE_NAME in NodeGroup NODE_GROUP_NAME detected a new update, requested and received initial approval, and started the update. However, it reached a stage that could cause downtime and was unable to obtain disruption approval. A disruption approval is normally issued automatically by the update_approval hook of the node-manager module.

To resolve this issue, investigate why the approval could not be granted to proceed with the update.
D8ProblematicNodeGroupConfiguration CE S8
Node NODE_NAME cannot begin the update.
There is a new update available for Nodes in the NODE_GROUP_NAME NodeGroup. However, Node NODE_NAME cannot begin the update.

The Node is missing the node.deckhouse.io/configuration-checksum annotation, which may indicate that its bootstrap process did not complete correctly.

Troubleshooting options:
- Check the cloud-init log (/var/log/cloud-init-output.log) on the node.
- Check the NodeGroupConfiguration resource associated with the NODE_GROUP_NAME NodeGroup for potential issues.
NodeGroupHasStaticInternalNetworkCIDRsField CE S9

NodeGroup NAME uses deprecated field spec.static.internalNetworkCIDRs.

The spec.static.internalNetworkCIDRs field is deprecated and has been moved to the static cluster configuration.

To resolve this issue, delete the field from NodeGroup NAME. This is safe, as the setting has already been migrated automatically.
NodeGroupMasterTaintIsAbsent CE S4
The master NodeGroup is missing the required control-plane taint.
The master NodeGroup doesn’t have the node-role.kubernetes.io/control-plane: NoSchedule taint. This may indicate a misconfiguration where control-plane nodes can run non-control-plane Pods.

To resolve the issue, add the following to the master NodeGroup spec:
nodeTemplate: taints: - effect: NoSchedule key: node-role.kubernetes.io/control-plane
Note that the taint key: node-role.kubernetes.io/master is deprecated and has no effect starting from Kubernetes 1.24.
NodeGroupReplicasUnavailable CE S7
NodeGroup NODE_GROUP_NAME has no available instances.
This probably means that machine-controller-manager is unable to create Machines using the cloud provider module.

Possible causes:
1. Cloud provider resource limits.
2. No access to the cloud provider API.
3. Misconfiguration of the cloud provider or instance class.
4. Problems with bootstrapping the Machine.
Recommended actions:
1. Check the status of the NodeGroup:
  
  d8 k get ng NODE_GROUP_NAME -o yaml
  
  Look for errors in the .status.lastMachineFailures field.
2. If no Instances stay in the Pending state for more than a couple of minutes, it likely means that Instances are being continuously created and deleted due to an error:
  
  d8 k get instances
3. If logs don’t show errors, and an Instance continues to be Pending, check its bootstrap status:
  
  d8 k get instances INSTANCE_NAME -o yaml | grep 'bootstrapStatus' -B0 -A2
4. If the output looks like the example below, use the logsEndpoint (or the command in description) to examine bootstrap logs:
  
  bootstrapStatus: description: Use 'curl -N http://192.168.199.158:8000' to get bootstrap logs. logsEndpoint: http://192.168.199.158:8000
5. If there’s no bootstrap log endpoint, cloudInit may not be working correctly. This could indicate a misconfigured instance class in the cloud provider.
NodeGroupReplicasUnavailable CE S8
Too many unavailable instances in the NODE_GROUP_NAME NodeGroup.
The number of simultaneously unavailable instances in the NODE_GROUP_NAME NodeGroup exceeds the allowed threshold. This may indicate that the autoscaler has provisioned too many nodes at once. Check the state of the Machine in the cluster. This probably means that machine-controller-manager is unable to create Machines using the cloud provider module.

Possible causes:
1. Cloud provider resource limits.
2. No access to the cloud provider API.
3. Misconfiguration of the cloud provider or instance class.
4. Problems with bootstrapping the Machine.
Recommended actions:
1. Check the status of the NodeGroup:
  
  d8 k get ng NODE_GROUP_NAME -o yaml
  
  Look for errors in the .status.lastMachineFailures field.
2. If no Instances stay in the Pending state for more than a couple of minutes, it likely means that Instances are being continuously created and deleted due to an error:
  
  d8 k get instances
3. If logs don’t show errors, and an Instance continues to be Pending, check its bootstrap status:
  
  d8 k get instances INSTANCE_NAME -o yaml | grep 'bootstrapStatus' -B0 -A2
4. If the output looks like the example below, use the logsEndpoint (or the command in description) to examine bootstrap logs:
  
  bootstrapStatus: description: Use 'curl -N http://192.168.199.158:8000' to get bootstrap logs. logsEndpoint: http://192.168.199.158:8000
5. If there’s no bootstrap log endpoint, cloudInit may not be working correctly. This could indicate a misconfigured instance class in the cloud provider.
NodeGroupReplicasUnavailable CE S8
NodeGroup NODE_GROUP_NAME has unavailable instances.
There are VALUE unavailable instances in the NodeGroup NODE_GROUP_NAME. This probably means that machine-controller-manager is unable to create Machines using the cloud provider module.

Possible causes:
1. Cloud provider resource limits.
2. No access to the cloud provider API.
3. Misconfiguration of the cloud provider or instance class.
4. Problems with bootstrapping the Machine.
Recommended actions:
1. Check the status of the NodeGroup:
  
  d8 k get ng NODE_GROUP_NAME -o yaml
  
  Look for errors in the .status.lastMachineFailures field.
2. If no Instances stay in the Pending state for more than a couple of minutes, it likely means that Instances are being continuously created and deleted due to an error:
  
  d8 k get instances
3. If logs don’t show errors, and an Instance continues to be Pending, check its bootstrap status:
  
  d8 k get instances INSTANCE_NAME -o yaml | grep 'bootstrapStatus' -B0 -A2
4. If the output looks like the example below, use the logsEndpoint (or the command in description) to examine bootstrap logs:
  
  bootstrapStatus: description: Use 'curl -N http://192.168.199.158:8000' to get bootstrap logs. logsEndpoint: http://192.168.199.158:8000
5. If there’s no bootstrap log endpoint, cloudInit may not be working correctly. This could indicate a misconfigured instance class in the cloud provider.
NodeGroupUsesDeprecatedInCoreGPU CE S9

NodeGroup NAME uses the deprecated built-in GPU support.

NodeGroup NAME has spec.gpu configured while the gpu module is disabled. As a result, GPU nodes in the NodeGroup use the built-in GPU support, which is deprecated and will be removed in a future DKP release.

To use the supported GPU implementation, enable the gpu module.
NodeRequiresDisruptionApprovalForUpdate CE S8
Node NODE_NAME requires disruption approval to proceed with the update.
Node NODE_NAME in NodeGroup NODE_GROUP_NAME has detected a new update, received initial approval, and started the update. However, it encountered a stage that may cause downtime and requires manual disruption approval, because the NodeGroup is configured with disruptions.approvalMode: Manual.

To resolve the issue, ensure the Node is ready for unsafe updates (drained) and grant disruption approval by annotating the Node with update.node.deckhouse.io/disruption-approved=.

Caution:
- Nodes in manual mode aren’t drained automatically.
- Do not drain master nodes.
1. To drain the Node and grant the update approval, run the following command:
  
  d8 k drain NODE_NAME --delete-emptydir-data --ignore-daemonsets --force=true && d8 k annotate node NODE_NAME update.node.deckhouse.io/disruption-approved=
2. Uncordon the Node once the update is complete and the annotation update.node.deckhouse.io/approved is removed:
  
  while d8 k get node NODE_NAME -o json | jq -e '.metadata.annotations | has("update.node.deckhouse.io/approved")' > /dev/null; do sleep 1; done d8 k uncordon NODE_NAME
If the NodeGroup has multiple Nodes, repeat this process for each one, since only one Node is updated at a time. Consider temporarily switching to automatic disruption approval (disruptions.approvalMode: Automatic).
NodeStuckInDraining CE S6
Node NODE_NAME is stuck in draining.
Node NODE_NAME in NodeGroup NODE_GROUP_NAME is stuck in the draining process.

To get more details, run the following:
d8 k -n default get event --field-selector involvedObject.name=NODE_NAME,reason=DrainFailed --sort-by='.metadata.creationTimestamp'
Error message: MESSAGE_CONTENTS
NodeStuckInDrainingForDisruptionDuringUpdate CE S6
Node NODE_NAME is stuck in draining.
Node NODE_NAME in NodeGroup NODE_GROUP_NAME has detected a new update, requested and received approval, started the update, and reached a step that could cause downtime. It is currently stuck in the draining process while waiting for automatic disruption approval.

To get more details, run the following:
d8 k -n default get event --field-selector involvedObject.name=NODE_NAME,reason=ScaleDown --sort-by='.metadata.creationTimestamp'
StandbyHolderDeploymentReplicasUnavailable CE S5

No available replicas in Deployment standby-holder-NODE_GROUP_NAME.

Deckhouse has detected that there are no available replicas in the standby-holder-NODE_GROUP_NAME Deployment in the d8-cloud-instance-manager namespace.

Check the Deployment and associated Pods for issues.
UnmetCloudConditions CE S6
Deckhouse update is unavailable due to unmet cloud provider conditions.
Deckhouse has detected that some cloud provider–specific conditions have not been met. Until these issues are resolved, updating to the new Deckhouse release is not possible.

Troubleshooting details:
- Name: NAME
- Message: MESSAGE_CONTENTS

Module okmeter

D8OkmeterAgentPodIsNotReady CE S6

Okmeter agent is not Ready.

Module openvpn

OpenVPNClientCertificateExpired CE S4
OpenVPN client certificate expired for XXX
The OpenVPN client certificate for XXX has expired.

Check the certificate information and make sure it actually expires:
d8 k -n d8-openvpn get secrets -l name="XXX" -o jsonpath='{.items[0].data.tls\.crt}' | base64 -d | openssl x509 -text -noout
Renew or delete the expired certificate. Check the documentation for details.
OpenVPNClientCertificateExpired CE S5
OpenVPN client certificate expires for XXX in less than 7 days.
The OpenVPN client certificate for XXX is set to expire in VALUE days.

Renewal of certificate required. Check the documentation for details.

View certificate details to display the exact expiration date:
d8 k -n d8-openvpn get secrets -l name=XXX -o jsonpath='{.items[0].data.tls\.crt}' | base64 -d | openssl x509 -text -noout
OpenVPNClientCertificateExpired CE S6
OpenVPN client certificate expires for XXX in less than 30 days.
The OpenVPN client certificate for XXX is set to expire in VALUE days.

Renewal of certificate required. Check the documentation for details.

View certificate details to display the exact expiration date:
d8 k -n d8-openvpn get secrets -l name=XXX -o jsonpath='{.items[0].data.tls\.crt}' | base64 -d | openssl x509 -text -noout
OpenVPNServerCACertificateExpired CE S4
OpenVPN CA certificate has expired.
OpenVPN CA certificate renews automatically 1 days before expiration. Automatic rotation did not work. Check the module status or perform manual rotation.

View certificate details to display the exact expiration date:
d8 k -n d8-openvpn get secrets openvpn-pki-ca -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout
The hook should rotate CA and server certificates before expiry.
OpenVPNServerCACertificateExpiresTomorrow CE S4
OpenVPN CA certificate will expire tomorrow
OpenVPN CA certificate renews automatically 1 days before expiration. Automatic rotation did not work. Check the module status or perform manual rotation.

View certificate details to display the exact expiration date:
d8 k -n d8-openvpn get secrets openvpn-pki-ca -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text -noout
The hook should rotate CA and server certificates before expiry.
OpenVPNServerCACertificateExpiringInAWeek CE S5
OpenVPN CA certificate expires in less than 7 days.
The OpenVPN CA certificate is set to expire in VALUE days.

Renew the CA certificate if necessary. After renewing the certificate, you will need to reissue all client and server certificates. Check the documentation for details.

View certificate details to display the exact expiration date:
d8 k -n d8-openvpn get secrets -l name=server -o jsonpath='{.items[0].data.tls\.crt}' | base64 -d | openssl x509 -text -noout
OpenVPNServerCACertificateExpiringSoon CE S5
OpenVPN CA certificate expires in less than 30 days.
The OpenVPN CA certificate is set to expire in VALUE days.

Renew the CA certificate if necessary. After renewing the certificate, you will need to reissue all client and server certificates. Check the documentation for details.

View certificate details to display the exact expiration date:
d8 k -n d8-openvpn get secrets -l name=server -o jsonpath='{.items[0].data.tls\.crt}' | base64 -d | openssl x509 -text -noout
OpenVPNServerCertificateExpired CE S4
OpenVPN server certificate has expired.
The OpenVPN server certificate has expired.

OpenVPN server certificate renews automatically 1 days before expiration. Automatic rotation did not work. Check the module status or perform manual rotation.

Check the certificate information and make sure it actually expires.
d8 k -n d8-openvpn get secrets -l name=server -o jsonpath='{.items[0].data.tls\.crt}' | base64 -d | openssl x509 -text -noout
OpenVPNServerCertificateExpiresTomorrow CE S4
OpenVPN server certificate will expire tomorrow.
The OpenVPN server certificate is set to expire in less than 1 day.

OpenVPN server certificate renews automatically 1 days before expiration. Automatic rotation did not work. Check the module status or perform manual rotation.

Check the certificate information and make sure it actually expires:
d8 k -n d8-openvpn get secrets -l name=server -o jsonpath='{.items[0].data.tls\.crt}' | base64 -d | openssl x509 -text -noout

Module secret-copier

D8SecretCopierDeprecatedLabels CE S9

Obsolete antiopa_secret_copier=yes label detected.

The secret-copier module has changed its labeling approach for original secrets in the default namespace.

The label antiopa-secret-copier: "yes" is deprecated and will be removed soon.

To resolve this issue, replace the label antiopa-secret-copier: "yes" with secret-copier.deckhouse.io/enabled: "" for all secrets used by the secret-copier module in the default namespace.

Module terraform-manager

D8TerraformStateExporterClusterStateChanged CE S8
Terraform-state-exporter cluster state change detected.
The current Kubernetes cluster state is STATUS_REFERENCE compared to the Terraform state.

It is important to reconcile the states.

Troubleshooting steps:
1. View the differences:
  
  dhctl terraform check
2. Apply the necessary changes to bring the cluster in sync:
  
  dhctl converge
D8TerraformStateExporterClusterStateError CE S8
Terraform-state-exporter cluster state error.
The terraform-state-exporter can’t check difference between the Kubernetes cluster state and the Terraform state.

That was likely caused by terraform-state-exporter failing to run Terraform with the current state and configuration.

Troubleshooting steps:
1. View the differences:
  
  dhctl terraform check
2. Apply the necessary changes to bring the cluster in sync:
  
  dhctl converge
D8TerraformStateExporterHasErrors CE S8
Terraform-state-exporter has encountered errors.
Errors occurred during the operation of the terraform-state-exporter.

To get more details, check the Pod logs:
d8 k -n d8-system logs -l app=terraform-state-exporter -c exporter
D8TerraformStateExporterNodeStateChanged CE S8
Terraform-state-exporter node state change detected.
The current state of node NODE_GROUP_NAME/NAME is STATUS_REFERENCE compared to the Terraform state.

It is important to reconcile the states.

Troubleshooting steps:
1. View the differences:
  
  dhctl terraform check
2. Apply the necessary changes to bring the cluster in sync:
  
  dhctl converge
D8TerraformStateExporterNodeStateError CE S8
Terraform-state-exporter node state error.
The terraform-state-exporter can’t check the difference between the node NODE_GROUP_NAME/NAME state and the Terraform state.

Probably, it occurred because terraform-manager had failed to run Terraform with the current state and configuration.

Troubleshooting steps:
1. View the differences:
  
  dhctl terraform check
2. Apply the necessary changes to bring the cluster in sync:
  
  dhctl converge
D8TerraformStateExporterNodeTemplateChanged CE S8
Terraform-state-exporter node template change detected.
The terraform-state-exporter has detected a mismatch between the node template in the cluster provider configuration and the one specified in the NodeGroup NAME`.

Node template is STATUS_REFERENCE.

Troubleshooting steps:
1. View the differences:
  
  dhctl terraform check
2. Adjust NodeGroup settings to fix the issue or bring the cluster in sync via the following command:
  
  dhctl converge
D8TerraformStateExporterPodIsNotReady CE S8
Terraform-state-exporter Pod is not Ready.
The terraform-state-exporter cannot check the difference between the actual Kubernetes cluster state and the Terraform state.

To resolve the issue, check the following:
1. Deployment description:
  
  d8 k -n d8-system describe deployment terraform-state-exporter
2. Pod status:
  
  d8 k -n d8-system describe pod -l app=terraform-state-exporter
D8TerraformStateExporterPodIsNotRunning CE S8
Terraform-state-exporter Pod is not Running.
The terraform-state-exporter cannot check the difference between the actual Kubernetes cluster state and the Terraform state.

To resolve the issue, check the following:
1. Deployment description:
  
  d8 k -n d8-system describe deployment terraform-state-exporter
2. Pod status:
  
  d8 k -n d8-system describe pod -l app=terraform-state-exporter
D8TerraformStateExporterTargetAbsent CE S8
Terraform-state-exporter target is missing in Prometheus.
Prometheus cannot find the terraform-state-exporter target.

To investigate the details:
- Check the Pod status:
  
  d8 k -n d8-system get pod -l app=terraform-state-exporter
- Check the container logs:
  
  d8 k -n d8-system logs -l app=terraform-state-exporter -c exporter
D8TerraformStateExporterTargetDown CE S8
Prometheus can't scrape terraform-state-exporter.
Prometheus is unable to scrape metrics from the terraform-state-exporter.

To investigate the details:
- Check the Pod status:
  
  d8 k -n d8-system get pod -l app=terraform-state-exporter
- Check the container logs:
  
  d8 k -n d8-system logs -l app=terraform-state-exporter -c exporter

Module user-authn

D8DexAllTargetsDown CE S6

Prometheus is unable to scrape Dex metrics.
D8DexAuthRequestsQuotaExhausted CE S3
Dex AuthRequest objects ResourceQuota is exhausted in namespace "d8-user-authn".
The ResourceQuota for authrequests.dex.coreos.com objects in namespace d8-user-authn has been fully exhausted.

Dex is currently unable to create new AuthRequest custom resources, which will cause:
- Dex health checks to fail immediately.
- Authentication through external identity providers to stop working.
- Users unable to log in via OAuth/OIDC flows.
Immediate actions required:
1. Check current AuthRequest objects and clean up stale ones:
  d8 k get authrequests.dex.coreos.com -n d8-user-authn --no-headers | wc -l d8 k delete authrequests.dex.coreos.com --all -n d8-user-authn
2. Temporarily increase the ResourceQuota limit:
  d8 k patch resourcequota authrequest-quota -n d8-user-authn --type='json' -p='[{"op": "replace", "path": "/spec/hard/count\/authrequests.dex.coreos.com", "value": "20000"}]'
3. Investigate why Dex is not cleaning up AuthRequest objects automatically.
4. Review Dex configuration for AuthRequest TTL settings.
D8DexAuthRequestsQuotaHigh CE S4
Dex AuthRequest objects quota usage is higher than 80% in namespace "d8-user-authn".
The ResourceQuota for authrequests.dex.coreos.com objects in namespace d8-user-authn is approaching its limit.

Current usage: XXX%

When the quota is exhausted, Dex will be unable to create new AuthRequest custom resources, which may cause:
- Dex health checks to fail.
- Authentication through external identity providers (GitLab, GitHub, etc.) to stop working.
Investigation steps:
1. Check for stale AuthRequest objects that Dex should have cleaned up:
  d8 k get authrequests.dex.coreos.com -n d8-user-authn
2. Review Dex configuration for AuthRequest TTL settings.
3. Consider temporarily increasing the ResourceQuota limit if needed.
D8DexProviderLDAPTLSConflict CE S8
DexProvider NAME has a conflicting LDAP TLS configuration.
DexProvider NAME combines mutually exclusive LDAP TLS settings: XXX. Dex silently ignores the redundant flag, which masks misconfiguration:
- insecureNoSSL + startTLS: TLS is disabled (ldap://, port 389), so startTLS is ignored.
- insecureNoSSL + insecureSkipVerify: TLS is disabled, there is no certificate to verify.
- insecureNoSSL + rootCAData: TLS is disabled, the CA chain is ignored.
The CRD schema rejects this combination on create/update; this DexProvider was created before the validation was added and remains active.

Update the DexProvider to remove the conflict:
- Plaintext LDAP (port 389): keep spec.ldap.insecureNoSSL: true, remove startTLS, insecureSkipVerify, and rootCAData.
- STARTTLS on port 389: set spec.ldap.insecureNoSSL: false, spec.ldap.startTLS: true. Use insecureSkipVerify or rootCAData as needed.
- ldaps:// on port 636: leave insecureNoSSL and startTLS at false. Use insecureSkipVerify or rootCAData as appropriate.
Audit all LDAP DexProvider objects:
kubectl get dexproviders.deckhouse.io -o yaml \ | yq '.items[] | select(.spec.type=="LDAP") | {name: .metadata.name, ldap: .spec.ldap | pick(["insecureNoSSL","startTLS","insecureSkipVerify","rootCAData"])}'
D8UserAuthnControllerReconcileErrorsHigh CE S8
user-authn-controller reconcile errors are high for CONTROLLER_NAME.
Controller CONTROLLER_NAME has a reconcile error rate above 0.1 errors per second for more than 10 minutes.

Check logs:
d8 k -n d8-user-authn logs deploy/user-authn-controller --all-pods=true
D8UserAuthnControllerTargetDown CE S6
Prometheus is unable to scrape user-authn-controller.
Prometheus cannot scrape metrics from user-authn-controller in namespace d8-user-authn.

The controller is down, not Ready (informer cache has not synced), or the PodMonitor is missing. Static user login, UserAccount, expire, and provider checks will not run.

Check:
d8 k -n d8-user-authn get pods -l app=user-authn-controller d8 k -n d8-user-authn describe pods -l app=user-authn-controller
D8UserAuthnControllerUnavailable CE S6
user-authn-controller has unavailable replicas.
Deckhouse has detected unavailable replicas in the user-authn-controller deployment in the d8-user-authn namespace.

While this alert is active, static Users may not get Dex Password objects, UserAccount projections stall, expired Users are not deleted, and DexProviderCheck / UserOperation stay pending.

Check deployment and pod states:
d8 k -n d8-user-authn get deploy user-authn-controller d8 k -n d8-user-authn get pods -l app=user-authn-controller d8 k -n d8-user-authn logs deploy/user-authn-controller --all-pods=true
D8UserAuthnIDTokenTTLTooLong CE S6
user-authn idTokenTTL is XXX; the next settings version lowers it to 5h59m.
The ModuleConfig user-authn sets idTokenTTL: XXX, which is 6 hours or more.

Dex is configured with expiry.signingKeys: 6h: every 6 hours it rotates the key used to sign ID tokens and keeps previous keys only while tokens signed with them can still be valid.

idTokenTTL maps to Dex expiry.idTokens. If that lifetime is 6 hours or more, an ID token outlives the rotation of the key that signed it, and Dex keeps and publishes one more previous key for every 6 hours of idTokenTTL. That is why idTokenTTL should be less than 6h.

The next version of the module settings (spec.version of the ModuleConfig) accepts only values less than 6h. When this ModuleConfig is converted to it, idTokenTTL is replaced with 5h59m, which Dex and DexAuthenticator then use. The alert fires until the ModuleConfig itself sets a value less than 6h.

Lowering idTokenTTL, in the ModuleConfig or by this conversion, has the following effects:
- DexAuthenticator sessions whose keepUsersLoggedInFor is less than the previous idTokenTTL become shorter. A session lasts keepUsersLoggedInFor or idTokenTTL + 1s, whichever is longer. For example, when idTokenTTL goes from 24h to 5h59m, a session with keepUsersLoggedInFor: 12h lasts 12 hours instead of 24 hours and 1 second.
- ID tokens that Dex signed before the change with the signing key current at that moment can be rejected about 6 to 12 hours after the change, before they expire, because Dex then keeps a replaced signing key only for as long as the new idTokenTTL requires. A client that refreshes an ID token only when it expires, such as kubectl with auth-provider: oidc in the kubeconfig, keeps sending the rejected token until the token expires. To make such a client request a new token, remove id-token from its kubeconfig.
To choose when this happens, lower idTokenTTL yourself, for example:
apiVersion: deckhouse.io/v1alpha1 kind: ModuleConfig metadata: name: user-authn spec: version: 2 settings: idTokenTTL: 1h

Module user-authz

D8UserAuthzLegacyRBACv2CapabilityBindingFound CE S7
The XXX XXX binds the capability XXX, which the update renames without an alias.
The XXX XXXXXX in the NAMESPACE namespaceXXX has no heritage: deckhouse label, and its roleRef field names the XXX capability directly.

In the new role model, the capabilities of the modules shipped with Deckhouse and of the user-authz module get new names, and the old names get no compatibility aliases. This capability becomes XXX. After the update, the XXX ClusterRole no longer exists, and the binding stops granting the permissions of the capability.

The names change as follows:
- d8:manage:permission:module:MODULE:ACTION becomes d8:system-capability:MODULE:ACTION.
- d8:use:capability:module:MODULE:ACTION becomes d8:namespace-capability:MODULE:ACTION.
- d8:manage:permission:subsystem:SUBSYSTEM:ACTION becomes d8:subsystem-capability:SUBSYSTEM:ACTION, where the kubernetes subsystem becomes cluster and networking becomes network.
- d8:use:capability:kubernetes:ACTION becomes d8:namespace-capability:kubernetes:ACTION.
Capabilities are building blocks that roles aggregate, and the new role model refuses a ClusterRoleBinding to a capability. Before the update, create a custom role of the new scheme that aggregates the capability, and bind the subjects of this binding to it:
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: d8:custom:SCOPE:NAME labels: rbac.deckhouse.io/kind: custom-role rbac.deckhouse.io/scope: SCOPE aggregationRule: clusterRoleSelectors: - matchLabels: rbac.deckhouse.io/kind: capability rbac.deckhouse.io/capability: XXX rules: []
Replace SCOPE with namespace for a namespace capability and with system for the others, and bind a role of the namespace scope with a RoleBinding. The capability gets the rbac.deckhouse.io/kind: capability and rbac.deckhouse.io/capability labels with the update, so the role aggregates nothing until then and gets the permissions of the capability with it. The new role model does not let a user put rbac.deckhouse.io/kind: capability on a ClusterRole, so the selector takes the built-in capability only. After the update, check the permissions of the subjects with d8 k auth can-i VERB RESOURCE --as=USER_NAME, and then delete this binding.

A module installed from a module source renames its capabilities in a release of its own, and the bindings to them are not listed in this alert. The following command lists the bindings to the capabilities of the experimental role model, those of such modules included:
d8 k get clusterrolebindings,rolebindings -A -o json | jq -r '.items[] | select(.metadata.labels.heritage != "deckhouse" and .roleRef.kind == "ClusterRole" and (.roleRef.name | test("^d8:(manage:permission|use:capability):"))) | "\(.kind) \(.metadata.namespace // "-") \(.metadata.name) -> \(.roleRef.name)"'
D8UserAuthzLegacyRBACv2CustomCapabilityFound CE S7
The custom capability NAME extends d8:use:role:XXX through the labels of the old scheme only.
The ClusterRole NAME has no heritage: deckhouse label and no aggregationRule field, and it carries the rbac.deckhouse.io/kind: use and rbac.deckhouse.io/aggregate-to-kubernetes-as: XXX labels. The d8:use:role:XXX roles aggregate it through these labels.

In the new label scheme of the role model, these roles and the d8:namespace:XXX role select the rbac.deckhouse.io/aggregate-to-namespace-as label instead. Without it, the capability adds nothing to them, and the subjects bound to them lose its permissions.

Relabel the capability now, and replace it after the update:
1. Add the label and keep the old ones, which the current roles read:
  
  d8 k label clusterrole NAME rbac.deckhouse.io/aggregate-to-namespace-as=XXX
  
  If the capability is applied from a Git repository, add the label to its manifest as well, so that the manifest describes what is in the cluster.
2. After the update, create the d8:custom:namespace-capability:NAME capability with the same rules and the rbac.deckhouse.io/kind: custom-capability, rbac.deckhouse.io/scope: namespace and rbac.deckhouse.io/aggregate-to-namespace-as: XXX labels, check the permissions of the subjects, and then delete this capability. After the update, the “Replacing a capability” section of the user-authz module FAQ gives the commands.
  
  Until the capability is replaced, the rbacv2-cluster-roles.deckhouse.io webhook of the new role model refuses every update of it, because it carries aggregation labels without rbac.deckhouse.io/kind: custom-capability. An apply of its manifest from a Git repository fails as well, and the D8UserAuthzForeignAggregationLabel alert names the capability.
For details, refer to the corresponding section in the user-authz module FAQ.

List all such capabilities:
d8 k get clusterroles -o json | jq -r '.items[] | select(.metadata.labels.heritage != "deckhouse" and .aggregationRule == null and .metadata.labels["rbac.deckhouse.io/kind"] == "use" and .metadata.labels["rbac.deckhouse.io/aggregate-to-kubernetes-as"] != null and .metadata.labels["rbac.deckhouse.io/aggregate-to-namespace-as"] == null) | .metadata.name'
D8UserAuthzLegacyRBACv2CustomRoleFound CE S7
The custom role NAME relies on labels of the experimental role model that lose their effect in the new one.
XXX The ClusterRole NAME has no heritage: deckhouse label, has an aggregationRule field and carries the rbac.deckhouse.io/kind: manage and rbac.deckhouse.io/use-role labels. For every ClusterRoleBinding to such a role, Deckhouse creates a RoleBinding to d8:use:role:LEVEL with the same subjects in each namespace of a module whose capabilities the role aggregates.

In the new role model, Deckhouse creates such RoleBinding objects only for a role with the rbac.deckhouse.io/scope: system or subsystem label, which a custom role of the old scheme does not have. The update deletes the RoleBinding objects of this role, and its subjects lose the access they give in the namespaces of the modules, such as reading the logs of the pods there. The selectors of the role do not rely on the labels that the new role model drops, so the role keeps its own permissions.

Before the update, bind d8:use:role:LEVEL explicitly in these namespaces. These are the RoleBinding objects with the heritage: deckhouse and rbac.deckhouse.io/automated: "true" labels whose rbac.deckhouse.io/related-with annotation names a ClusterRoleBinding to the role. Next to each of them, create a RoleBinding with the same namespace, roleRef, and subjects, under another name and without these labels. The role itself needs no change.

The explicit RoleBinding objects point at the d8:use:role:LEVEL roles, which the new scheme keeps only as deprecated aliases. Deckhouse does not install the release that follows the update and removes these aliases until the explicit RoleBinding objects are moved to d8:namespace:LEVEL. The role stays in this alert until it is replaced with a role of the new scheme after the update. XXX The ClusterRole NAME has no heritage: deckhouse label, and a selector of its aggregationRule field requires the rbac.deckhouse.io/kind: manage or use label of the experimental role model, or the aggregation label of a lineage that no role of the new role model collects and no custom capability carries.

In the new label scheme of the role model, the built-in capabilities do not carry the rbac.deckhouse.io/kind: manage and use labels. The new role model collects only the system, namespace and project lineages and the lineages of the iam, security, cluster, delivery, network, storage, observability and managed-services subsystems, and the modules label their capabilities only with these. A selector by another lineage, such as rbac.deckhouse.io/aggregate-to-all-as, rbac.deckhouse.io/aggregate-to-deckhouse-as or rbac.deckhouse.io/aggregate-to-virtualization-as, matches none of them either.

A selector by a lineage that a custom capability carries is not counted, because it keeps getting the rules of that capability after the update. A custom capability here is a ClusterRole without the heritage: deckhouse label and without an aggregationRule field. Neither is a selector by the lineage of the subsystem in the rbac.deckhouse.io/subsystem label of the role, because the custom capabilities of that subsystem carry it. Neither exception applies to the all, deckhouse, infrastructure, kubernetes and networking lineages, which do not count as custom ones.

Such selectors then match none of the built-in capabilities, the Kubernetes aggregation controller leaves the role without their rules, and its subjects lose these permissions. Custom roles get no compatibility aliases.

Freeze the role while the built-in capabilities still carry these labels, in this order:
1. Bind d8:use:role:LEVEL explicitly in the namespaces of the automatic RoleBinding objects the role leads to. These are the RoleBinding objects with the heritage: deckhouse and rbac.deckhouse.io/automated: "true" labels whose rbac.deckhouse.io/related-with annotation names a ClusterRoleBinding to the role, if the role has the rbac.deckhouse.io/use-role label, or to a role with the rbac.deckhouse.io/kind: manage label that the role is aggregated into, such as d8:manage:all:LEVEL. Next to each of them, create a RoleBinding with the same namespace, roleRef, and subjects, under another name and without these labels.
2. If the role is aggregated into another role, bind the role itself to the subjects of the ClusterRoleBinding objects to that other role. The built-in roles of the new scheme select other labels, so after the update those subjects keep the permissions of the role only through such a binding.
3. Only then remove the aggregationRule field from the role and keep its rules field:
  
  d8 k patch clusterrole NAME --type=json -p '[{"op": "remove", "path": "/aggregationRule"}]'
  
  Without aggregationRule, the role keeps the permissions it has at this moment. Deckhouse deletes its automatic RoleBinding objects, and the explicit ones from the first step keep the access in those namespaces. A frozen role that carries the rbac.deckhouse.io/kind: use and rbac.deckhouse.io/aggregate-to-kubernetes-as labels without rbac.deckhouse.io/aggregate-to-namespace-as then appears in the D8UserAuthzLegacyRBACv2CustomCapabilityFound alert and needs that label as well.
The frozen role does not gain permissions for resources added later, such as the resources of a new module. Add them to its rules field yourself, or replace the role with a role of the new scheme once the built-in capabilities carry the new labels. The explicit RoleBinding objects from the first step point at the d8:use:role:LEVEL roles, which the new scheme keeps only as deprecated aliases. Deckhouse does not install the release that follows the update and removes these aliases until the explicit RoleBinding objects are moved to d8:namespace:LEVEL. If the role is applied from a Git repository, make the same change in its manifest, otherwise the next apply restores aggregationRule.

After the update, the rbacv2-cluster-roles.deckhouse.io webhook refuses every update of a ClusterRole that carries the aggregation label of a built-in lineage, such as rbac.deckhouse.io/aggregate-to-all-as or rbac.deckhouse.io/aggregate-to-kubernetes-as, without the rbac.deckhouse.io/kind: custom-capability label, an apply of its manifest from a Git repository included. A frozen role with such a label cannot be changed until it is replaced with a role of the new scheme, and the D8UserAuthzForeignAggregationLabel alert names it. XXX

For the commands, refer to the corresponding section in the user-authz module FAQ.

List all such roles:
d8 k get clusterroles -o json | jq -r ' def lineages: [((.matchLabels // {}) | keys[]), (.matchExpressions[]? | select(.operator == "In" or .operator == "Exists") | .key)] | map(capture("^rbac\\.deckhouse\\.io/aggregate-to-(?lineage.+)-as$").lineage); ([.items[] | select(.metadata.labels.heritage != "deckhouse" and .aggregationRule == null) | (.metadata.labels // {}) | keys[] | capture("^rbac\\.deckhouse\\.io/aggregate-to-(?lineage.+)-as$").lineage]) as $carried | .items[] | select(.metadata.labels.heritage != "deckhouse" and .aggregationRule != null) | (.metadata.labels["rbac.deckhouse.io/subsystem"] // "") as $own | select( ( .metadata.labels["rbac.deckhouse.io/kind"] == "manage" and ((.metadata.labels["rbac.deckhouse.io/use-role"] // "") != "") ) or ( [ .aggregationRule.clusterRoleSelectors[]? | (.matchLabels["rbac.deckhouse.io/kind"] // empty), (.matchExpressions[]? | select(.key == "rbac.deckhouse.io/kind" and .operator == "In") | .values[]) ] | any(IN("manage", "use")) ) or ( [ .aggregationRule.clusterRoleSelectors[]? | lineages[] | select((IN("system", "namespace", "project", "iam", "security", "cluster", "delivery", "network", "storage", "observability", "managed-services") | not) and (IN("all", "deckhouse", "infrastructure", "kubernetes", "networking") or (. != $own and (IN($carried[]) | not)))) ] | length > 0 ) ) | .metadata.name'
D8UserAuthzLimitNamespacesEntryInvalid CE S6
A limitNamespaces entry of ClusterAuthorizationRule NAME is not a valid regular expression.
The ClusterAuthorizationRule NAME has the limitNamespaces entry XXX, which is not a valid regular expression in the RE2 syntax the way the authorization webhook anchors it, with ^ in front and $ at the end. For example, team-(a has an unclosed parenthesis, and ^(?!kube-).* uses a lookahead, which RE2 does not have.

The rule has no namespaceSelector, so the authorization webhook reads its limitNamespaces entries. It cannot compile this entry and leaves it out, and permission-browser does the same. The rule still limits its subjects to namespaces, so they have no access to the namespaces that only this entry was meant to cover. If no entry of the rule compiles, the rule adds no namespace through its entries, though its allowAccessToSystemNamespaces is still merged with the other rules of the same subjects. Other rules and their subjects are not affected, and changes of ClusterAuthorizationRules keep being applied.

The webhook logs the entry it leaves out:
d8 k -n d8-user-authz logs -l app=user-authz-webhook -c webhook --tail=-1 --prefix | grep 'the pattern is left out'
Fix the entry. Remove it only when the rule keeps other limitNamespaces entries. A rule with neither limitNamespaces entries nor a namespaceSelector covers every namespace except the system ones, so removing its only entry widens the access of its subjects.

Review the rule:
d8 k get clusterauthorizationrule NAME -o yaml
D8UserAuthzLimitNamespacesEntryNeedsRewrite CE S6
A limitNamespaces entry of ClusterAuthorizationRule NAME has to be rewritten.
The ClusterAuthorizationRule NAME has the limitNamespaces entry XXX (reason XXX).

The authorization webhook anchors an entry by adding ^ at the start and $ at the end, unless the entry already starts with ^ or ends with $. For an entry with the reason alternation, optional-anchor or anchored-invalid, the anchors do not apply to the whole pattern:
- alternation: The entry has a | outside parentheses and character classes, so the anchors apply to the first and the last alternative only. For example, team-a|team-b becomes ^team-a|team-b$ and covers every namespace whose name starts with team-a or ends with team-b, such as team-a-dev. An alternative in the middle matches any part of the name.
- optional-anchor: The entry starts with a ^ that a quantifier makes optional, such as ^*team, and the webhook adds no ^ of its own. Such an entry covers every namespace whose name ends with team.
- anchored-invalid: The entry is a valid regular expression only the way the webhook anchors it. *-dev starts with a quantifier, which the webhook reads as ^*-dev$, so it covers every namespace whose name ends with -dev. \Qteam has no \E, so the quote takes in the anchors. Anchored as a whole, such an entry is not a valid expression and covers no namespace. The D8UserAuthzWebhookRulesQuarantined alert appears with the update, and after the update the webhook reports the rule of such an entry in it. An entry such as \Qteam already covers no namespace, so for it only the alert is new.
Anchored as a whole, ^(?:team-a|team-b)$ or ^(?:^*team)$, the same entry covers only the names that match it completely: team-a and team-b, or team. When entries are anchored as a whole, the subjects of the rule lose access to the namespaces that only the partial match covers.

Rewrite the entry so that it covers the same namespaces under both anchorings:
- Put the alternatives in parentheses, (team-a|team-b), or into separate entries.
- Remove the quantifier after the leading ^.
- Write what a leading quantifier stands for, .*-dev instead of *-dev, and close a \Q quote with \E.
- If the subjects of the rule need the namespaces that only the partial match covers, write them into the entry explicitly, for example team-a.* or .*team-b.
With the reason invalid-unused, the entry is not a valid regular expression, for example team-(a with an unclosed parenthesis or team-a\ with a trailing backslash. The rule has a namespaceSelector, so the webhook does not use its limitNamespaces entries, and the entry has no effect now. After the update, a ClusterAuthorizationRule cannot be created or changed while one of its entries has an unbalanced parenthesis or bracket, a trailing backslash, or a construct RE2 does not have, such as lookaround, a backreference, a possessive quantifier or a repetition count over 1000, whatever its namespaceSelector. Remove the entry, which changes no access, or fix it.

Review the rule:
d8 k get clusterauthorizationrule NAME -o yaml
D8UserAuthzPermissionBrowserUnavailable CE S6
Permission Browser API server has unavailable replicas.
Deckhouse has detected unavailable replicas in the permission-browser-apiserver deployment in the d8-user-authz namespace. Permission Browser requests can fail while this alert is active.

Check deployment and pod states:
d8 k -n d8-user-authz get deploy permission-browser-apiserver d8 k -n d8-user-authz get pods -l app=permission-browser-apiserver
D8UserAuthzReleaseSlow CE S6
The user-authz module release repeatedly takes more than 5 minutes.
Applying the user-authz Helm release took more than 5 minutes on average over the last hour, and it was applied at least twice in that hour, so this is not the single slow pass of a cluster bootstrap or of the one-time migration to user-authz-controller. A release that does not finish within the module release timeout (20 minutes) fails, changes to ClusterAuthorizationRule and AuthorizationRule objects stop being applied and platform updates stall (the D8DeckhouseCouldNotRunModule alert fires in that case).

The release itself no longer grows with the number of rules: bindings are managed by user-authz-controller, and custom ClusterRoles are aggregated. A slow release therefore points outside the module: an API server that answers slowly, an object stuck with a finalizer, a Deckhouse Pod short of CPU, or the release engine itself. Check what the release is waiting on:
d8 s queue list d8 k -n d8-system logs deploy/deckhouse | grep '"module":"user-authz"'
If the cluster has just been updated to the release that moved the bindings to the controller, one slow pass is expected and does not raise this alert by itself; if it fires anyway, the pass is being retried. It is proportional to the number of bindings and can exceed the timeout on clusters with more than 5000 of them; the module FAQ describes what to do in that case.
D8UserAuthzRenderSlow CE S7
Rendering the user-authz module templates repeatedly takes more than 30 seconds.
Rendering the templates of the user-authz module took more than 30 seconds on average over the last 30 minutes, across at least three renders; normally one render takes about a second. Rendering is CPU work inside the Deckhouse Pod and does not depend on the number of ClusterAuthorizationRule or AuthorizationRule objects, so the cause is the node, not the module: most often the Deckhouse Pod is short of CPU (CPU steal on a virtual machine, a node running at its limits, a bootstrap that starts every module at once).

A single slow render during a cluster bootstrap does not raise this alert by itself. If it fires, compare the module with the others and look at the node:
topk(10, increase(deckhouse_helm_operation_seconds_sum{operation="template"}[30m]) / increase(deckhouse_helm_operation_seconds_count{operation="template"}[30m])) rate(node_cpu_seconds_total{mode="steal"}[5m])
d8 k -n d8-system logs deploy/deckhouse | grep '"module":"user-authz"'

Module vertical-pod-autoscaler

VerticalPodAutoscalerDeprecatedUpdateModeAuto CE S9
VerticalPodAutoscaler objects use the deprecated Auto update mode.
There are VALUE VerticalPodAutoscaler object(s) in the cluster configured with the deprecated update mode (spec.updatePolicy.updateMode: Auto).

The Auto update mode is deprecated. It currently behaves like Recreate (the VPA updater evicts pods to apply new resource requests) and its semantics are reserved to change in future versions, which makes cluster behavior ambiguous and potentially disruptive.

Set the affected VPA object(s) to one of the supported update modes (spec.updatePolicy.updateMode):
- InPlaceOrRecreate: Apply recommendations in place when possible, fall back to recreation (recommended).
- Recreate: Evict and recreate pods to apply recommendations.
- Initial: Apply only when pods are (re)created.
- Off: Recommendations only, no changes to running pods.
To list the affected objects, open the VPA Deprecated Update Mode Grafana dashboard, or run:
kube_verticalpodautoscaler_spec_updatepolicy_updatemode{update_mode="Auto"} == 1

Was the page useful?

Thanks for the review!

Your response is being processed, and changes can be made in 5 minutes.

An error has occurred

Please try again later

Tell us what you didn’t like.

Little useful There are errors It’s written incomprehensibly Outdated information Other

The list of alerts

Alert severity

Module admission-policy-engine

Module cert-manager

Module chrony

Module cloud-provider-dvp

Module cloud-provider-huaweicloud

Module cloud-provider-yandex

Module cni-cilium

Module cni-flannel

Module cni-simple-bridge

Module control-plane-manager

Module deckhouse

Module descheduler

Module documentation

Module istio

Module kube-dns

Module metallb

Module multitenancy-manager

Module node-local-dns

Module node-manager

Module okmeter

Module openvpn

Module secret-copier

Module terraform-manager

Module user-authn

Module user-authz

Module vertical-pod-autoscaler

An error has occurred

Tell us what you didn’t like.

Request trial access

Thank you

Error

Request callback

Thank you

Something went wrong

Book your sessions

Thank you

Error

Request demo

Thank you

Error

Get the PCI SSC compliance report

Thank you

Error

Request partner program details

Request received

Error