The module lifecycle stage: Generally available version

The module has requirements for installation

Experimental feature. The parameter alertDiagnostics.enabled that turns it on is experimental, and so is everything it brings: reports, built-in playbooks with their checks, findings and texts, the schemas of the report and playbook resources. All of it may change or be removed in future versions without a migration; do not build automation on it yet.

What it is

Alert diagnostics creates a diagnostic report for every firing alert. The controller alert-diagnostics-controller finds the Kubernetes object the alert is about (the object of the report), reads its state, runs checks and writes findings into the report: what is wrong, why, and what to do.

A report is a Kubernetes resource:

Playbooks describe what to check for an alert and which findings to make. The built-in playbooks for DKP alerts ship with the module; an administrator can add their own.

The diagnostics is deterministic: findings come from the rules of the playbooks over facts read from the cluster. No language model is used. The diagnostics changes nothing in the objects the alerts are about: the controller writes only reports and the status of playbooks.

Enabling and configuring

The diagnostics is off by default. To turn it on, set the parameter alertDiagnostics.enabled:

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: observability
spec:
  version: 2
  enabled: true
  settings:
    alertDiagnostics:
      enabled: true

When the parameter is turned off, the module removes the controller and all reports. Playbooks created by an administrator stay.

Commands in containers

Some checks run a fixed command of a playbook in a container, for example df in a container with a full volume. While the parameter alertDiagnostics.podExec.enabled is true (the default), the controller has the right pods/exec in every namespace of the cluster, including the namespaces of the platform. It runs only the commands written in the playbooks.

Where this right is not acceptable, set podExec.enabled: false: the controller has no right pods/exec, and such checks end with the outcome Skipped.

Node agent

The causes of node problems often lie on the host. The agent alert-diagnostics-node-agent reads them: a DaemonSet in the namespace d8-observability whose pod runs on every node. The agent reads only on request of the controller and changes nothing on the node:

  • the state and the journal of the systemd units kubelet, containerd, containerd-deckhouse, bashible, cloud-init;
  • the kernel log;
  • the usage of the file systems of the node and of the writable layers of containers;
  • the listening sockets of a pod;
  • the counters of the connection tracking table (conntrack);
  • the state of the clock;
  • on control-plane nodes, the audit log of kube-apiserver.

The agent mounts the root file system of the node read-only. The controller connects to the agent at the node address on port 4294/TCP with mutual TLS; the module issues and renews the certificates itself. If a firewall between the nodes closes this port, allow connections to 4294/TCP of every node from the nodes where the pod of the controller runs.

The agent is on while the diagnostics is on. To work without it, set alertDiagnostics.nodeAgent.enabled to false: the agent is not deployed, the checks that read a node end with the outcome Skipped, and the findings about a node name the commands to run on the node by hand.

Other parameters

Parameter Default Meaning
retention.resolvedTTL 15m How long the report of an alert that resolved is kept
retention.lostTTL 15m How long the report of an alert whose object disappeared without resolving is kept
retention.maxReports 300 How many reports of both kinds the cluster may have, 10 to 3000
rerun.minInterval 5m Minimum interval between the start of a run and the next manual run of the same alert
logs.maxBytesPerCheck 32768 How many bytes of one container log a report keeps, 1024 to 262144
linkedTargets.maxDepth 5 How many objects in a row a report follows from the object of the alert (the backend of an Ingress, the dependencies the containers connect to), 0 to 5; 0 turns it off

All the parameters are in the section alertDiagnostics. resolvedTTL and lostTTL take hours and minutes, at least one minute: 15m, 24h, 1h30m.

What is diagnosed

The table lists the alerts the module has a built-in playbook for. The columns:

  • “Object of the report” — the kind of the object the report is about;
  • “Class of the object” — the objects the playbook runs on: User — user objects, Platform — objects of Deckhouse (with the label heritage: deckhouse on the object or its namespace, in the namespaces d8-* and kube-system, nodes, cluster resources of Deckhouse), Any — any;
  • “Report” — the kind of the report: odr for an alert of a project, codr for a cluster alert.

An alert may have several playbooks, for example for a user object and for an object of the platform; a report runs those that fit the object.

Alert Object of the report Class of the object Report Playbook
CapsInstanceUnavailable Deployment Platform codr observability-caps
CertmanagerCertificateChallengePending Deployment Platform odr observability-certmanager-challenge-pending
CertmanagerCertificateChallengeStuck Deployment Platform odr observability-certmanager-challenge-stuck
CertmanagerCertificateExpired Certificate Any codr observability-certmanager-certificate-expired
CertmanagerCertificateExpiredSoon Certificate Any codr observability-certmanager-certificate-expired
CiliumAgentEndpointsNotReady Pod Platform codr observability-cilium-agent
CiliumAgentMapPressureCritical Pod Platform codr observability-cilium-agent
CiliumAgentMetricNotFound Pod Platform codr observability-cilium-agent
CiliumAgentPolicyImportErrors Pod Platform codr observability-cilium-agent
CiliumAgentUnreachableHealthEndpoints Pod Platform codr observability-cilium-agent
CronJobAuthenticationFailure CronJob User odr observability-image-availability-cronjob
CronJobAuthorizationFailure CronJob User odr observability-image-availability-cronjob
CronJobBadImageFormat CronJob User odr observability-image-availability-cronjob
CronJobFailed Job User odr observability-cronjob-failed
CronJobImageAbsent CronJob User odr observability-image-availability-cronjob
CronJobImageAvailabilityUnknownError CronJob User odr observability-image-availability-cronjob
CronJobRegistryUnavailable CronJob User odr observability-image-availability-cronjob
CronJobSchedulingError CronJob User odr observability-cronjob-scheduling-error
D8CNIEnabledMoreThanOne Deployment Platform codr observability-deckhouse
D8ControlPlaneManagerPodNotRunning Node Platform codr observability-node-control-plane-manager
D8DeckhouseConfigInvalid Deployment Platform codr observability-deckhouse
D8DeckhouseCouldNotDeleteModule Deployment Platform codr observability-deckhouse
D8DeckhouseCouldNotDiscoverModules Deployment Platform codr observability-deckhouse
D8DeckhouseCouldNotRunGlobalHook Deployment Platform codr observability-deckhouse
D8DeckhouseCouldNotRunModule Deployment Platform codr observability-deckhouse
D8DeckhouseCouldNotRunModuleHook Deployment Platform codr observability-deckhouse
D8DeckhouseGlobalHookFailsTooOften Deployment Platform codr observability-deckhouse
D8DeckhouseHasNoAccessToRegistry Deployment Platform codr observability-deckhouse
D8DeckhouseModuleHookFailsTooOften Deployment Platform codr observability-deckhouse
D8DeckhousePodIsNotReady Deployment Platform codr observability-deckhouse
D8DeckhousePodIsRestartingTooOften Deployment Platform codr observability-deckhouse
D8DeckhouseQueueIsHung Deployment Platform codr observability-deckhouse-queue
D8DeckhouseQueueIsHungCritical Deployment Platform codr observability-deckhouse-queue
D8DeckhouseQueueIsHungGlobal Deployment Platform codr observability-deckhouse-queue
D8DeckhouseSelfTargetDown Deployment Platform codr observability-deckhouse
D8DeckhouseWatchErrorOccurred Deployment Platform codr observability-deckhouse
D8EtcdDatabaseHighFragmentationRatio Pod Platform codr observability-etcd
D8EtcdExcessiveDatabaseGrowth Pod Platform codr observability-etcd
D8EtcdHighRequestsLatency Pod Platform codr observability-etcd
D8GrafanaDeploymentReplicasUnavailable Deployment Platform codr observability-grafana
D8GrafanaPodIsRestartingTooOften Deployment Platform codr observability-grafana
D8GrafanaTargetAbsent Deployment Platform codr observability-grafana
D8GrafanaTargetDown Deployment Platform codr observability-grafana
D8IPv4OptionsGuardAuditMode Deployment Platform codr observability-ipv4-options-guard-audit
D8IPv4OptionsGuardBackendMismatch Node Platform codr observability-ipv4-options-guard
D8IPv4OptionsGuardFilterBypassed Node Platform codr observability-ipv4-options-guard
D8IPv4OptionsGuardMatchedTraffic Node Platform codr observability-ipv4-options-guard
D8IPv4OptionsGuardMetricsAbsent Node Platform codr observability-ipv4-options-guard
D8IPv4OptionsGuardNodeUnprotected Node Platform codr observability-ipv4-options-guard
D8IPv4OptionsGuardReconcileFailed Node Platform codr observability-ipv4-options-guard
D8ImageAvailabilityExporterMalfunctioning Deployment Platform codr observability-image-availability-exporter
D8ImageAvailabilityExporterPodIsNotReady Deployment Platform codr observability-image-availability-exporter
D8ImageAvailabilityExporterPodIsNotRunning Deployment Platform codr observability-image-availability-exporter
D8ImageAvailabilityExporterTargetAbsent Deployment Platform codr observability-image-availability-exporter
D8ImageAvailabilityExporterTargetDown Deployment Platform codr observability-image-availability-exporter
D8KubeEtcdDatabaseSizeCloseToTheLimit Pod Platform codr observability-etcd
D8KubernetesStaleTokensDetected DaemonSet Platform codr observability-control-plane-configuration
D8KubernetesVersionIsDeprecated DaemonSet Platform codr observability-control-plane-configuration
D8LogShipperAgentNotScheduledInCluster DaemonSet Platform odr, codr observability-log-shipper-not-scheduled
D8LogShipperDestinationErrors Node Platform codr observability-node-log-shipper
D8LogShipperSinkStuck Node Platform codr observability-node-log-shipper
D8MetallbNotSupportedServiceAnnotationsDetected Service Any codr observability-metallb-service-annotations
D8ModuleOutdatedByMajorVersion ModuleRelease Platform codr observability-module-release
D8MonitoringApplicationsLegacyBearerTokenEnabled ModuleConfig Platform codr observability-monitoring-legacy-bearer-token
D8MonitoringCustomLegacyBearerTokenEnabled ModuleConfig Platform codr observability-monitoring-legacy-bearer-token
D8NeedDecreaseEtcdQuotaBackendBytes Pod Platform codr observability-etcd
D8NodeIsNotUpdating Node Platform codr observability-node-update
D8NodeLocalDNSCacheDenialEvictionsHigh Node Platform codr observability-node-local-dns
D8NodeLocalDNSCacheSuccessEvictionsHigh Node Platform codr observability-node-local-dns
D8NodeLocalDNSForwardMaxConcurrentRejects Node Platform codr observability-node-local-dns
D8NodeLocalDNSKubeforwardRequestLatencyP95High Node Platform codr observability-node-local-dns
D8NodeLocalDNSUpstreamHealthcheckFailed Node Platform codr observability-node-local-dns
D8NodeUpdateStuckWaitingForDisruptionApproval Node Platform codr observability-node-update
D8ObsoletePublishAPIinUserAuthn ModuleConfig Platform codr observability-user-authn-publish-api
D8ProblematicFeatureGateInUse DaemonSet Platform codr observability-control-plane-configuration
D8ProblematicNodeGroupConfiguration Node Platform codr observability-node-update
D8PrometheusConfigReloadFailed Pod Platform odr, codr observability-prometheus-pod
D8PrometheusLongtermFederationTargetDown StatefulSet Platform codr observability-prometheus-longterm
D8PrometheusLongtermTargetAbsent StatefulSet Platform codr observability-prometheus-longterm
D8PrometheusOperatorPodIsNotReady Deployment Platform codr observability-prometheus-operator
D8PrometheusOperatorPodIsNotRunning Deployment Platform codr observability-prometheus-operator
D8PrometheusOperatorTargetAbsent Deployment Platform codr observability-prometheus-operator
D8PrometheusOperatorTargetDown Deployment Platform codr observability-prometheus-operator
D8SignatureErrorsDetected DaemonSet Platform codr observability-control-plane-configuration
D8SmokeMiniNotBoundPersistentVolumeClaims PersistentVolumeClaim Platform codr observability-smoke-mini-pvc
D8SnapshotControllerPodIsNotReady Deployment Platform codr observability-snapshot-controller
D8SnapshotControllerPodIsNotRunning Deployment Platform codr observability-snapshot-controller
D8SnapshotControllerTargetAbsent Deployment Platform codr observability-snapshot-controller
D8SnapshotControllerTargetDown Deployment Platform codr observability-snapshot-controller
D8TricksterTargetAbsent Deployment Platform codr observability-trickster
D8UpmeterAgentOutOfOrder DaemonSet Platform codr observability-upmeter-agent
D8UpmeterAgentPodIsNotReady DaemonSet Platform codr observability-upmeter-agent
D8UpmeterAgentReplicasUnavailable DaemonSet Platform codr observability-upmeter-agent
D8UpmeterLegacyMigrationFailed StatefulSet Platform codr observability-upmeter-server
D8UpmeterMaterializationStalled StatefulSet Platform codr observability-upmeter-server
D8UpmeterProbeGarbagePodsFromDeployments Pod Platform codr observability-upmeter-controller-manager-probe
D8UpmeterPromppHeadNotPersisted StatefulSet Platform codr observability-upmeter-server
D8UpmeterRemoteWriteLagging StatefulSet Platform codr observability-upmeter-server
D8UpmeterServerPodIsNotReady StatefulSet Platform codr observability-upmeter-server
D8UpmeterServerPodIsRestartingTooOften StatefulSet Platform codr observability-upmeter-server
D8UpmeterServerReplicasUnavailable StatefulSet Platform codr observability-upmeter-server
D8UsingFeatureGateWillBeDeprecated DaemonSet Platform codr observability-control-plane-configuration
DaemonSetAuthenticationFailure DaemonSet User odr observability-image-availability-daemonset
DaemonSetAuthorizationFailure DaemonSet User odr observability-image-availability-daemonset
DaemonSetBadImageFormat DaemonSet User odr observability-image-availability-daemonset
DaemonSetImageAbsent DaemonSet User odr observability-image-availability-daemonset
DaemonSetImageAvailabilityUnknownError DaemonSet User odr observability-image-availability-daemonset
DaemonSetRegistryUnavailable DaemonSet User odr observability-image-availability-daemonset
DeckhouseModuleUpdatingFailedModuleIsNotValid Deployment Platform codr observability-deckhouse-module-not-valid
DeckhouseReleaseDisruptionApprovalRequired DeckhouseRelease Platform codr observability-deckhouse-release
DeckhouseReleaseIsBlocked DeckhouseRelease Platform codr observability-deckhouse-release
DeckhouseReleaseIsWaitingManualApproval DeckhouseRelease Platform codr observability-deckhouse-release
DeckhouseReleaseNotificationNotSent DeckhouseRelease Platform codr observability-deckhouse-release
DeploymentAuthenticationFailure Deployment User odr observability-image-availability-deployment
DeploymentAuthorizationFailure Deployment User odr observability-image-availability-deployment
DeploymentBadImageFormat Deployment User odr observability-image-availability-deployment
DeploymentImageAbsent Deployment User odr observability-image-availability-deployment
DeploymentImageAvailabilityUnknownError Deployment User odr observability-image-availability-deployment
DeploymentRegistryUnavailable Deployment User odr observability-image-availability-deployment
ExtendedMonitoringTargetDown Deployment Platform codr observability-extended-monitoring-exporter
GrafanaDashboardUIDDuplicatedCluster Deployment Platform codr observability-grafana-dashboard-uid
IngressResponses5xx Ingress User odr observability-ingress-responses-5xx
K8SApiserverDown Pod Platform codr observability-control-plane-apiserver
K8SControllerManagerTargetDown Pod Platform codr observability-control-plane-controller-manager
K8SKubeletDown DaemonSet Platform codr observability-node-exporter-daemonset
K8SManyNodesNotReady DaemonSet Platform codr observability-node-exporter-daemonset
K8SNodeNotReady Node Platform codr observability-node-not-ready
K8SSchedulerTargetDown Pod Platform codr observability-control-plane-scheduler
K8sCertificateExpiration DaemonSet Platform codr observability-control-plane-client-certificates
KubeEtcdHighFsyncDurations Pod Platform codr observability-etcd
KubeEtcdHighNumberOfLeaderChanges Pod Platform codr observability-etcd
KubeEtcdInsufficientMembers Pod Platform codr observability-etcd
KubeEtcdNoLeader Pod Platform codr observability-etcd
KubeEtcdTargetAbsent Pod Platform codr observability-etcd
KubeEtcdTargetDown Pod Platform codr observability-etcd
KubeletImageFSBytesUsage Node Platform codr observability-node-disk
KubeletImageFSInodesUsage Node Platform codr observability-node-disk
KubeletNodeFSBytesUsage Node Platform codr observability-node-disk
KubeletNodeFSInodesUsage Node Platform codr observability-node-disk
KubernetesDaemonSetNotUpToDate DaemonSet User odr observability-daemonset-not-up-to-date
KubernetesDaemonSetReplicasUnavailable DaemonSet User odr observability-daemonset-replicas-unavailable
KubernetesDaemonSetReplicasUnavailable DaemonSet Platform odr observability-platform-daemonset-replicas
KubernetesDeploymentReplicasUnavailable Deployment User odr observability-deployment-replicas-unavailable
KubernetesDeploymentReplicasUnavailable Deployment Platform odr observability-platform-deployment-replicas
KubernetesServiceExternalIPsDeprecated DaemonSet Platform codr observability-control-plane-configuration
KubernetesStatefulSetReplicasUnavailable StatefulSet Platform odr observability-platform-statefulset-replicas
KubernetesStatefulSetReplicasUnavailable StatefulSet User odr observability-statefulset-replicas-unavailable
LoadAverageHigh Node Platform codr observability-node-load
LokiDiscardedSamplesWarning StatefulSet Platform codr observability-loki-module
LokiInsufficientDiskForRetention StatefulSet Platform codr observability-loki-module
ModuleConfigObsoleteVersion ModuleConfig Platform codr observability-module-config-obsolete
ModuleReleaseIsOutdated ModuleRelease Platform codr observability-module-release
ModuleReleaseIsWaitingManualApproval ModuleRelease Platform codr observability-module-release
NginxIngressDaemonSetNotUpToDate Pod Platform codr observability-ingress-nginx-controller
NginxIngressDaemonSetReplicasUnavailable Pod Platform codr observability-ingress-nginx-controller
NginxIngressSslExpired Ingress User odr observability-ingress-ssl-expiry
NginxIngressSslExpired Ingress Platform odr observability-platform-ingress-ssl-expiry
NginxIngressSslWillExpire Ingress User odr observability-ingress-ssl-expiry
NginxIngressSslWillExpire Ingress Platform odr observability-platform-ingress-ssl-expiry
NodeDiskBytesUsage Node Platform codr observability-node-disk
NodeDiskInodesUsage Node Platform codr observability-node-disk
NodeExporterDown DaemonSet Platform codr observability-node-exporter-daemonset
NodePingPacketLoss Node Platform codr observability-node-ping-packet-loss
NodeRequiresDisruptionApprovalForUpdate Node Platform codr observability-node-update
NodeStuckInDraining Node Platform codr observability-node-update
NodeStuckInDrainingForDisruptionDuringUpdate Node Platform codr observability-node-update
NodeUnschedulable Node Platform codr observability-node-unschedulable
PodOOMKilled Pod Platform odr observability-platform-pod-oomkilled
PodOOMKilled Pod Platform codr observability-platform-pod-oomkilled-cluster
PodOOMKilled Pod User odr observability-pod-oomkilled
PodStatusIsIncorrect Pod Any odr observability-pod-status-is-incorrect
PrometheusConfigReloadFailing Pod Platform odr, codr observability-prometheus-pod
PrometheusDiskUsage Pod Platform codr observability-prometheus-disk-usage
PrometheusNotConnectedToAlertmanager Pod Platform odr, codr observability-prometheus-pod
PrometheusNotificationQueueRunningFull Pod Platform odr, codr observability-prometheus-pod
PrometheusNotificationsDropping Pod Platform odr, codr observability-prometheus-pod
PrometheusRemoteWriteEndpointUnavailable StatefulSet Platform codr observability-prometheus-remote-write
PrometheusRuleEvaluationFailing Pod Platform odr, codr observability-prometheus-pod
PrometheusTSDBCompactionsFailing Pod Platform odr, codr observability-prometheus-pod
PrometheusTSDBReloadsFailing Pod Platform odr, codr observability-prometheus-pod
PrometheusTargetDown Pod Platform odr, codr observability-platform-prometheus-target-down
PrometheusTargetDown Pod User odr, codr observability-prometheus-target-down
PrometheusUnavailable StatefulSet Platform codr observability-prometheus-unavailable
StatefulSetAuthenticationFailure StatefulSet User odr observability-image-availability-statefulset
StatefulSetAuthorizationFailure StatefulSet User odr observability-image-availability-statefulset
StatefulSetBadImageFormat StatefulSet User odr observability-image-availability-statefulset
StatefulSetImageAbsent StatefulSet User odr observability-image-availability-statefulset
StatefulSetImageAvailabilityUnknownError StatefulSet User odr observability-image-availability-statefulset
StatefulSetRegistryUnavailable StatefulSet User odr observability-image-availability-statefulset
TargetDown Pod User odr observability-target-down-pod
TargetSampleLimitExceeded Pod User odr observability-target-sample-limit-exceeded

An alert without a built-in playbook gets a report too. The controller finds the object by the standard labels of DKP alerts (pod, job_name, deployment, statefulset, daemonset, cronjob, ingress, persistentvolumeclaim, and node for a cluster alert) and reads its state. For a Deployment, StatefulSet, DaemonSet and Pod, a fitting built-in playbook runs on it; its findings describe problems of the object (the role ObjectProblem) without claiming they are the cause of the alert. If the labels name no object or several unrelated objects, the report is created without diagnostics: the phase Failed, the reason NoDiagnostics, without checks and findings.

Reading a report

A report is named after the alert object: in lower case, with characters outside a-z0-9.- replaced by -. The labels observability.deckhouse.io/alertname and observability.deckhouse.io/alert-fingerprint help find the report of an alert.

# Reports of a namespace.
kubectl -n <namespace> get odr
# Reports of one alert.
kubectl -n <namespace> get odr -l observability.deckhouse.io/alertname=KubernetesDeploymentReplicasUnavailable
# Cluster reports.
kubectl get codr
# A whole report.
kubectl -n <namespace> get odr <report> -o yaml

kubectl get shows the alert, the state of the report, the state of the alert, the headline and the number of findings.

Summary

Field Meaning
status.summary.headline One line: the title of the most severe finding that names a cause, else of the most severe finding
status.summary.causeEstablished true when the headline comes from a finding that names a cause
status.summary.causeGap With causeEstablished: false, one sentence on what the report could not read or follow; the headline then starts with Cause not established:
status.summary.severity The highest severity of the findings: Info, Warning or Critical
status.summary.findingsCount The number of findings

The texts of the findings and of the summary are in English.

Findings

Every finding in status.findings has:

  • title — the title, explanation — the explanation, actions — what to do; in a command of a text, a value that cannot be written into a command as it is shows as a placeholder (<value>, <name>, <image>, <registry>), a command that cannot be shown whole shows as [command withheld] or [command cut], and a value the text sets in code formatting without a command around it shows in plain quotes ('reboot'); a value of a fact keeps its commands by the same rules;
  • severity — Info, Warning or Critical;
  • role — what the finding tells of the cause: Cause — a cause, ConfigurationCause — a cause in a setting, Symptom — a symptom that does not establish the cause by itself, Fact — a fact, ObjectProblem — a problem of the object of an alert without a built-in playbook;
  • causedBy — the findings this one follows from;
  • facts — the values that prove the finding;
  • evidenceRefs — the checks the finding is based on.

The checks are in status.checks: each has an outcome (Ok, ProblemFound, NotFound, InsufficientInput, Forbidden, Timeout, Error, Skipped) and the collected data in evidence.

States

Field Values
status.summary.state Collecting — collection goes on, Ready — the report is final, Failed — the facts could not be collected
status.collection.phase Pending, Running, Completed, Partial (some checks could not finish, status.collection.message names them) or Failed (the reason is in status.collection.reason)
status.alertState.state Firing, Resolved or Lost — the alert object has been missing for 3 minutes without resolving

In the Deckhouse web UI, the report of an alert is shown on the “Diagnostics” tab of the alert.

Manual runs and reruns

A manual run requires the rights to create reports (see “Access”).

Creating a report

A report is created automatically for every firing alert. If an alert has no report (for example, it was deleted), create one with a reference to the alert object; the name of the report may be any:

apiVersion: observability.deckhouse.io/v1alpha1
kind: ObservabilityDiagnosticReport
metadata:
  name: my-check
  namespace: <namespace>
spec:
  alert:
    kind: ObservabilityAlert
    name: <name of the alert object>
    namespace: <namespace>

kubectl -n <namespace> get observabilityalerts shows the name of the alert object. For a cluster alert, create a ClusterObservabilityDiagnosticReport without metadata.namespace, and set kind: ClusterObservabilityAlert and name (the name from kubectl get clusterobservabilityalerts) in spec.alert.

Set only kind, name and namespace in spec.alert: the controller fills the rest. The report ends with the phase Failed without collecting facts when:

  • other fields of spec.alert are set, or the namespace of the alert is not the namespace of the report (InvalidSpec);
  • the alert object does not exist (AlertNotFound), the alert is not firing (AlertNotFiring) or is silenced (AlertSilenced);
  • the alert already has a report (DuplicateReport).

Rerunning

To run the diagnostics again, set a new value of the annotation observability.deckhouse.io/rerun:

kubectl -n <namespace> annotate odr <report> observability.deckhouse.io/rerun="$(date +%s)" --overwrite
kubectl annotate codr <report> observability.deckhouse.io/rerun="$(date +%s)" --overwrite

The previous run moves to status.history. The request is refused when the alert is not firing (AlertNotFiring), is silenced (AlertSilenced) or the previous run started less than rerun.minInterval ago (TooFrequent); the reason is written to the condition Rerun of the report. A request that comes during a run is served when the run ends.

Retention and limits

The report of a firing alert is not deleted. The report of an alert that resolved is deleted after resolvedTTL, the report of an alert that disappeared after lostTTL.

When the number of reports reaches maxReports, creating a new report first deletes a report of a finished alert: failed reports without facts first, then reports of alerts that disappeared, then reports of resolved alerts, the oldest first. When every report belongs to a firing alert, the automatic creation waits for room, and a report created by a user is accepted while there are at most maxReports plus 50 reports. Reports without diagnostics (NoDiagnostics) are not counted and are not deleted to make room.

To keep a report a small object, it has limits:

What Limit
Checks of one run 250; a run that dropped checks over the limit gets the phase Partial
Data of the checks of one run (evidence) 512 KiB for all checks together; the data of later checks is cut
Container log The last 100 lines (a check of a playbook may ask for up to 200) and at most logs.maxBytesPerCheck bytes
History of runs (status.history) 5 past runs, each with the headline and up to 10 short findings

A report holds no data of Secret and ConfigMap objects: the controller has no rights to read them. Tokens, passwords and keys found in logs, messages and commands are replaced by [REDACTED], inside a command by REDACTED.

Access

The roles of the module give the rights on reports (see Permissions):

Role Rights
d8:namespace-capability:observability:view Reading the reports of a namespace (odr)
d8:namespace-capability:observability:edit In addition, creating, changing (including the rerun annotation) and deleting the reports of a namespace
d8:system-capability:observability:view Reading cluster reports (codr) and playbooks
d8:system-capability:observability:edit In addition, creating, changing and deleting cluster reports

In DKP below 1.78 these roles are named d8:use:capability:module:observability:view|edit and d8:manage:permission:module:observability:view|edit respectively.

The access levels of the module user-authz give the same: User reads the reports of a namespace, Editor also creates, changes and deletes them, ClusterEditor reads, creates, changes and deletes cluster reports and reads playbooks.

The rights on reports exist only while the diagnostics is on. No role of the module gives the right to change the status of a report or to create and change playbooks: playbooks are created by an administrator with full rights in the cluster.

Custom playbooks

The built-in playbooks ship in the image of the controller and are not objects of the cluster: they cannot be changed or turned off. The resource ClusterObservabilityDiagnosticPlaybook lets an administrator add a playbook, for example for an alert of their own rule. The schema of the resource is experimental. A playbook of the cluster runs next to the built-in ones; the names of the built-in playbooks and names that start with observability- are reserved.

Example: the project alert ShopApiErrorRateHigh names a Deployment in the label deployment. The playbook reads the last log lines of the pods of the Deployment that contain connection refused and makes a finding when there are such lines:

apiVersion: observability.deckhouse.io/v1alpha1
kind: ClusterObservabilityDiagnosticPlaybook
metadata:
  name: shop-api-database-refused
spec:
  match:
    alertNames: [ShopApiErrorRateHigh]
    scope: Namespaced
    requiredLabels: [deployment]
  target: {apiVersion: apps/v1, kind: Deployment, nameFromLabel: deployment}
  checks:
  - id: refused
    type: PodLogs
    title: Refused connections
    podLogs: {subject: Target, tailLines: 200, grep: "connection refused"}
  findings:
  - id: connection-refused
    title: "Pods of {{ .labels.deployment }} cannot connect to a dependency"
    severity: Critical
    when: 'checks.refused.evidence.?logs.orValue([]).exists(l, size(l.?lines.orValue([])) > 0)'
    explanation: "The logs of the pods have `connection refused` errors."
    actions:
    - "Check that the service the pods connect to is running and has ready endpoints."

Check that the controller accepted the playbook: the condition Valid must have the status True, and on an error its message names the problem.

kubectl get clusterobservabilitydiagnosticplaybook shop-api-database-refused -o jsonpath='{.status.conditions}'

The report of the next firing of the alert lists the playbook in status.run.playbooks with source: Cluster, and its findings with the IDs shop-api-database-refused/<id>. The full schema is on the page ClusterObservabilityDiagnosticPlaybook.

Limitations

  • The feature is experimental: the fields of a report, the built-in playbooks and their texts may change.
  • Silenced alerts are not diagnosed: no report is created for them, and manual runs and reruns are refused.
  • Only the objects a playbook or the standard labels of an alert name are diagnosed; an alert whose labels name no object gets a report without diagnostics.
  • A report of a project reads only the objects of its namespace. The one exception is the log of image-availability-exporter: the report reads it only for an image its workload runs and names the answer of the registry by its class (for example, UNAUTHORIZED), not by its text.
  • Without the node agent and without commands in containers, the checks that depend on them are skipped, and no findings are made from them.
  • A report shows the state at the time of the run; to see the current state, run the diagnostics again.