The module lifecycle stage: Generally available version
The module has requirements for installation
Experimental feature. The parameter alertDiagnostics.enabled that turns it on is experimental, and so is everything it brings: reports, built-in playbooks with their checks, findings and texts, the schemas of the report and playbook resources. All of it may change or be removed in future versions without a migration; do not build automation on it yet.
What it is
Alert diagnostics creates a diagnostic report for every firing alert. The controller alert-diagnostics-controller finds the Kubernetes object the alert is about (the object of the report), reads its state, runs checks and writes findings into the report: what is wrong, why, and what to do.
A report is a Kubernetes resource:
- ObservabilityDiagnosticReport (
odr) — for an alert of a project, in the namespace of the alert; - ClusterObservabilityDiagnosticReport (
codr) — for a cluster alert.
Playbooks describe what to check for an alert and which findings to make. The built-in playbooks for DKP alerts ship with the module; an administrator can add their own.
The diagnostics is deterministic: findings come from the rules of the playbooks over facts read from the cluster. No language model is used. The diagnostics changes nothing in the objects the alerts are about: the controller writes only reports and the status of playbooks.
Enabling and configuring
The diagnostics is off by default. To turn it on, set the parameter alertDiagnostics.enabled:
apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
name: observability
spec:
version: 2
enabled: true
settings:
alertDiagnostics:
enabled: trueWhen the parameter is turned off, the module removes the controller and all reports. Playbooks created by an administrator stay.
Commands in containers
Some checks run a fixed command of a playbook in a container, for example df in a container with a full volume. While the parameter alertDiagnostics.podExec.enabled is true (the default), the controller has the right pods/exec in every namespace of the cluster, including the namespaces of the platform. It runs only the commands written in the playbooks.
Where this right is not acceptable, set podExec.enabled: false: the controller has no right pods/exec, and such checks end with the outcome Skipped.
Node agent
The causes of node problems often lie on the host. The agent alert-diagnostics-node-agent reads them: a DaemonSet in the namespace d8-observability whose pod runs on every node. The agent reads only on request of the controller and changes nothing on the node:
- the state and the journal of the systemd units
kubelet,containerd,containerd-deckhouse,bashible,cloud-init; - the kernel log;
- the usage of the file systems of the node and of the writable layers of containers;
- the listening sockets of a pod;
- the counters of the connection tracking table (conntrack);
- the state of the clock;
- on control-plane nodes, the audit log of kube-apiserver.
The agent mounts the root file system of the node read-only. The controller connects to the agent at the node address on port 4294/TCP with mutual TLS; the module issues and renews the certificates itself. If a firewall between the nodes closes this port, allow connections to 4294/TCP of every node from the nodes where the pod of the controller runs.
The agent is on while the diagnostics is on. To work without it, set alertDiagnostics.nodeAgent.enabled to false: the agent is not deployed, the checks that read a node end with the outcome Skipped, and the findings about a node name the commands to run on the node by hand.
Other parameters
| Parameter | Default | Meaning |
|---|---|---|
retention.resolvedTTL |
15m |
How long the report of an alert that resolved is kept |
retention.lostTTL |
15m |
How long the report of an alert whose object disappeared without resolving is kept |
retention.maxReports |
300 |
How many reports of both kinds the cluster may have, 10 to 3000 |
rerun.minInterval |
5m |
Minimum interval between the start of a run and the next manual run of the same alert |
logs.maxBytesPerCheck |
32768 |
How many bytes of one container log a report keeps, 1024 to 262144 |
linkedTargets.maxDepth |
5 |
How many objects in a row a report follows from the object of the alert (the backend of an Ingress, the dependencies the containers connect to), 0 to 5; 0 turns it off |
All the parameters are in the section alertDiagnostics. resolvedTTL and lostTTL take hours and minutes, at least one minute: 15m, 24h, 1h30m.
What is diagnosed
The table lists the alerts the module has a built-in playbook for. The columns:
- “Object of the report” — the kind of the object the report is about;
- “Class of the object” — the objects the playbook runs on:
User— user objects,Platform— objects of Deckhouse (with the labelheritage: deckhouseon the object or its namespace, in the namespacesd8-*andkube-system, nodes, cluster resources of Deckhouse),Any— any; - “Report” — the kind of the report:
odrfor an alert of a project,codrfor a cluster alert.
An alert may have several playbooks, for example for a user object and for an object of the platform; a report runs those that fit the object.
| Alert | Object of the report | Class of the object | Report | Playbook |
|---|---|---|---|---|
CapsInstanceUnavailable |
Deployment | Platform |
codr |
observability-caps |
CertmanagerCertificateChallengePending |
Deployment | Platform |
odr |
observability-certmanager-challenge-pending |
CertmanagerCertificateChallengeStuck |
Deployment | Platform |
odr |
observability-certmanager-challenge-stuck |
CertmanagerCertificateExpired |
Certificate | Any |
codr |
observability-certmanager-certificate-expired |
CertmanagerCertificateExpiredSoon |
Certificate | Any |
codr |
observability-certmanager-certificate-expired |
CiliumAgentEndpointsNotReady |
Pod | Platform |
codr |
observability-cilium-agent |
CiliumAgentMapPressureCritical |
Pod | Platform |
codr |
observability-cilium-agent |
CiliumAgentMetricNotFound |
Pod | Platform |
codr |
observability-cilium-agent |
CiliumAgentPolicyImportErrors |
Pod | Platform |
codr |
observability-cilium-agent |
CiliumAgentUnreachableHealthEndpoints |
Pod | Platform |
codr |
observability-cilium-agent |
CronJobAuthenticationFailure |
CronJob | User |
odr |
observability-image-availability-cronjob |
CronJobAuthorizationFailure |
CronJob | User |
odr |
observability-image-availability-cronjob |
CronJobBadImageFormat |
CronJob | User |
odr |
observability-image-availability-cronjob |
CronJobFailed |
Job | User |
odr |
observability-cronjob-failed |
CronJobImageAbsent |
CronJob | User |
odr |
observability-image-availability-cronjob |
CronJobImageAvailabilityUnknownError |
CronJob | User |
odr |
observability-image-availability-cronjob |
CronJobRegistryUnavailable |
CronJob | User |
odr |
observability-image-availability-cronjob |
CronJobSchedulingError |
CronJob | User |
odr |
observability-cronjob-scheduling-error |
D8CNIEnabledMoreThanOne |
Deployment | Platform |
codr |
observability-deckhouse |
D8ControlPlaneManagerPodNotRunning |
Node | Platform |
codr |
observability-node-control-plane-manager |
D8DeckhouseConfigInvalid |
Deployment | Platform |
codr |
observability-deckhouse |
D8DeckhouseCouldNotDeleteModule |
Deployment | Platform |
codr |
observability-deckhouse |
D8DeckhouseCouldNotDiscoverModules |
Deployment | Platform |
codr |
observability-deckhouse |
D8DeckhouseCouldNotRunGlobalHook |
Deployment | Platform |
codr |
observability-deckhouse |
D8DeckhouseCouldNotRunModule |
Deployment | Platform |
codr |
observability-deckhouse |
D8DeckhouseCouldNotRunModuleHook |
Deployment | Platform |
codr |
observability-deckhouse |
D8DeckhouseGlobalHookFailsTooOften |
Deployment | Platform |
codr |
observability-deckhouse |
D8DeckhouseHasNoAccessToRegistry |
Deployment | Platform |
codr |
observability-deckhouse |
D8DeckhouseModuleHookFailsTooOften |
Deployment | Platform |
codr |
observability-deckhouse |
D8DeckhousePodIsNotReady |
Deployment | Platform |
codr |
observability-deckhouse |
D8DeckhousePodIsRestartingTooOften |
Deployment | Platform |
codr |
observability-deckhouse |
D8DeckhouseQueueIsHung |
Deployment | Platform |
codr |
observability-deckhouse-queue |
D8DeckhouseQueueIsHungCritical |
Deployment | Platform |
codr |
observability-deckhouse-queue |
D8DeckhouseQueueIsHungGlobal |
Deployment | Platform |
codr |
observability-deckhouse-queue |
D8DeckhouseSelfTargetDown |
Deployment | Platform |
codr |
observability-deckhouse |
D8DeckhouseWatchErrorOccurred |
Deployment | Platform |
codr |
observability-deckhouse |
D8EtcdDatabaseHighFragmentationRatio |
Pod | Platform |
codr |
observability-etcd |
D8EtcdExcessiveDatabaseGrowth |
Pod | Platform |
codr |
observability-etcd |
D8EtcdHighRequestsLatency |
Pod | Platform |
codr |
observability-etcd |
D8GrafanaDeploymentReplicasUnavailable |
Deployment | Platform |
codr |
observability-grafana |
D8GrafanaPodIsRestartingTooOften |
Deployment | Platform |
codr |
observability-grafana |
D8GrafanaTargetAbsent |
Deployment | Platform |
codr |
observability-grafana |
D8GrafanaTargetDown |
Deployment | Platform |
codr |
observability-grafana |
D8IPv4OptionsGuardAuditMode |
Deployment | Platform |
codr |
observability-ipv4-options-guard-audit |
D8IPv4OptionsGuardBackendMismatch |
Node | Platform |
codr |
observability-ipv4-options-guard |
D8IPv4OptionsGuardFilterBypassed |
Node | Platform |
codr |
observability-ipv4-options-guard |
D8IPv4OptionsGuardMatchedTraffic |
Node | Platform |
codr |
observability-ipv4-options-guard |
D8IPv4OptionsGuardMetricsAbsent |
Node | Platform |
codr |
observability-ipv4-options-guard |
D8IPv4OptionsGuardNodeUnprotected |
Node | Platform |
codr |
observability-ipv4-options-guard |
D8IPv4OptionsGuardReconcileFailed |
Node | Platform |
codr |
observability-ipv4-options-guard |
D8ImageAvailabilityExporterMalfunctioning |
Deployment | Platform |
codr |
observability-image-availability-exporter |
D8ImageAvailabilityExporterPodIsNotReady |
Deployment | Platform |
codr |
observability-image-availability-exporter |
D8ImageAvailabilityExporterPodIsNotRunning |
Deployment | Platform |
codr |
observability-image-availability-exporter |
D8ImageAvailabilityExporterTargetAbsent |
Deployment | Platform |
codr |
observability-image-availability-exporter |
D8ImageAvailabilityExporterTargetDown |
Deployment | Platform |
codr |
observability-image-availability-exporter |
D8KubeEtcdDatabaseSizeCloseToTheLimit |
Pod | Platform |
codr |
observability-etcd |
D8KubernetesStaleTokensDetected |
DaemonSet | Platform |
codr |
observability-control-plane-configuration |
D8KubernetesVersionIsDeprecated |
DaemonSet | Platform |
codr |
observability-control-plane-configuration |
D8LogShipperAgentNotScheduledInCluster |
DaemonSet | Platform |
odr, codr |
observability-log-shipper-not-scheduled |
D8LogShipperDestinationErrors |
Node | Platform |
codr |
observability-node-log-shipper |
D8LogShipperSinkStuck |
Node | Platform |
codr |
observability-node-log-shipper |
D8MetallbNotSupportedServiceAnnotationsDetected |
Service | Any |
codr |
observability-metallb-service-annotations |
D8ModuleOutdatedByMajorVersion |
ModuleRelease | Platform |
codr |
observability-module-release |
D8MonitoringApplicationsLegacyBearerTokenEnabled |
ModuleConfig | Platform |
codr |
observability-monitoring-legacy-bearer-token |
D8MonitoringCustomLegacyBearerTokenEnabled |
ModuleConfig | Platform |
codr |
observability-monitoring-legacy-bearer-token |
D8NeedDecreaseEtcdQuotaBackendBytes |
Pod | Platform |
codr |
observability-etcd |
D8NodeIsNotUpdating |
Node | Platform |
codr |
observability-node-update |
D8NodeLocalDNSCacheDenialEvictionsHigh |
Node | Platform |
codr |
observability-node-local-dns |
D8NodeLocalDNSCacheSuccessEvictionsHigh |
Node | Platform |
codr |
observability-node-local-dns |
D8NodeLocalDNSForwardMaxConcurrentRejects |
Node | Platform |
codr |
observability-node-local-dns |
D8NodeLocalDNSKubeforwardRequestLatencyP95High |
Node | Platform |
codr |
observability-node-local-dns |
D8NodeLocalDNSUpstreamHealthcheckFailed |
Node | Platform |
codr |
observability-node-local-dns |
D8NodeUpdateStuckWaitingForDisruptionApproval |
Node | Platform |
codr |
observability-node-update |
D8ObsoletePublishAPIinUserAuthn |
ModuleConfig | Platform |
codr |
observability-user-authn-publish-api |
D8ProblematicFeatureGateInUse |
DaemonSet | Platform |
codr |
observability-control-plane-configuration |
D8ProblematicNodeGroupConfiguration |
Node | Platform |
codr |
observability-node-update |
D8PrometheusConfigReloadFailed |
Pod | Platform |
odr, codr |
observability-prometheus-pod |
D8PrometheusLongtermFederationTargetDown |
StatefulSet | Platform |
codr |
observability-prometheus-longterm |
D8PrometheusLongtermTargetAbsent |
StatefulSet | Platform |
codr |
observability-prometheus-longterm |
D8PrometheusOperatorPodIsNotReady |
Deployment | Platform |
codr |
observability-prometheus-operator |
D8PrometheusOperatorPodIsNotRunning |
Deployment | Platform |
codr |
observability-prometheus-operator |
D8PrometheusOperatorTargetAbsent |
Deployment | Platform |
codr |
observability-prometheus-operator |
D8PrometheusOperatorTargetDown |
Deployment | Platform |
codr |
observability-prometheus-operator |
D8SignatureErrorsDetected |
DaemonSet | Platform |
codr |
observability-control-plane-configuration |
D8SmokeMiniNotBoundPersistentVolumeClaims |
PersistentVolumeClaim | Platform |
codr |
observability-smoke-mini-pvc |
D8SnapshotControllerPodIsNotReady |
Deployment | Platform |
codr |
observability-snapshot-controller |
D8SnapshotControllerPodIsNotRunning |
Deployment | Platform |
codr |
observability-snapshot-controller |
D8SnapshotControllerTargetAbsent |
Deployment | Platform |
codr |
observability-snapshot-controller |
D8SnapshotControllerTargetDown |
Deployment | Platform |
codr |
observability-snapshot-controller |
D8TricksterTargetAbsent |
Deployment | Platform |
codr |
observability-trickster |
D8UpmeterAgentOutOfOrder |
DaemonSet | Platform |
codr |
observability-upmeter-agent |
D8UpmeterAgentPodIsNotReady |
DaemonSet | Platform |
codr |
observability-upmeter-agent |
D8UpmeterAgentReplicasUnavailable |
DaemonSet | Platform |
codr |
observability-upmeter-agent |
D8UpmeterLegacyMigrationFailed |
StatefulSet | Platform |
codr |
observability-upmeter-server |
D8UpmeterMaterializationStalled |
StatefulSet | Platform |
codr |
observability-upmeter-server |
D8UpmeterProbeGarbagePodsFromDeployments |
Pod | Platform |
codr |
observability-upmeter-controller-manager-probe |
D8UpmeterPromppHeadNotPersisted |
StatefulSet | Platform |
codr |
observability-upmeter-server |
D8UpmeterRemoteWriteLagging |
StatefulSet | Platform |
codr |
observability-upmeter-server |
D8UpmeterServerPodIsNotReady |
StatefulSet | Platform |
codr |
observability-upmeter-server |
D8UpmeterServerPodIsRestartingTooOften |
StatefulSet | Platform |
codr |
observability-upmeter-server |
D8UpmeterServerReplicasUnavailable |
StatefulSet | Platform |
codr |
observability-upmeter-server |
D8UsingFeatureGateWillBeDeprecated |
DaemonSet | Platform |
codr |
observability-control-plane-configuration |
DaemonSetAuthenticationFailure |
DaemonSet | User |
odr |
observability-image-availability-daemonset |
DaemonSetAuthorizationFailure |
DaemonSet | User |
odr |
observability-image-availability-daemonset |
DaemonSetBadImageFormat |
DaemonSet | User |
odr |
observability-image-availability-daemonset |
DaemonSetImageAbsent |
DaemonSet | User |
odr |
observability-image-availability-daemonset |
DaemonSetImageAvailabilityUnknownError |
DaemonSet | User |
odr |
observability-image-availability-daemonset |
DaemonSetRegistryUnavailable |
DaemonSet | User |
odr |
observability-image-availability-daemonset |
DeckhouseModuleUpdatingFailedModuleIsNotValid |
Deployment | Platform |
codr |
observability-deckhouse-module-not-valid |
DeckhouseReleaseDisruptionApprovalRequired |
DeckhouseRelease | Platform |
codr |
observability-deckhouse-release |
DeckhouseReleaseIsBlocked |
DeckhouseRelease | Platform |
codr |
observability-deckhouse-release |
DeckhouseReleaseIsWaitingManualApproval |
DeckhouseRelease | Platform |
codr |
observability-deckhouse-release |
DeckhouseReleaseNotificationNotSent |
DeckhouseRelease | Platform |
codr |
observability-deckhouse-release |
DeploymentAuthenticationFailure |
Deployment | User |
odr |
observability-image-availability-deployment |
DeploymentAuthorizationFailure |
Deployment | User |
odr |
observability-image-availability-deployment |
DeploymentBadImageFormat |
Deployment | User |
odr |
observability-image-availability-deployment |
DeploymentImageAbsent |
Deployment | User |
odr |
observability-image-availability-deployment |
DeploymentImageAvailabilityUnknownError |
Deployment | User |
odr |
observability-image-availability-deployment |
DeploymentRegistryUnavailable |
Deployment | User |
odr |
observability-image-availability-deployment |
ExtendedMonitoringTargetDown |
Deployment | Platform |
codr |
observability-extended-monitoring-exporter |
GrafanaDashboardUIDDuplicatedCluster |
Deployment | Platform |
codr |
observability-grafana-dashboard-uid |
IngressResponses5xx |
Ingress | User |
odr |
observability-ingress-responses-5xx |
K8SApiserverDown |
Pod | Platform |
codr |
observability-control-plane-apiserver |
K8SControllerManagerTargetDown |
Pod | Platform |
codr |
observability-control-plane-controller-manager |
K8SKubeletDown |
DaemonSet | Platform |
codr |
observability-node-exporter-daemonset |
K8SManyNodesNotReady |
DaemonSet | Platform |
codr |
observability-node-exporter-daemonset |
K8SNodeNotReady |
Node | Platform |
codr |
observability-node-not-ready |
K8SSchedulerTargetDown |
Pod | Platform |
codr |
observability-control-plane-scheduler |
K8sCertificateExpiration |
DaemonSet | Platform |
codr |
observability-control-plane-client-certificates |
KubeEtcdHighFsyncDurations |
Pod | Platform |
codr |
observability-etcd |
KubeEtcdHighNumberOfLeaderChanges |
Pod | Platform |
codr |
observability-etcd |
KubeEtcdInsufficientMembers |
Pod | Platform |
codr |
observability-etcd |
KubeEtcdNoLeader |
Pod | Platform |
codr |
observability-etcd |
KubeEtcdTargetAbsent |
Pod | Platform |
codr |
observability-etcd |
KubeEtcdTargetDown |
Pod | Platform |
codr |
observability-etcd |
KubeletImageFSBytesUsage |
Node | Platform |
codr |
observability-node-disk |
KubeletImageFSInodesUsage |
Node | Platform |
codr |
observability-node-disk |
KubeletNodeFSBytesUsage |
Node | Platform |
codr |
observability-node-disk |
KubeletNodeFSInodesUsage |
Node | Platform |
codr |
observability-node-disk |
KubernetesDaemonSetNotUpToDate |
DaemonSet | User |
odr |
observability-daemonset-not-up-to-date |
KubernetesDaemonSetReplicasUnavailable |
DaemonSet | User |
odr |
observability-daemonset-replicas-unavailable |
KubernetesDaemonSetReplicasUnavailable |
DaemonSet | Platform |
odr |
observability-platform-daemonset-replicas |
KubernetesDeploymentReplicasUnavailable |
Deployment | User |
odr |
observability-deployment-replicas-unavailable |
KubernetesDeploymentReplicasUnavailable |
Deployment | Platform |
odr |
observability-platform-deployment-replicas |
KubernetesServiceExternalIPsDeprecated |
DaemonSet | Platform |
codr |
observability-control-plane-configuration |
KubernetesStatefulSetReplicasUnavailable |
StatefulSet | Platform |
odr |
observability-platform-statefulset-replicas |
KubernetesStatefulSetReplicasUnavailable |
StatefulSet | User |
odr |
observability-statefulset-replicas-unavailable |
LoadAverageHigh |
Node | Platform |
codr |
observability-node-load |
LokiDiscardedSamplesWarning |
StatefulSet | Platform |
codr |
observability-loki-module |
LokiInsufficientDiskForRetention |
StatefulSet | Platform |
codr |
observability-loki-module |
ModuleConfigObsoleteVersion |
ModuleConfig | Platform |
codr |
observability-module-config-obsolete |
ModuleReleaseIsOutdated |
ModuleRelease | Platform |
codr |
observability-module-release |
ModuleReleaseIsWaitingManualApproval |
ModuleRelease | Platform |
codr |
observability-module-release |
NginxIngressDaemonSetNotUpToDate |
Pod | Platform |
codr |
observability-ingress-nginx-controller |
NginxIngressDaemonSetReplicasUnavailable |
Pod | Platform |
codr |
observability-ingress-nginx-controller |
NginxIngressSslExpired |
Ingress | User |
odr |
observability-ingress-ssl-expiry |
NginxIngressSslExpired |
Ingress | Platform |
odr |
observability-platform-ingress-ssl-expiry |
NginxIngressSslWillExpire |
Ingress | User |
odr |
observability-ingress-ssl-expiry |
NginxIngressSslWillExpire |
Ingress | Platform |
odr |
observability-platform-ingress-ssl-expiry |
NodeDiskBytesUsage |
Node | Platform |
codr |
observability-node-disk |
NodeDiskInodesUsage |
Node | Platform |
codr |
observability-node-disk |
NodeExporterDown |
DaemonSet | Platform |
codr |
observability-node-exporter-daemonset |
NodePingPacketLoss |
Node | Platform |
codr |
observability-node-ping-packet-loss |
NodeRequiresDisruptionApprovalForUpdate |
Node | Platform |
codr |
observability-node-update |
NodeStuckInDraining |
Node | Platform |
codr |
observability-node-update |
NodeStuckInDrainingForDisruptionDuringUpdate |
Node | Platform |
codr |
observability-node-update |
NodeUnschedulable |
Node | Platform |
codr |
observability-node-unschedulable |
PodOOMKilled |
Pod | Platform |
odr |
observability-platform-pod-oomkilled |
PodOOMKilled |
Pod | Platform |
codr |
observability-platform-pod-oomkilled-cluster |
PodOOMKilled |
Pod | User |
odr |
observability-pod-oomkilled |
PodStatusIsIncorrect |
Pod | Any |
odr |
observability-pod-status-is-incorrect |
PrometheusConfigReloadFailing |
Pod | Platform |
odr, codr |
observability-prometheus-pod |
PrometheusDiskUsage |
Pod | Platform |
codr |
observability-prometheus-disk-usage |
PrometheusNotConnectedToAlertmanager |
Pod | Platform |
odr, codr |
observability-prometheus-pod |
PrometheusNotificationQueueRunningFull |
Pod | Platform |
odr, codr |
observability-prometheus-pod |
PrometheusNotificationsDropping |
Pod | Platform |
odr, codr |
observability-prometheus-pod |
PrometheusRemoteWriteEndpointUnavailable |
StatefulSet | Platform |
codr |
observability-prometheus-remote-write |
PrometheusRuleEvaluationFailing |
Pod | Platform |
odr, codr |
observability-prometheus-pod |
PrometheusTSDBCompactionsFailing |
Pod | Platform |
odr, codr |
observability-prometheus-pod |
PrometheusTSDBReloadsFailing |
Pod | Platform |
odr, codr |
observability-prometheus-pod |
PrometheusTargetDown |
Pod | Platform |
odr, codr |
observability-platform-prometheus-target-down |
PrometheusTargetDown |
Pod | User |
odr, codr |
observability-prometheus-target-down |
PrometheusUnavailable |
StatefulSet | Platform |
codr |
observability-prometheus-unavailable |
StatefulSetAuthenticationFailure |
StatefulSet | User |
odr |
observability-image-availability-statefulset |
StatefulSetAuthorizationFailure |
StatefulSet | User |
odr |
observability-image-availability-statefulset |
StatefulSetBadImageFormat |
StatefulSet | User |
odr |
observability-image-availability-statefulset |
StatefulSetImageAbsent |
StatefulSet | User |
odr |
observability-image-availability-statefulset |
StatefulSetImageAvailabilityUnknownError |
StatefulSet | User |
odr |
observability-image-availability-statefulset |
StatefulSetRegistryUnavailable |
StatefulSet | User |
odr |
observability-image-availability-statefulset |
TargetDown |
Pod | User |
odr |
observability-target-down-pod |
TargetSampleLimitExceeded |
Pod | User |
odr |
observability-target-sample-limit-exceeded |
An alert without a built-in playbook gets a report too. The controller finds the object by the standard labels of DKP alerts (pod, job_name, deployment, statefulset, daemonset, cronjob, ingress, persistentvolumeclaim, and node for a cluster alert) and reads its state. For a Deployment, StatefulSet, DaemonSet and Pod, a fitting built-in playbook runs on it; its findings describe problems of the object (the role ObjectProblem) without claiming they are the cause of the alert. If the labels name no object or several unrelated objects, the report is created without diagnostics: the phase Failed, the reason NoDiagnostics, without checks and findings.
Reading a report
A report is named after the alert object: in lower case, with characters outside a-z0-9.- replaced by -. The labels observability.deckhouse.io/alertname and observability.deckhouse.io/alert-fingerprint help find the report of an alert.
# Reports of a namespace.
kubectl -n <namespace> get odr
# Reports of one alert.
kubectl -n <namespace> get odr -l observability.deckhouse.io/alertname=KubernetesDeploymentReplicasUnavailable
# Cluster reports.
kubectl get codr
# A whole report.
kubectl -n <namespace> get odr <report> -o yamlkubectl get shows the alert, the state of the report, the state of the alert, the headline and the number of findings.
Summary
| Field | Meaning |
|---|---|
status.summary.headline |
One line: the title of the most severe finding that names a cause, else of the most severe finding |
status.summary.causeEstablished |
true when the headline comes from a finding that names a cause |
status.summary.causeGap |
With causeEstablished: false, one sentence on what the report could not read or follow; the headline then starts with Cause not established: |
status.summary.severity |
The highest severity of the findings: Info, Warning or Critical |
status.summary.findingsCount |
The number of findings |
The texts of the findings and of the summary are in English.
Findings
Every finding in status.findings has:
title— the title,explanation— the explanation,actions— what to do; in a command of a text, a value that cannot be written into a command as it is shows as a placeholder (<value>,<name>,<image>,<registry>), a command that cannot be shown whole shows as[command withheld]or[command cut], and a value the text sets in code formatting without a command around it shows in plain quotes ('reboot'); a value of a fact keeps its commands by the same rules;severity—Info,WarningorCritical;role— what the finding tells of the cause:Cause— a cause,ConfigurationCause— a cause in a setting,Symptom— a symptom that does not establish the cause by itself,Fact— a fact,ObjectProblem— a problem of the object of an alert without a built-in playbook;causedBy— the findings this one follows from;facts— the values that prove the finding;evidenceRefs— the checks the finding is based on.
The checks are in status.checks: each has an outcome (Ok, ProblemFound, NotFound, InsufficientInput, Forbidden, Timeout, Error, Skipped) and the collected data in evidence.
States
| Field | Values |
|---|---|
status.summary.state |
Collecting — collection goes on, Ready — the report is final, Failed — the facts could not be collected |
status.collection.phase |
Pending, Running, Completed, Partial (some checks could not finish, status.collection.message names them) or Failed (the reason is in status.collection.reason) |
status.alertState.state |
Firing, Resolved or Lost — the alert object has been missing for 3 minutes without resolving |
In the Deckhouse web UI, the report of an alert is shown on the “Diagnostics” tab of the alert.
Manual runs and reruns
A manual run requires the rights to create reports (see “Access”).
Creating a report
A report is created automatically for every firing alert. If an alert has no report (for example, it was deleted), create one with a reference to the alert object; the name of the report may be any:
apiVersion: observability.deckhouse.io/v1alpha1
kind: ObservabilityDiagnosticReport
metadata:
name: my-check
namespace: <namespace>
spec:
alert:
kind: ObservabilityAlert
name: <name of the alert object>
namespace: <namespace>kubectl -n <namespace> get observabilityalerts shows the name of the alert object. For a cluster alert, create a ClusterObservabilityDiagnosticReport without metadata.namespace, and set kind: ClusterObservabilityAlert and name (the name from kubectl get clusterobservabilityalerts) in spec.alert.
Set only kind, name and namespace in spec.alert: the controller fills the rest. The report ends with the phase Failed without collecting facts when:
- other fields of
spec.alertare set, or thenamespaceof the alert is not the namespace of the report (InvalidSpec); - the alert object does not exist (
AlertNotFound), the alert is not firing (AlertNotFiring) or is silenced (AlertSilenced); - the alert already has a report (
DuplicateReport).
Rerunning
To run the diagnostics again, set a new value of the annotation observability.deckhouse.io/rerun:
kubectl -n <namespace> annotate odr <report> observability.deckhouse.io/rerun="$(date +%s)" --overwrite
kubectl annotate codr <report> observability.deckhouse.io/rerun="$(date +%s)" --overwriteThe previous run moves to status.history. The request is refused when the alert is not firing (AlertNotFiring), is silenced (AlertSilenced) or the previous run started less than rerun.minInterval ago (TooFrequent); the reason is written to the condition Rerun of the report. A request that comes during a run is served when the run ends.
Retention and limits
The report of a firing alert is not deleted. The report of an alert that resolved is deleted after resolvedTTL, the report of an alert that disappeared after lostTTL.
When the number of reports reaches maxReports, creating a new report first deletes a report of a finished alert: failed reports without facts first, then reports of alerts that disappeared, then reports of resolved alerts, the oldest first. When every report belongs to a firing alert, the automatic creation waits for room, and a report created by a user is accepted while there are at most maxReports plus 50 reports. Reports without diagnostics (NoDiagnostics) are not counted and are not deleted to make room.
To keep a report a small object, it has limits:
| What | Limit |
|---|---|
| Checks of one run | 250; a run that dropped checks over the limit gets the phase Partial |
Data of the checks of one run (evidence) |
512 KiB for all checks together; the data of later checks is cut |
| Container log | The last 100 lines (a check of a playbook may ask for up to 200) and at most logs.maxBytesPerCheck bytes |
History of runs (status.history) |
5 past runs, each with the headline and up to 10 short findings |
A report holds no data of Secret and ConfigMap objects: the controller has no rights to read them. Tokens, passwords and keys found in logs, messages and commands are replaced by [REDACTED], inside a command by REDACTED.
Access
The roles of the module give the rights on reports (see Permissions):
| Role | Rights |
|---|---|
d8:namespace-capability:observability:view |
Reading the reports of a namespace (odr) |
d8:namespace-capability:observability:edit |
In addition, creating, changing (including the rerun annotation) and deleting the reports of a namespace |
d8:system-capability:observability:view |
Reading cluster reports (codr) and playbooks |
d8:system-capability:observability:edit |
In addition, creating, changing and deleting cluster reports |
In DKP below 1.78 these roles are named d8:use:capability:module:observability:view|edit and d8:manage:permission:module:observability:view|edit respectively.
The access levels of the module user-authz give the same: User reads the reports of a namespace, Editor also creates, changes and deletes them, ClusterEditor reads, creates, changes and deletes cluster reports and reads playbooks.
The rights on reports exist only while the diagnostics is on. No role of the module gives the right to change the status of a report or to create and change playbooks: playbooks are created by an administrator with full rights in the cluster.
Custom playbooks
The built-in playbooks ship in the image of the controller and are not objects of the cluster: they cannot be changed or turned off. The resource ClusterObservabilityDiagnosticPlaybook lets an administrator add a playbook, for example for an alert of their own rule. The schema of the resource is experimental. A playbook of the cluster runs next to the built-in ones; the names of the built-in playbooks and names that start with observability- are reserved.
Example: the project alert ShopApiErrorRateHigh names a Deployment in the label deployment. The playbook reads the last log lines of the pods of the Deployment that contain connection refused and makes a finding when there are such lines:
apiVersion: observability.deckhouse.io/v1alpha1
kind: ClusterObservabilityDiagnosticPlaybook
metadata:
name: shop-api-database-refused
spec:
match:
alertNames: [ShopApiErrorRateHigh]
scope: Namespaced
requiredLabels: [deployment]
target: {apiVersion: apps/v1, kind: Deployment, nameFromLabel: deployment}
checks:
- id: refused
type: PodLogs
title: Refused connections
podLogs: {subject: Target, tailLines: 200, grep: "connection refused"}
findings:
- id: connection-refused
title: "Pods of {{ .labels.deployment }} cannot connect to a dependency"
severity: Critical
when: 'checks.refused.evidence.?logs.orValue([]).exists(l, size(l.?lines.orValue([])) > 0)'
explanation: "The logs of the pods have `connection refused` errors."
actions:
- "Check that the service the pods connect to is running and has ready endpoints."Check that the controller accepted the playbook: the condition Valid must have the status True, and on an error its message names the problem.
kubectl get clusterobservabilitydiagnosticplaybook shop-api-database-refused -o jsonpath='{.status.conditions}'The report of the next firing of the alert lists the playbook in status.run.playbooks with source: Cluster, and its findings with the IDs shop-api-database-refused/<id>. The full schema is on the page ClusterObservabilityDiagnosticPlaybook.
Limitations
- The feature is experimental: the fields of a report, the built-in playbooks and their texts may change.
- Silenced alerts are not diagnosed: no report is created for them, and manual runs and reruns are refused.
- Only the objects a playbook or the standard labels of an alert name are diagnosed; an alert whose labels name no object gets a report without diagnostics.
- A report of a project reads only the objects of its namespace. The one exception is the log of
image-availability-exporter: the report reads it only for an image its workload runs and names the answer of the registry by its class (for example,UNAUTHORIZED), not by its text. - Without the node agent and without commands in containers, the checks that depend on them are skipped, and no findings are made from them.
- A report shows the state at the time of the run; to see the current state, run the diagnostics again.