The module lifecycle stage: Generally available version
The module has requirements for installation
The observability module provides access control mechanisms for cluster-level and project-level metrics,
as well as tools for interactive metric analysis.
Depending on the scope, two types of metrics are distinguished:
| Metric type | Scope | Description |
|---|---|---|
| ClusterMetrics | Cluster level | Metrics related to cluster components and system namespaces whose names match the d8-* and kube-* patterns. |
| Metrics | Project level | Metrics related to resources of a specific project based on the value of the namespace label. Access to these metrics is limited to the corresponding project. |
The module allows running arbitrary PromQL queries against cluster or project metrics directly from the Deckhouse web UI.
Interactive metric analysis capabilities:
- Running arbitrary PromQL queries against metrics.
- Selecting the data source (main Prometheus, Longterm, and custom sources).
- Visualizing results as tables and graphs.
- Metric and label autocomplete while typing queries.
Available data exploration modes:
-
System data exploration: Available to DKP administrators in the Deckhouse web UI under “System” → “System management” → “Monitoring” → “Data exploration”. This section provides access to cluster metrics.
-
Project data exploration: Available to DKP project users in the Deckhouse web UI in the corresponding project under “Monitoring” → “Data exploration”. In this section, queries are limited to metrics related to the current project (within its namespace).
External access to metrics
The module allows configuring external access to cluster metrics for integration with external monitoring systems (for example, external Grafana) or for writing metrics from external sources.
To enable external access, use the externalMetricsAccess parameter.
Request authorization is performed using a service account with read or write permissions for metrics.
The following scenarios are supported:
- Reading cluster metrics: Access to system metrics across the entire cluster.
- Reading project metrics: Access to metrics of a specific namespace through a Prometheus-compatible API.
- Writing metrics: Sending metrics from external sources using the Prometheus Remote Write protocol.
- Writing cluster metrics: Sending metrics that belong to the whole cluster rather than to a namespace.
External access for reading cluster metrics
To provide external access to cluster metrics, follow these steps:
-
Enable external access to metrics. To do this, enable the
externalMetricsAccessparameter. -
Create a ServiceAccount for request authorization:
apiVersion: v1 kind: ServiceAccount metadata: name: cluster-metrics-access --- apiVersion: v1 kind: Secret metadata: name: cluster-metrics-access annotations: kubernetes.io/service-account.name: cluster-metrics-access type: kubernetes.io/service-account-token -
Grant read permissions for metrics to the created ServiceAccount using ClusterRole and ClusterRoleBinding:
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: observability-cluster-metrics-viewer rules: - apiGroups: ["observability.deckhouse.io"] resources: ["clustermetrics"] verbs: ["get", "list", "watch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: bind-observability-cluster-metrics-viewer subjects: - kind: ServiceAccount name: cluster-metrics-access namespace: default roleRef: kind: ClusterRole name: observability-cluster-metrics-viewer apiGroup: rbac.authorization.k8s.io -
Retrieve the authorization token for metric access.
When the ServiceAccount was created, a secret containing the token was also created. The token is stored as a Base64-encoded string. To extract and decode it, run the following command:
d8 k -n my-namespace get secret cluster-metrics-access -ojsonpath='{ .data.token }' | base64 -dThis token will be required in the next step when configuring a Grafana data source.
-
Configure metric access in external Grafana.
Add a Prometheus data source with the following parameters:
Name: Any data source name.URL: External endpoint address for metric access in the formathttps://observability.%publicDomainTemplate%/<prefix>, where:%publicDomainTemplate%: DNS name template of your cluster configured in DKP global settings.<prefix>: One of the Prometheus access prefixes:/metrics/main: For main Prometheus metrics./metrics/longterm: For Prometheus Longterm metrics.
HTTP Headers: Additional HTTP headers for authorization:Header:Authorization.Value:Bearer <TOKEN_VALUE>, where<TOKEN_VALUE>is the token obtained from the secret in the previous step.
External access for reading project metrics
To provide external access to project metrics, follow these steps:
-
Enable external access to metrics. To do this, enable the
externalMetricsAccessmodule parameter. -
Create a ServiceAccount for request authorization:
apiVersion: v1 kind: ServiceAccount metadata: name: metrics-access namespace: my-namespace --- apiVersion: v1 kind: Secret metadata: name: metrics-access annotations: kubernetes.io/service-account.name: metrics-access type: kubernetes.io/service-account-token -
Grant read permissions for metrics to the created ServiceAccount using Role and RoleBinding:
apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: my-namespace name: metrics-access rules: - apiGroups: ["observability.deckhouse.io"] resources: ["metrics"] verbs: ["get", "watch", "list"] - apiGroups: [""] resources: ["namespaces"] verbs: ["get", "watch", "list"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: metrics-access namespace: my-namespace subjects: - kind: ServiceAccount name: metrics-access namespace: my-namespace roleRef: kind: Role name: metrics-access apiGroup: rbac.authorization.k8s.io -
Retrieve the authorization token for metric access.
When the ServiceAccount was created, a secret containing the token was also created. The token is stored as a Base64-encoded string. To extract and decode it, run the following command:
d8 k -n my-namespace get secret metrics-access -ojsonpath='{ .data.token }' | base64 -dThis token will be required in the next step when configuring a Grafana data source.
-
Configure metric access in external Grafana.
Add a Prometheus data source with the following parameters:
Name: Any data source name.URL: External endpoint address for metric access in the formathttps://observability.%publicDomainTemplate%/<prefix>, where:%publicDomainTemplate%: DNS name template of your cluster configured in DKP global settings.<prefix>: One of the Prometheus access prefixes:/metrics/main: For main Prometheus metrics./metrics/longterm: For Prometheus Longterm metrics.
HTTP Headers: Additional HTTP headers for authorization:Header:Authorization.Value:Bearer <TOKEN_VALUE>, where<TOKEN_VALUE>is the token obtained from the secret in the previous step.
External access for writing metrics
To provide external access for writing metrics, follow these steps:
-
Enable external access to metrics. To do this, enable the
externalMetricsAccessmodule parameter. -
Create a ServiceAccount for request authorization:
apiVersion: v1 kind: ServiceAccount metadata: name: metrics-access namespace: my-namespace --- apiVersion: v1 kind: Secret metadata: name: metrics-access annotations: kubernetes.io/service-account.name: metrics-access type: kubernetes.io/service-account-token -
Grant write permissions for metrics to the created ServiceAccount using Role and RoleBinding:
apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: my-namespace name: metrics-access rules: - apiGroups: ["observability.deckhouse.io"] resources: ["metrics"] verbs: ["create"] - apiGroups: [""] resources: ["namespaces"] verbs: ["get", "watch", "list"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: metrics-access namespace: my-namespace subjects: - kind: ServiceAccount name: metrics-access namespace: my-namespace roleRef: kind: Role name: metrics-access apiGroup: rbac.authorization.k8s.io -
Retrieve the authorization token for writing metrics.
When the ServiceAccount was created, a secret containing the token was also created. The token is stored as a Base64-encoded string. To extract and decode it, run the following command:
d8 k -n my-namespace get secret metrics-access -ojsonpath='{ .data.token }' | base64 -d -
To write metrics, send requests using Prometheus Remote Write V1 or V2 with the following parameters:
URL: External endpoint address for writing metrics in the formathttps://observability.%publicDomainTemplate%/metrics/main/api/v1/write, where:%publicDomainTemplate%: DNS name template of your cluster configured in DKP global settings.
HTTP Headers: Additional HTTP headers for authorization:Header:Authorization.Value:Bearer <TOKEN_VALUE>, where<TOKEN_VALUE>is the token obtained from the secret in the previous step.
Every series must carry the
k8s_namespacelabel with the name of a namespace the ServiceAccount may write into. The module sets thenamespacelabel to the same value, so the metrics become visible to the users of that project. Series without the label, or with a namespace the ServiceAccount has no write access to, are dropped.
External access for writing cluster metrics
Cluster metrics are metrics that describe the cluster as a whole rather than a project (for example, node metrics).
They are stored without the namespace label and are visible only to users with access to system metrics.
To provide external access for writing cluster metrics, follow these steps:
-
Enable external access to metrics. To do this, enable the
externalMetricsAccessmodule parameter. -
Create a ServiceAccount for request authorization:
apiVersion: v1 kind: ServiceAccount metadata: name: cluster-metrics-writer --- apiVersion: v1 kind: Secret metadata: name: cluster-metrics-writer annotations: kubernetes.io/service-account.name: cluster-metrics-writer type: kubernetes.io/service-account-token -
Grant write permissions for cluster metrics to the created ServiceAccount using ClusterRole and ClusterRoleBinding:
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: observability-cluster-metrics-writer rules: - apiGroups: ["observability.deckhouse.io"] resources: ["clustermetrics"] verbs: ["create"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: bind-observability-cluster-metrics-writer subjects: - kind: ServiceAccount name: cluster-metrics-writer namespace: default roleRef: kind: ClusterRole name: observability-cluster-metrics-writer apiGroup: rbac.authorization.k8s.io -
Retrieve the authorization token for writing metrics:
d8 k -n default get secret cluster-metrics-writer -ojsonpath='{ .data.token }' | base64 -d -
Send metrics using Prometheus Remote Write with the same
URLandHTTP Headersas in the previous section.Series without the
k8s_namespacelabel are written as cluster metrics: the module does not add thenamespacelabel to them, so they are visible only to cluster metrics readers. Anamespacelabel sent by the client is renamed toexported_namespace, the same way Prometheus handles a conflicting scraped label, so it cannot make the series look like a project metric. Series that do carryk8s_namespaceare handled as project metrics: they are written only if the ServiceAccount also has thecreatepermission for themetricsresource in that namespace (see the previous section), otherwise they are dropped.