Monitoring

Stronghold monitoring relies on three data sources:

• telemetry metrics: exposed at the /v1/sys/metrics endpoint and, if configured, pushed to external systems (StatsD, DogStatsD, and so on);
• the /v1/sys/health health check endpoint;
• server logs and audit logs.

The full list of metrics with their types and labels is in the Metrics catalog.

Connection variants

Stronghold has a single metrics source (the telemetry subsystem) and several ways to deliver metrics to a monitoring system. The in-memory sink that backs the JSON response of sys/metrics is always on. The Prometheus sink is on by default (prometheus_retention_time = 24h), and the push sinks are enabled by their parameters in the telemetry block. All enabled sinks work at the same time.

Notes:

• All the listed sinks are available in all Stronghold editions.
• Push sinks can be combined with Prometheus and with each other. When more than one sink is enabled, set disable_hostname = true: otherwise Stronghold prints a warning at startup because hostname-prefixed metric names are inconvenient for Prometheus.
• Push sinks are configured only in the configuration file, so they are available for standalone installations only. In DP, the module generates the telemetry block and does not expose its parameters, see Monitoring in DP.

Enabling telemetry

Telemetry settings are defined in the telemetry block of the server configuration file. Only one telemetry block is allowed. The full list of parameters is in Configuration parameters.

Prometheus

The Prometheus sink is on by default: the retention time defaults to 24h. Set it explicitly and disable the hostname prefix so that metric names match across all nodes:

telemetry {
  prometheus_retention_time = "24h"
  disable_hostname          = true
}

• prometheus_retention_time — how long metrics are retained for Prometheus output. The value 0 disables the Prometheus format.
• disable_hostname — do not prefix metrics with the node hostname.
• metrics_prefix — metric name prefix. By default, metrics have the stronghold prefix.

Then configure access to the endpoint, see Accessing sys/metrics.

Push sinks

Push sinks send metrics to an external collector. Each sink is enabled by the presence of its address (or credentials) in the telemetry block.

StatsD (UDP):

telemetry {
  statsd_address   = "127.0.0.1:8125"
  disable_hostname = true
}

Statsite (TCP):

telemetry {
  statsite_address = "127.0.0.1:8125"
  disable_hostname = true
}

DogStatsD:

telemetry {
  dogstatsd_addr   = "127.0.0.1:8125"
  dogstatsd_tags   = ["env:prod", "service:stronghold"]
  disable_hostname = true
}

Circonus. The sink is enabled when circonus_api_token or circonus_submission_url is set:

telemetry {
  circonus_api_token           = "<API_TOKEN>"
  circonus_api_app             = "stronghold"
  circonus_submission_interval = "10s"
  circonus_check_tags          = "env:prod"
  circonus_broker_select_tag   = "dc:dc1"
  disable_hostname             = true
}

In this example, <API_TOKEN> is the Circonus API token.

Google Stackdriver (Cloud Monitoring). The sink is enabled when stackdriver_project_id is set; credentials are taken from the process environment (Application Default Credentials, for example the GOOGLE_APPLICATION_CREDENTIALS variable):

telemetry {
  stackdriver_project_id = "my-project"
  stackdriver_location   = "europe-west1"
  stackdriver_namespace  = "stronghold-prod"
  disable_hostname       = true
}

If you only need push delivery, you can disable the Prometheus sink with prometheus_retention_time = 0.

Restart the service after changing the configuration (standalone):

systemctl restart stronghold

Verifying metrics collection

Standalone, on the node (use a token with the read capability on sys/metrics, or the listener with unauthenticated_metrics_access):

curl -s --header "X-Vault-Token: ${STRONGHOLD_TOKEN}" "${STRONGHOLD_ADDR}/v1/sys/metrics?format=prometheus" | grep '^stronghold_core_unsealed'

Expected output:

stronghold_core_unsealed 1

Check collection in Prometheus:

1. Open the Status → Targets page (or /targets) and make sure that the stronghold job targets are in the UP state.

2. Run a query, for example:

   stronghold_core_unsealed

   The result must contain one series per node.

For push sinks, check the receiving side: the metrics must appear with the prefix set in metrics_prefix (stronghold by default). The server prints sink initialization errors at startup (Error initializing telemetry). To check DP, see Verifying collection in DP.

Accessing sys/metrics

The GET /sys/metrics endpoint returns metrics of the node that received the request. It is registered at the exact path /v1/sys/metrics of each listener.

Response formats

Rules for choosing the format:

• the format query parameter: prometheus returns the Prometheus text format, any other non-empty value returns 400 with the text metric response format "..." unknown;
• if format is not set, the Accept header is checked: if the first Accept value starts with application/openmetrics-text, or any Accept value contains prometheus/telemetry, the Prometheus format is returned;
• otherwise the response is JSON.

To get the metrics in the Prometheus format, run the following command:

curl --header "X-Vault-Token: ${STRONGHOLD_TOKEN}" "${STRONGHOLD_ADDR}/v1/sys/metrics?format=prometheus"

Example output:

# HELP stronghold_core_handle_request stronghold_core_handle_request
# TYPE stronghold_core_handle_request summary
stronghold_core_handle_request{quantile="0.5"} 0.071
stronghold_core_handle_request{quantile="0.9"} 0.139
stronghold_core_handle_request{quantile="0.99"} 0.612
stronghold_core_handle_request_sum 1203.4
stronghold_core_handle_request_count 8417
# HELP stronghold_core_unsealed stronghold_core_unsealed
# TYPE stronghold_core_unsealed gauge
stronghold_core_unsealed 1

To get the metrics in the JSON format, run the following command:

curl --header "X-Vault-Token: ${STRONGHOLD_TOKEN}" "${STRONGHOLD_ADDR}/v1/sys/metrics"

Example output (the values are illustrative):

{
  "Timestamp": "2026-01-15 10:21:30 +0000 UTC",
  "Gauges": [
    { "Name": "stronghold.core.unsealed", "Value": 1, "Labels": {} },
    { "Name": "stronghold.token.count", "Value": 12, "Labels": { "namespace": "root" } }
  ],
  "Points": [],
  "Counters": [
    {
      "Name": "stronghold.audit.log_request_failure",
      "Count": 0, "Rate": 0, "Sum": 0, "Min": 0, "Max": 0,
      "Mean": 0, "Stddev": 0, "Labels": {}
    }
  ],
  "Samples": [
    {
      "Name": "stronghold.core.handle_request",
      "Count": 42, "Rate": 4.2, "Sum": 3.1, "Min": 0.04, "Max": 0.3,
      "Mean": 0.074, "Stddev": 0.05, "Labels": {}
    }
  ]
}

In the JSON response, names are separated by dots; in the Prometheus format, by underscores. Gauges hold gauges, Counters hold counters, and Samples hold timers and sampled values (the summary type in Prometheus).

Access requirements

• Token. By default, the request requires a token (the X-Vault-Token or Authorization: Bearer header) whose policy grants the read capability on the sys/metrics path. The sudo capability is not required. Without a token, the response is 403 (permission denied).
• Unsealed node. A request with a token to a sealed node returns 503.
• Standby nodes. The request is not forwarded to the active node, because the token can only be validated on the active node. A standby node responds to a request with a token with the 307 code and the Location header pointing to the active node. Prometheus follows redirects by default, so scraping a standby node by token returns the metrics of the active node. To get the metrics of each node, use unauthenticated access.
• TLS. The endpoint is served by the listener over the protocol configured for it: if TLS is enabled for the listener, use HTTPS and specify the CA certificate (--cacert).
• Method. GET only for the unauthenticated variant: other methods return 405.
• Prometheus sink disabled. With prometheus_retention_time = 0, a request with format=prometheus returns 400 with the text prometheus is not enabled. The JSON format keeps working.

Token-based access

Create a dedicated least-privilege policy for the metrics collector:

path "sys/metrics" {
  capabilities = ["read"]
}

Save the policy to the prometheus-metrics.hcl file, write it to Stronghold and create a token with this policy:

d8 stronghold policy write prometheus-metrics prometheus-metrics.hcl
d8 stronghold token create -policy=prometheus-metrics -orphan -period=24h

Use a periodic token and set up its renewal, or issue the token through an auth method (for example, Kubernetes or AppRole).

Unauthenticated access

If metrics are collected from a trusted network, allow unauthenticated access at the listener level:

listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/opt/stronghold/tls/node-1-cert.pem"
  tls_key_file  = "/opt/stronghold/tls/node-1-key.pem"

  telemetry {
    unauthenticated_metrics_access = true
  }
}

Features of the unauthenticated_metrics_access parameter:

• the default value is false;
• it applies only to the listener in whose telemetry block it is set, and only to the /v1/sys/metrics path (other sys/ paths still require a token);
• the request is processed by the node itself without checking a token or forwarding, so standby nodes (and sealed ones) also return their own metrics;
• only the GET method is allowed.

Prometheus scrape job example

The following Prometheus scrape job collects metrics from three cluster nodes with token-based access:

scrape_configs:
  - job_name: stronghold
    metrics_path: /v1/sys/metrics
    params:
      format: ["prometheus"]
    scheme: https
    tls_config:
      ca_file: /etc/prometheus/stronghold-ca.pem
    authorization:
      credentials_file: /etc/prometheus/stronghold-token
    static_configs:
      - targets:
          - raft-node-1.demo.tld:8200
          - raft-node-2.demo.tld:8200
          - raft-node-3.demo.tld:8200

Scrape every cluster node rather than only the load balancer address: some metrics (Raft, runtime) are node-specific. Take into account the standby node behavior described above: with token-based access, targets that are standby nodes are redirected to the active node.

Monitoring in DP

In DP, Stronghold runs in the d8-stronghold namespace, and its configuration (including telemetry) is managed by the module through the stronghold-config ConfigMap. Metrics are collected as follows:

• the telemetry settings set metrics_prefix = "stronghold" (and also prometheus_retention_time = "1m" and disable_hostname = true), so all metrics have the stronghold_ prefix (for example, stronghold_core_unsealed);
• metrics are served by a loopback listener 127.0.0.1:8400 (without TLS, with unauthenticated_metrics_access = true) that is not reachable from outside the pod;
• the kube-rbac-proxy sidecar publishes them on port 9889 (https-metrics, path /metrics; it forwards the request to /v1/sys/metrics?format=prometheus of the loopback listener); access requires the get permission on statefulsets/prometheus-metrics (the stronghold resource, the apps API group) in the d8-stronghold namespace. The module creates the access-to-stronghold-metrics Role with this permission and, if the prometheus module is enabled, binds it to the prometheus ServiceAccount in d8-monitoring and to the d8-monitoring:scraper user.

The module creates the monitoring objects itself, no manual setup is required:

• the stronghold PodMonitor in the d8-monitoring namespace (created only if the monitoring.coreos.com/v1 PodMonitor CRD is present in the cluster);
• a PrometheusRule with the module alerts;
• three Grafana dashboards: stronghold.json, replication.json and snapshot_auto.json.

Use this variant when the DP prometheus module is enabled: nothing else is required.

Verifying collection in DP

To check that the metrics collection object exists, run the following command:

d8 k -n d8-monitoring get podmonitor stronghold

To check that Prometheus collects the metrics, open the Prometheus UI (Status → Targets): the d8-monitoring/stronghold job must have one target in the UP state per Stronghold pod. Then run a query:

stronghold_core_unsealed{job="d8-monitoring/stronghold"}

Manual metrics request in DP

To get the metrics manually, send a request to port 9889 of a pod from inside the cluster with a ServiceAccount token that has the get permission on statefulsets/prometheus-metrics. Grant the permission by binding the identity to the module Role:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: access-to-stronghold-metrics-my-sa
  namespace: d8-stronghold
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: access-to-stronghold-metrics
subjects:
  - kind: ServiceAccount
    name: my-sa
    namespace: my-namespace

Then send the request from a pod that uses this ServiceAccount. The proxy uses its own certificate, so the command uses --insecure, like the module PodMonitor:

TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
curl --insecure --header "Authorization: Bearer ${TOKEN}" "https://<POD_IP>:9889/metrics"

In this command, <POD_IP> is the IP address of a Stronghold pod. To get the pod IP addresses, run the following command:

d8 k -n d8-stronghold get pods -l app.kubernetes.io/name=stronghold,component=server -o wide

External Prometheus in DP

The module configures metrics collection only for the Prometheus of the DP monitoring stack. An external Prometheus can collect metrics if:

• it can reach port 9889 of the Stronghold pods over HTTPS;
• it scrapes the /metrics path and sends a Kubernetes bearer token of an identity with the get permission on statefulsets/prometheus-metrics (see the RoleBinding above);
• you create the scrape configuration yourself: the module PodMonitor is meant for the main Prometheus of the DP monitoring stack.

Pushing metrics to StatsD, DogStatsD, Circonus, and Stackdriver is not available in DP, because the telemetry block is generated by the module and cannot be changed through the module parameters.

Module alerts in DP

The alerts are defined in the module’s PrometheusRule and use the severity_level label, as DP alerts do:

Custom alerts in DP are defined with CustomPrometheusRules: move the spec.groups content from the examples below into it. For example, to get alerts on automatic snapshot failures, use the stronghold.snapshots group.

Module parameters affecting monitoring

The module has no dedicated monitoring parameters. These settings affect monitoring and alerts:

The telemetry block of the DP configuration is generated by the module and contains only the following values: prometheus_retention_time = "1m", disable_hostname = true, metrics_prefix = "stronghold". The listener 127.0.0.1:8400 has unauthenticated_metrics_access = true. Other telemetry parameters are not available in DP.

Configuration parameters

The telemetry block

Parameters of the telemetry block of the server configuration file (standalone). In DP, these parameters cannot be changed, see Module parameters affecting monitoring.

Duration parameters accept a string with a unit suffix (s, m, h, and d for days) or an integer number of seconds.

The circonus_* parameters are described in more detail in the Circonus gometrics documentation.

Listener telemetry block

The telemetry block inside listener "tcp" has its own parameter:

Key metrics

Metric names are given with the stronghold_ prefix: Stronghold uses it by default in both standalone installations and DP. You can change the prefix with the metrics_prefix parameter of the telemetry block. The full list of metrics is in the Metrics catalog.

The type is given as in the Prometheus format: the summary type includes the quantile label (0.5, 0.9, 0.99) and the _sum and _count series.

Alerting rule examples

A PrometheusRule example with a basic set of alerts. It is a template for standalone installations: it uses the severity label and standalone paths. For DP, see Module alerts in DP. Tune the thresholds to your workload.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: stronghold
spec:
  groups:
    - name: stronghold.availability
      rules:
        - alert: StrongholdSealed
          expr: stronghold_core_unsealed == 0
          for: 1m
          labels:
            severity: critical
          annotations:
            summary: "Stronghold node {{ $labels.instance }} is sealed"
            description: "The node does not serve requests. Unseal it or check auto unseal."
        - alert: StrongholdNoActiveNode
          expr: sum(stronghold_core_active) < 1
          for: 2m
          labels:
            severity: critical
          annotations:
            summary: "The Stronghold cluster has no active node"
            description: "Check Raft and quorum state with d8 stronghold operator raft list-peers."
        - alert: StrongholdLeaderFlapping
          expr: increase(stronghold_raft_state_candidate[15m]) > 3
          labels:
            severity: warning
          annotations:
            summary: "Frequent Raft leader elections on {{ $labels.instance }}"
            description: "Check network connectivity between nodes and disk latency."
        - alert: StrongholdAutopilotUnhealthy
          expr: stronghold_autopilot_healthy == 0
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Autopilot reports the Stronghold cluster as unhealthy"
    - name: stronghold.performance
      rules:
        - alert: StrongholdHighRequestLatency
          expr: stronghold_core_handle_request{quantile="0.99"} > 500
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "99th percentile request handling time is above 500 ms"
        - alert: StrongholdLeaseCountGrowth
          expr: delta(stronghold_expire_num_leases[1h]) > 10000
          labels:
            severity: warning
          annotations:
            summary: "Lease count grew by more than 10000 in an hour"
            description: "Check lease TTLs and clients that do not reuse tokens."
        - alert: StrongholdLeaseCountHigh
          expr: stronghold_expire_num_leases > 250000
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "Active lease count exceeds 250000"
    - name: stronghold.audit
      rules:
        - alert: StrongholdAuditFailures
          expr: increase(stronghold_audit_log_request_failure[5m]) > 0 or increase(stronghold_audit_log_response_failure[5m]) > 0
          labels:
            severity: critical
          annotations:
            summary: "Stronghold audit device write failures"
            description: "If all audit devices fail, Stronghold stops serving requests."
    - name: stronghold.snapshots
      rules:
        - alert: StrongholdAutoSnapshotFailed
          expr: increase(stronghold_autosnapshots_save_errors[1h]) > 0
          labels:
            severity: warning
          annotations:
            summary: "Automatic snapshot {{ $labels.snapshot_config_name }} failed"
            description: "Check the server logs and the snapshot storage."
    - name: stronghold.certificates
      rules:
        - alert: StrongholdTLSCertificateExpiringSoon
          expr: (probe_ssl_earliest_cert_expiry{job="stronghold-tls"} - time()) / 86400 < 21
          for: 1h
          labels:
            severity: warning
          annotations:
            summary: "Stronghold TLS certificate on {{ $labels.instance }} expires in less than 21 days"
    - name: stronghold.storage
      rules:
        - alert: StrongholdRaftDiskUsageHigh
          expr: |
            (1 - node_filesystem_avail_bytes{mountpoint="/opt/stronghold/data"}
              / node_filesystem_size_bytes{mountpoint="/opt/stronghold/data"}) > 0.8
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "Raft data volume on {{ $labels.instance }} is more than 80% full"

Notes on the example:

• The TLS expiry alert uses the probe_ssl_earliest_cert_expiry metric from blackbox_exporter probing the Stronghold HTTPS address. Monitor root and intermediate certificates issued by the Stronghold PKI engine the same way, for example with a certificate exporter that checks files or service addresses.
• The disk usage alert uses node-exporter metrics. Specify the mount point of the directory set in the path parameter of the storage "raft" block. In DP, the data of each replica is stored in the data-stronghold-N PVC, which is mounted at /stronghold/data inside the pod. If the storageClass parameter is not set in the module configuration, the PVC is bound to a local PV in the /var/lib/deckhouse/stronghold/<LOCAL_PATH_UUID> directory on a master or arbiter node: specify the mount point of the file system that contains this directory. If storageClass is set, the volume is provisioned by that storage class, and the example’s mount point does not apply.
• For DP, move spec.groups into CustomPrometheusRules.

Stronghold does not export PKI certificate expiry metrics (only the tidy metrics secrets_pki_tidy_* exist). Monitor certificate expiry with external tools, for example blackbox-exporter.

Health checks via sys/health

The GET /sys/health endpoint requires no authentication and is suitable for load balancer and external monitoring checks:

curl -s -o /dev/null -w "%{http_code}\n" "${STRONGHOLD_ADDR}/v1/sys/health"

Response codes:

Response codes can be overridden with query parameters:

• standbyok=true — return 200 for standby nodes. Use it for a load balancer that distributes requests across all unsealed nodes;
• activecode, standbycode, sealedcode, uninitcode, drsecondarycode, performancestandbycode, isleaderreadycode, raftautopilotcode — set a custom code for the corresponding state;
• perfstandbyok=true — return 200 for performance standby nodes.

Example check for a load balancer that should route traffic to the active node only:

curl -s -o /dev/null -w "%{http_code}\n" "${STRONGHOLD_ADDR}/v1/sys/health?standbycode=503"

To check Raft state, use:

d8 stronghold operator raft list-peers
d8 stronghold operator raft autopilot state