Metrics catalog

This page lists the metrics that Stronghold exports in the Prometheus format. For how to enable telemetry and collect the metrics, see Monitoring.

To get the actual list of metrics from a live node, request the endpoint and select the lines with types:

curl -s --header "X-Vault-Token: ${STRONGHOLD_TOKEN}" "${STRONGHOLD_ADDR}/v1/sys/metrics?format=prometheus" | grep '^# TYPE'

Naming rules

A metric name in the Prometheus output is built from the metric key by the following rules:

1. The name starts with the prefix stronghold_. The prefix is the metrics_prefix parameter of the telemetry block (stronghold by default). The names on this page are given with the default prefix.
2. The parts of the key are joined with _, and then the characters space, ., =, - and / are replaced with _. The case is kept: a key raft, commitTime gives stronghold_raft_commitTime; a key core, seal-with-request gives stronghold_core_seal_with_request.
3. Names with variable parts are written with a placeholder in angle brackets, for example stronghold_audit_<device>_log_request.
4. With the Prometheus sink alone, the node hostname is never added to metric names. If two or more sinks are enabled and disable_hostname = false, the hostname is inserted into the names of gauges (and only gauges), for example stronghold_<hostname>_core_active. With disable_hostname = true, metric names are the same on all nodes. With enable_hostname_label = true the host label with the node name is added to all metrics instead.

Mapping of metric kinds to Prometheus types:

Features of the Prometheus output:

• A series that has not been updated for prometheus_retention_time (24 hours by default) is removed from the output. Rarely updated metrics can disappear and appear again. With prometheus_retention_time = 0 the Prometheus sink is disabled.
• The HELP text of a metric is the same as its name, the sink does not set other descriptions.
• Values are passed as 32-bit floating-point numbers, so large values (Unix timestamps, Raft indexes, byte counts) are rounded to about 7 significant digits.
• In addition to the Stronghold metrics, the endpoint returns the standard Go and process metrics (go_*, process_*).
• A number of timers and counters share one name. Such a collision is noted in the corresponding sections.

Labels

The set of labels depends on the metric. The cluster label is added to most Stronghold core metrics; the Raft metrics and the metrics of some components have no cluster label. In the tables, the labels of every metric are listed explicitly.

Metrics used in DP alerts

The Deckhouse module alerts use the following metrics. The alerts are described in Monitoring.

Metrics by subsystem

The tables are grouped by subsystem. The type is the type in the Prometheus output; the timers (summary) are in milliseconds unless stated otherwise.

Core, seal state and HA

State gauges are refreshed every second on all nodes of the cluster and, additionally, at the moment of the event (unseal, seal, leadership change).

State

Timers

Seal and unseal key protection

Metrics of encrypt and decrypt calls to the seal (the unseal key protection mechanism: Shamir, KMS, Transit, PKCS#11 and others). Metrics with <type> in the name are duplicated for each seal wrapper type: shamir, aead, awskms, azurekeyvault, gcpckms, alicloudkms, ocikms, transit, pkcs11, hsm-auto (becomes hsm_auto), and so on.

Requests, login and routing

Request handling

Request forwarding between nodes (standby nodes)

Rollback manager and events

Tokens, secret leases and usage gauges

Usage gauges (token_count*, expire_leases_by_expiration, secret_kv_count, identity_entity_*count, identity_entity_active_partial_month) are collected only on the active node by background collection processes. The period is set by usage_gauge_period (10 minutes by default; none disables collection), the number of series per gauge is limited by maximum_gauge_cardinality (500 by default, the series with the biggest values are kept). Collection is not performed on standby nodes and DR secondary clusters. If a collection takes more than a set share of the period, the period is doubled until restart.

Creation and operations

Usage gauges (active node)

Service metrics of gauge collection

Leases and expiration

The expiration manager metrics. expire_num_leases and expire_num_irrevocable_leases are emitted every second by the active node only.

Counters and queue

Timers

Identity and client activity

Audit

In the metrics with <device>, <device> is the name of the audit device (the mount path with - and / replaced by _). Failure counters are incremented by 1 on a failed request or response logging and by 0 on success.

Policies and ACL

Barrier, cache and storage backends

The barrier encrypts all data before it is written to the physical backend. The storage timers measure the calls of the Stronghold core to the backend; for Raft see also the Raft section.

Barrier

Physical storage cache

Storage packer

Other storage backends (only for the configured backend)

Raft storage

Metrics of the integrated Raft storage emitted by Stronghold itself. The gauges raft_storage_stats_* and raft_storage_bolt_* are updated every second on every node. The database label of the BoltDB statistics is logstore (raft.db, the log store) or fsm (the FSM database). The metrics of the Raft library are described in the next section.

Raft statistics

BoltDB statistics

Followers (emitted by the Raft leader)

Operation timers

Raft library

Metrics of the Raft storage. They have no cluster label. Metrics emitted only by the leader or only by followers are noted in the descriptions. Stronghold also emits legacy metrics with the peer ID as a name suffix (stronghold_raft_replication_heartbeat_<peer_id>, stronghold_raft_replication_installSnapshot_<peer_id>, stronghold_raft_replication_appendEntries_rpc_<peer_id>, stronghold_raft_replication_appendEntries_logs_<peer_id>), which duplicate the labeled metrics; use the labeled ones. The rpcType label of raft_net_* is one of Heartbeat, AppendEntries, RequestVote, RequestPreVote, InstallSnapshot, TimeoutNow. Latency metrics are in milliseconds.

State and elections

Log replication and RPC

Replication from the leader and network transport

FSM, snapshots and goroutine saturation

BoltDB log store (raft-boltdb)

Raft autopilot and cluster health

Autopilot metrics are emitted by the Raft leader each time the autopilot state is updated. They have no cluster label (to count nodes, use sum/max over series from all nodes). The metrics of the autopilot library itself are not exported; the state is published by Stronghold.

Replication (PR, DR) and WAL

The metrics of replication components (WAL, Merkle trees, log shipping, replication RPC) and replication roles. Available in Stronghold EE. They are emitted only if the node uses the corresponding replication mode. They have no cluster label, except the role gauges core_replication_*.

Replication roles (every second on all nodes)

WAL

Merkle trees and synchronization

Log shipper

Replication RPC: server

The method label (get or update) is present on user_failed_login_info metrics.

Replication RPC: client

Replication RPC: server for standby nodes

Replication RPC: DR server

KV replication

Metrics of KV secrets replication between mounts. The records are identified by the pair of the source mount and the local mount. The gauges are exported every second; the summaries and the counter are updated after each replication run. The metrics have no cluster label.

Automatic snapshots

Metrics of automatic Raft snapshots (sys/storage/raft/snapshot-auto). Available in Stronghold EE. snapshot_config_name is the name of the automatic snapshot configuration. The metrics are emitted by the node that performs the snapshot (the active node).

The DP module alerts D8StrongholdAutoSnapshotFailed and D8StrongholdAutoSnapshotRotationFailed use stronghold_core_snapshot_auto_failed and stronghold_autosnapshots_rotate_failed. For your own alerts on failed automatic snapshots, you can also use stronghold_autosnapshots_save_errors and stronghold_autosnapshots_last_success_time. An alert example is in Alerting rule examples.

PKI secrets engine

In the names of the request metrics, <mount> is the mount point of the PKI engine without the trailing slash, where /, - and . are replaced by _ (for example, the mount pki_int/ gives pki_int). The timer and the counter of one operation have the same name, so in the Prometheus output one of these series can be dropped as inconsistent in type; check the output on a live node.

Requests

Certificate counts

Tidy operation

Database secrets engine

The metrics are emitted by the wrapper of database plugins. In the names, <operation> is the plugin operation and <db_type> is the database type returned by the plugin (for example, postgresql; - is replaced by _). Operations of plugins of the current interface: Initialize, NewUser, UpdateUser, DeleteUser, Close, PluginVersion; of legacy plugins: Initialize, CreateUser, RenewUser, RevokeUser, RotateRootCredentials, GenerateCredentials, SetCredentials, Close. The timer and the counter of an operation have the same name (see the note in the PKI section).

The collection process of secrets_database_backend_pluginInstances_count reports its own metrics stronghold_metrics_collection_interval, stronghold_metrics_collection and stronghold_metrics_collection_error without labels. These metrics are described in Tokens, secret leases and usage gauges.

Other secrets engines and plugins

Quotas

Metrics of rate limit and lease count quotas. For rate limit quotas, see also Quotas.

Runtime and process

The Go runtime metrics are collected every second (enabled by default). They have no cluster label. In addition, the Prometheus output contains the standard Go and process metrics of the client_golang library (go_*, process_*), which are not described here.