Prometheus supports the remote_write mechanism for sending data from a local Prometheus instance to a separate long-term storage (for example, VictoriaMetrics). In DKP, support for this mechanism is implemented using the PrometheusRemoteWrite custom resource.

For VictoriaMetrics, detailed information about ways to send data to vmagent can be found in the VictoriaMetrics documentation.

Example of minimal PrometheusRemoteWrite

Below is an example of minimal configuration for the PrometheusRemoteWrite custom resource:

apiVersion: deckhouse.io/v1
kind: PrometheusRemoteWrite
metadata:
  name: test-remote-write
spec:
  url: https://victoriametrics-test.domain.com/api/v1/write

Example of advanced PrometheusRemoteWrite

Below is an example of advanced configuration for the PrometheusRemoteWrite custom resource:

apiVersion: deckhouse.io/v1
kind: PrometheusRemoteWrite
metadata:
  name: test-remote-write
spec:
  url: https://victoriametrics-test.domain.com/api/v1/write
  basicAuth:
    username: username
    password: password
  writeRelabelConfigs:
  - sourceLabels: [__name__]
    action: keep
    regex: prometheus_build_.*|my_cool_app_metrics_.*
  - sourceLabels: [__name__]
    action: drop
    regex: my_cool_app_metrics_with_sensitive_data

For a complete description of the custom resource fields, refer to the prometheus module documentation.