The security-events-manager module: Advanced usage

Viewing controller and gateway logs

Controller runs in the module namespace (e.g. d8-security-events-manager). Gateway runs in the same namespace.

# Controller logs
kubectl -n d8-security-events-manager logs -l app=security-events-manager-controller -c manager --tail=200

# Gateway (Vector) logs
kubectl -n d8-security-events-manager logs -l app=gateway -c vector --tail=200

# Reloader sidecar (applies config changes, triggers Vector reload)
kubectl -n d8-security-events-manager logs -l app=gateway -c reloader --tail=100

Inspecting the generated Vector config

The controller writes the full Vector config to Secret gateway-vector-config in the module namespace. To dump the current config:

kubectl -n d8-security-events-manager get secret gateway-vector-config -o jsonpath='{.data.vector\.json}' | base64 -d | jq .

Without jq, omit the trailing | jq . to get raw JSON. Check that sources, transforms, and sinks match your ClusterSecurityEventConfig and ClusterSecurityEventDestination. More details about transform names can be found in Vector gateway config and Detailed VRL logic.

Debugging Vector inside the gateway pod

The Vector API (port 8686) is bound to 127.0.0.1 (loopback only) and is not published in the gateway Service. This means other pods in the cluster cannot reach it, but you can access it from inside the gateway pod via kubectl exec or port-forward to the pod (not the Service).

vector top — live component metrics and event flow

# Interactive TUI inside the pod
kubectl -n d8-security-events-manager exec -it deploy/gateway -c vector -- vector top

Or from your workstation via port-forward to the pod:

kubectl -n d8-security-events-manager port-forward deploy/gateway 8686:8686
vector top --url http://127.0.0.1:8686

Vector CLI — config validation and topology

# Validate the runtime config (checks VRL compilation, sink/source schema, etc.)
kubectl -n d8-security-events-manager exec -it deploy/gateway -c vector -- vector validate /etc/vector/dynamic/vector.json

# Render the pipeline topology as a DOT graph
kubectl -n d8-security-events-manager exec -it deploy/gateway -c vector -- vector graph /etc/vector/dynamic/vector.json

# List all components (sources, transforms, sinks) and their config
kubectl -n d8-security-events-manager exec -it deploy/gateway -c vector -- vector list /etc/vector/dynamic/vector.json

Health and reload status via the reloader

The reloader sidecar listens on 0.0.0.0:9255 and exposes a composite health endpoint that checks Vector API health (GET /health on 127.0.0.1:8686), config apply state, and fsnotify watcher status — strictly better than probing Vector’s /health directly:

kubectl -n d8-security-events-manager exec deploy/gateway -c reloader -- wget -qO- http://127.0.0.1:9255/reloader/healthz

The reloader’s reloadVector() uses SIGHUP directly — there is no POST /reload HTTP endpoint. Config reloads are triggered automatically by the reloader when the gateway-vector-config Secret changes; there is no need to call any API manually.

Metrics and dashboards

Vector internal metrics are exported via prometheus_exporter on 127.0.0.1:9090 inside the pod and scraped by the gateway PodMonitor through kube-rbac-proxy (port 9254, path /metrics). Use the Grafana dashboards shipped with the module (security-events-manager, security-events-manager-loki) or query Prometheus directly:

# Example: check buffer usage via Prometheus
kubectl -n d8-monitoring exec prometheus-0 -- promtool query instant \
  'vector_buffer_byte_size{job="gateway"}'

Troubleshooting

• Events not reaching a destination — verify the source string in ClusterSecurityEventConfig (enabledSources or enabledSourcesMasks). Format is clusterSecurityEventShipper/<name>/<source> or podSecurityEventShipper/<namespace>/<name>/<source>. More details about event delivery diagnostics can be found in FAQ.

• Invalid Vector config — after changing CRs, check controller logs for reconcile errors. If the generated config is invalid, the reloader will not apply it and will set metric security_events_manager_gateway_config_validation_error=1. Check reloader logs for the validation error.

• TLS/certificate errors — ensure ClusterSecurityEventDestination TLS and auth settings match your Loki/Elasticsearch/etc. For gateway-to-destination TLS, the ca field is base64-encoded PEM. The controller may create a CA Secret for destinations; check Secret gateway-destination-cas in the module namespace.

• Plugin enrichment (enrich.source: Plugin) not resolving fields — the enrichment-cache sidecar serves the unified HTTP enrichment endpoint that Vector’s Lua transform calls. Check the sidecar is Ready (it reports 503 on /readyz until its Pod informer cache finishes the initial sync; the NodeUser cache is best-effort and does not block readiness) and that the gateway pod is Ready. Inspect logs and probes:

  kubectl -n d8-security-events-manager logs -l app=gateway -c enrichment-cache --tail=200
  kubectl -n d8-security-events-manager exec deploy/gateway -c enrichment-cache -- wget -qO- http://127.0.0.1:9260/readyz
  kubectl -n d8-security-events-manager exec deploy/gateway -c enrichment-cache -- wget -qO- "http://127.0.0.1:9261/api/v1/enrich?plugin=k8s-container-info&container_id=test"
  kubectl -n d8-security-events-manager exec deploy/gateway -c enrichment-cache -- wget -qO- "http://127.0.0.1:9261/api/v1/enrich?plugin=k8s-nodeuser-info&uid=1234"

  The sidecar exposes two listeners: 0.0.0.0:9260 for health and readiness probes only, and 127.0.0.1:9261 for the enrichment API and metrics (loopback only — reachable by Vector’s Lua transform and by the kube-rbac-proxy sidecar in the same pod, and by nothing else).

  Scrape metrics via the kube-rbac-proxy path /enrichment-cache/metrics and look at enrichment_cache_lookup_misses_total / enrichment_cache_nodeuser_lookup_misses_total (lookup returned not-found) and enrichment_cache_informer_synced / enrichment_cache_nodeuser_informer_synced (cache sync status). Common causes: pod_name/namespace args point at fields absent from the event, container_id carries a runtime prefix that was not stripped, the Pod was deleted (informer cache lag), the NodeUser with that uid does not exist (check kubectl get nodeuser), or value is not in the allowlist (serviceAccountName, name, namespace for pod plugins; username for k8s-nodeuser-info). Recall enrichment is non-fatal — a missing record leaves the target field unset and does not drop the event. See Detailed VRL logic for the generated behavior.

Buffer tuning

The module uses a two-layer buffer model to prevent data loss during temporary destination outages. Both layers default to Disk + Block for zero data loss of security events.

Choosing Disk vs Memory

Choosing Block vs DropNewest

Calculating buffer size

To estimate how long events can be buffered during a destination outage:

Buffer duration (hours) = maxSize_bytes / (average_event_size_bytes × events_per_second × 3600)

> `maxSize` is specified as a Kubernetes quantity string (e.g. `"512Mi"`, `"1Gi"`).
> Convert to bytes for the formula: `512Mi` = 536870912 bytes.

For example, with defaults (512 MiB buffer, ~500 bytes per event, 10 events/sec):

536870912 / (500 × 10 × 3600) ≈ 29.8 hours   (512Mi default)

With 10,000 events/sec and 1 KB events:

536870912 / (1024 × 10000 × 3600) ≈ 0.15 hours (~9 minutes)

Increase gateway.buffer.maxSize if your event rate is high or you expect prolonged outages.

Backpressure cascading

When a destination is unreachable and whenFull: Block is set:

1. Gateway sink buffer fills up → backpressure to route_by_destination transform
2. Upstream transforms block → gateway_vector source stops accepting events
3. Log-shipper agents buffer on their side (Layer 1, gateway.logShipperBuffer)
4. If agent buffers also fill, log-shipper applies backpressure to its sources (pod logs, file tails)

This cascading backpressure ensures no events are lost — the entire pipeline slows down instead of dropping data.

Resource considerations for Disk buffer

When using Disk buffer type, ensure the gateway pod has sufficient ephemeral storage:

• The vector-data emptyDir is sized via sizeLimit to match gateway.buffer.maxSize
• Each gateway replica uses up to maxSize (e.g. 512Mi) of node storage
• With multiple replicas, total disk usage = replicas × maxSize
• Consider adding node ephemeral storage limits or using nodes with sufficient local storage

Sending events in CEF format

Security events can be sent in Common Event Format (CEF) for integration with SIEM systems (Splunk, ArcSight, QRadar, etc.).

CEF encoding is configured per-destination in ClusterSecurityEventDestination. It is supported on Kafka, Vector, File, Console, and Socket destinations. Loki, Elasticsearch, and SplunkHEC always use JSON.

Basic CEF over Kafka

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: siem-kafka
spec:
  type: Kafka
  kafka:
    brokers:
      - "siem-kafka:9092"
    topic: "security-events"
    encoding:
      codec: CEF

Events will be sent as bare CEF strings. The gateway maps SecurityEvent.event.severity (Low/Medium/High/Critical) to CEF numeric severity (1/5/8/10), and uses event.code as the CEF signature ID and event.description as the CEF name.

CEF with syslog wrapping

For SIEM systems that expect syslog-framed CEF (common for UDP/TCP syslog collectors):

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: siem-syslog
spec:
  type: Kafka
  kafka:
    brokers:
      - "siem-kafka:9092"
    topic: "security-events"
    encoding:
      codec: CEF
      syslogWrapper: RFC5424

Supported syslog wrappers: RFC3164 (BSD syslog), RFC5424 (IETF syslog), or None (default, bare CEF).

Custom CEF metadata

Override the CEF device vendor/product/version per-destination:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: siem-custom
spec:
  type: Kafka
  kafka:
    brokers:
      - "siem-kafka:9092"
    topic: "security-events"
    encoding:
      codec: CEF
      cef:
        deviceVendor: "MyCompany"
        deviceProduct: "k8s-security"
        deviceVersion: "2.0"

Or set defaults globally in ClusterSecurityEventConfig:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventConfig
metadata:
  name: default
spec:
  defaultSeverityThreshold: Low
  destinations:
    - siem-kafka
  cef:
    deviceVendor: "MyCompany"
    deviceProduct: "k8s-security"
    deviceVersion: "2.0"

Per-destination encoding.cef overrides take precedence over CSEC spec.cef defaults.

The cluster-wide defaults come from a single ClusterSecurityEventConfig: the first one by name that sets spec.cef. Its spec.cef block is taken as a whole, and the spec.cef blocks of the other configs are ignored. The module generates the default config and fills its spec.cef from the securityEventConfig.cef section of the security-events-manager ModuleConfig, so set the cluster-wide defaults there rather than editing the generated object. A config of your own that sets spec.cef and sorts before default replaces these defaults entirely.

CEF format version

By default the gateway writes the CEF:0 prefix in the header of every event. Set encoding.cef.version to V1 when the receiving SIEM expects the CEF:1 prefix.

The example below sends events to a syslog collector that parses CEF:1:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: siem-socket
spec:
  type: Socket
  socket:
    address: "siem.example.com:514"
    mode: TCP
    encoding:
      codec: CEF
      cef:
        version: V1

Possible values are V0, which writes CEF:0 and is used by default, and V1, which writes CEF:1. Only the header prefix changes; the header fields and the extensions stay the same.

Do not confuse version with the neighbouring deviceVersion. version selects the format of the CEF: prefix and accepts only V0 and V1, while deviceVersion is the fourth field of the header and holds the version of the device that reports the event.

The default for the whole cluster is set in ClusterSecurityEventConfig as spec.cef.version, and a per-destination encoding.cef.version overrides it.

CEF extensions

The extension part of a CEF message carries the SecurityEvent fields as key=value pairs. Every message contains the base set of pairs listed in the table below. A pair is omitted when the event has no such field or the field is empty.

The event code is not repeated in the extensions: it is the signature ID field of the header.

To send other fields, for example the counters a module puts into metadata.extra, list them in encoding.cef.extensions. The example below adds the number of critical and high vulnerabilities and the image repository to the events sent to a syslog collector:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: siem-socket
spec:
  type: Socket
  socket:
    address: "siem.example.com:514"
    mode: TCP
    encoding:
      codec: CEF
      syslogWrapper: RFC5424
      cef:
        extensions:
          - key: cn1
            label: critical
            field: metadata.extra.critical
          - key: cn2
            label: high
            field: metadata.extra.high
          - key: cs4
            label: repository
            field: metadata.extra.repository

An event with these fields ends with the following pairs:

cn1=5 cn1Label=critical cn2=63 cn2Label=high cs4=excel-transformer/ci cs4Label=repository

Each item of the list has the following parameters:

• key — the key of the pair. Use a key of the CEF extension dictionary, so the SIEM parses the value into a known field. A key outside the dictionary is written as is; check that the receiving SIEM accepts it.
• label — the name of the value, written as the <KEY>Label pair. It is required for the custom keys cs1–cs6, cn1–cn3, cfp1–cfp4, c6a1–c6a4, deviceCustomDate1–deviceCustomDate2, flexString1–flexString2, flexNumber1–flexNumber2 and flexDate1, and is not allowed for other keys.
• field — the dot-separated path to the SecurityEvent field. An object or an array value is written as JSON.

An item whose key is in the base set replaces the pair of the base set in its place. For example, an item with key: cs1 sends another field in cs1 instead of object.name.

The list for the whole cluster is set in the securityEventConfig.cef.extensions parameter of the ModuleConfig, which fills spec.cef.extensions of the generated default ClusterSecurityEventConfig (see Custom CEF metadata for how the cluster-wide CEF defaults are chosen). A list in encoding.cef.extensions of a destination replaces it as a whole, and an empty list (extensions: []) leaves the destination with the base set only.