The security-events-manager module: Advanced usage

Viewing controller and gateway logs

Controller runs in the module namespace (e.g. d8-security-events-manager). Gateway runs in the same namespace.

# Controller logs
kubectl -n d8-security-events-manager logs -l app=security-events-manager-controller -c manager --tail=200

# Gateway (Vector) logs
kubectl -n d8-security-events-manager logs -l app=gateway -c vector --tail=200

# Reloader sidecar (applies config changes, triggers Vector reload)
kubectl -n d8-security-events-manager logs -l app=gateway -c reloader --tail=100

Inspecting the generated Vector config

The controller writes the full Vector config to Secret gateway-vector-config in the module namespace. To dump the current config:

kubectl -n d8-security-events-manager get secret gateway-vector-config -o jsonpath='{.data.vector\.json}' | base64 -d | jq .

Without jq, omit the trailing | jq . to get raw JSON. Check that sources, transforms, and sinks match your ClusterSecurityEventConfig and ClusterSecurityEventDestination. More details about transform names can be found in Vector gateway config and Detailed VRL logic.

Debugging Vector inside the gateway pod

The Vector API (port 8686) is bound to 127.0.0.1 (loopback only) and is not published in the gateway Service. This means other pods in the cluster cannot reach it, but you can access it from inside the gateway pod via kubectl exec or port-forward to the pod (not the Service).

vector top — live component metrics and event flow

# Interactive TUI inside the pod
kubectl -n d8-security-events-manager exec -it deploy/gateway -c vector -- vector top

Or from your workstation via port-forward to the pod:

kubectl -n d8-security-events-manager port-forward deploy/gateway 8686:8686
vector top --url http://127.0.0.1:8686

Vector CLI — config validation and topology

# Validate the runtime config (checks VRL compilation, sink/source schema, etc.)
kubectl -n d8-security-events-manager exec -it deploy/gateway -c vector -- vector validate /etc/vector/dynamic/vector.json

# Render the pipeline topology as a DOT graph
kubectl -n d8-security-events-manager exec -it deploy/gateway -c vector -- vector graph /etc/vector/dynamic/vector.json

# List all components (sources, transforms, sinks) and their config
kubectl -n d8-security-events-manager exec -it deploy/gateway -c vector -- vector list /etc/vector/dynamic/vector.json

Health and reload status via the reloader

The reloader sidecar listens on 0.0.0.0:9255 and exposes a composite health endpoint that checks Vector API health (GET /health on 127.0.0.1:8686), config apply state, and fsnotify watcher status — strictly better than probing Vector’s /health directly:

kubectl -n d8-security-events-manager exec deploy/gateway -c reloader -- wget -qO- http://127.0.0.1:9255/reloader/healthz

The reloader’s reloadVector() uses SIGHUP directly — there is no POST /reload HTTP endpoint. Config reloads are triggered automatically by the reloader when the gateway-vector-config Secret changes; there is no need to call any API manually.

Metrics and dashboards

Vector internal metrics are exported via prometheus_exporter on 127.0.0.1:9090 inside the pod and scraped by the gateway PodMonitor through kube-rbac-proxy (port 9254, path /metrics). Use the Grafana dashboards shipped with the module (security-events-manager, security-events-manager-loki) or query Prometheus directly:

# Example: check buffer usage via Prometheus
kubectl -n d8-monitoring exec prometheus-0 -- promtool query instant \
  'vector_buffer_byte_size{job="gateway"}'

Troubleshooting

• Events not reaching a destination — verify the source string in ClusterSecurityEventConfig (enabledSources or enabledSourcesMasks). Format is clusterSecurityEventShipper/<name>/<source> or podSecurityEventShipper/<namespace>/<name>/<source>. More details about event delivery diagnostics can be found in FAQ.

• Invalid Vector config — after changing CRs, check controller logs for reconcile errors. If the generated config is invalid, the reloader will not apply it and will set metric security_events_manager_gateway_config_validation_error=1. Check reloader logs for the validation error.

• TLS/certificate errors — ensure ClusterSecurityEventDestination TLS and auth settings match your Loki/Elasticsearch/etc. For gateway-to-destination TLS, the ca field is base64-encoded PEM. The controller may create a CA Secret for destinations; check Secret gateway-destination-cas in the module namespace.

• Plugin enrichment (enrich.source: Plugin) not resolving fields — the enrichment-cache sidecar serves the unified HTTP enrichment endpoint that Vector’s Lua transform calls. Check the sidecar is Ready (it reports 503 on /readyz until its Pod informer cache finishes the initial sync; the NodeUser cache is best-effort and does not block readiness) and that the gateway pod is Ready. Inspect logs and probes:

  kubectl -n d8-security-events-manager logs -l app=gateway -c enrichment-cache --tail=200
  kubectl -n d8-security-events-manager exec deploy/gateway -c enrichment-cache -- wget -qO- http://127.0.0.1:9260/readyz
  kubectl -n d8-security-events-manager exec deploy/gateway -c enrichment-cache -- wget -qO- "http://127.0.0.1:9261/api/v1/enrich?plugin=k8s-container-info&container_id=test"
  kubectl -n d8-security-events-manager exec deploy/gateway -c enrichment-cache -- wget -qO- "http://127.0.0.1:9261/api/v1/enrich?plugin=k8s-nodeuser-info&uid=1234"

  The sidecar exposes two listeners: 0.0.0.0:9260 for health/readiness/metrics (probes and kube-rbac-proxy) and 127.0.0.1:9261 for the enrichment API (loopback only — only Vector’s Lua transform in the same pod can reach it).

  Scrape metrics via the kube-rbac-proxy path /enrichment-cache/metrics and look at enrichment_cache_lookup_misses_total / enrichment_cache_nodeuser_lookup_misses_total (lookup returned not-found) and enrichment_cache_informer_synced / enrichment_cache_nodeuser_informer_synced (cache sync status). Common causes: pod_name/namespace args point at fields absent from the event, container_id carries a runtime prefix that was not stripped, the Pod was deleted (informer cache lag), the NodeUser with that uid does not exist (check kubectl get nodeuser), or value is not in the allowlist (serviceAccountName, name, namespace for pod plugins; username for k8s-nodeuser-info). Recall enrichment is non-fatal — a missing record leaves the target field unset and does not drop the event. See Detailed VRL logic for the generated behavior.

Buffer tuning

The module uses a two-layer buffer model to prevent data loss during temporary destination outages. Both layers default to Disk + Block for zero data loss of security events.

Choosing Disk vs Memory

Choosing Block vs DropNewest

Calculating buffer size

To estimate how long events can be buffered during a destination outage:

Buffer duration (hours) = maxSize_bytes / (average_event_size_bytes × events_per_second × 3600)

> `maxSize` is specified as a Kubernetes quantity string (e.g. `"512Mi"`, `"1Gi"`).
> Convert to bytes for the formula: `512Mi` = 536870912 bytes.

For example, with defaults (512 MiB buffer, ~500 bytes per event, 10 events/sec):

536870912 / (500 × 10 × 3600) ≈ 29.8 hours   (512Mi default)

With 10,000 events/sec and 1 KB events:

536870912 / (1024 × 10000 × 3600) ≈ 0.15 hours (~9 minutes)

Increase gateway.buffer.maxSize if your event rate is high or you expect prolonged outages.

Backpressure cascading

When a destination is unreachable and whenFull: Block is set:

1. Gateway sink buffer fills up → backpressure to route_by_destination transform
2. Upstream transforms block → gateway_vector source stops accepting events
3. Log-shipper agents buffer on their side (Layer 1, gateway.logShipperBuffer)
4. If agent buffers also fill, log-shipper applies backpressure to its sources (pod logs, file tails)

This cascading backpressure ensures no events are lost — the entire pipeline slows down instead of dropping data.

Resource considerations for Disk buffer

When using Disk buffer type, ensure the gateway pod has sufficient ephemeral storage:

• The vector-data emptyDir is sized via sizeLimit to match gateway.buffer.maxSize
• Each gateway replica uses up to maxSize (e.g. 512Mi) of node storage
• With multiple replicas, total disk usage = replicas × maxSize
• Consider adding node ephemeral storage limits or using nodes with sufficient local storage

Sending events in CEF format

Security events can be sent in Common Event Format (CEF) for integration with SIEM systems (Splunk, ArcSight, QRadar, etc.).

CEF encoding is configured per-destination in ClusterSecurityEventDestination. It is supported on Kafka, Vector, File, Console, and Socket destinations. Loki, Elasticsearch, and SplunkHEC always use JSON.

Basic CEF over Kafka

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: siem-kafka
spec:
  type: Kafka
  kafka:
    brokers:
      - "siem-kafka:9092"
    topic: "security-events"
    encoding:
      codec: CEF

Events will be sent as bare CEF strings. The gateway maps SecurityEvent.event.severity (Low/Medium/High/Critical) to CEF numeric severity (1/5/8/10), and uses event.code as the CEF signature ID and event.description as the CEF name.

CEF with syslog wrapping

For SIEM systems that expect syslog-framed CEF (common for UDP/TCP syslog collectors):

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: siem-syslog
spec:
  type: Kafka
  kafka:
    brokers:
      - "siem-kafka:9092"
    topic: "security-events"
    encoding:
      codec: CEF
      syslogWrapper: RFC5424

Supported syslog wrappers: RFC3164 (BSD syslog), RFC5424 (IETF syslog), or None (default, bare CEF).

Custom CEF metadata

Override the CEF device vendor/product/version per-destination:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: siem-custom
spec:
  type: Kafka
  kafka:
    brokers:
      - "siem-kafka:9092"
    topic: "security-events"
    encoding:
      codec: CEF
      cef:
        deviceVendor: "MyCompany"
        deviceProduct: "k8s-security"
        deviceVersion: "2.0"

Or set defaults globally in ClusterSecurityEventConfig:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventConfig
metadata:
  name: default
spec:
  defaultSeverityThreshold: Low
  destinations:
    - siem-kafka
  cef:
    deviceVendor: "MyCompany"
    deviceProduct: "k8s-security"
    deviceVersion: "2.0"

Per-destination encoding.cef overrides take precedence over CSEC spec.cef defaults.