The module lifecycle stage: General Availability

The module has requirements for installation

Enabling debugging logs

Falco

By default, the debug log level is used.

Falcosidekick

By default, the debug logging for Falcosidekick is disabled.

To enable debugging logging, set the spec.settings.debugLogging parameter to true:

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: runtime-audit-engine
spec:
  version: 1
  enabled: true
  settings:
    debugLogging: true

Viewing metrics

You can use the PromQL query falcosecurity_falcosidekick_falco_events_total{} to get metrics:

d8 k -n d8-monitoring exec -it prometheus-main-0 prometheus -- \
  curl -s "http://127.0.0.1:9090/api/v1/query?query=falcosecurity_falcosidekick_falco_events_total" | jq

The event severity is carried by two labels: priority holds a numeric value, and priority_raw holds the lowercase severity name. Use priority_raw to filter events by severity:

priority priority_raw
8 emergency
7 alert
6 critical
5 error
4 warning
3 notice
2 informational
1 debug

An event that arrives without a priority of its own is the exception to that table: it is exported with priority="0" and an empty priority_raw. Prometheus drops an empty label, so no priority_raw selector ever matches such an event — including an event delivered straight to the falcosidekick receiver with the priority field omitted. To count these events as well, select on priority or leave the severity filter out entirely.

The module also ships a Grafana dashboard named Runtime audit engine. It shows event throughput per node alongside the health of Falco, k8s-metacollector, falcosidekick and rules-loader.

Kubernetes metadata in event output

The output of every syscall rule ends with the namespace and the Pod name of the workload that triggered it:

ns=my-namespace pod=my-app-7d9f8b6c5-xk2vn

A syscall event identifies the workload only by container ID, so Falco resolves the namespace and the Pod name from that ID through the container runtime socket. The same two values are added to the output_fields property of the JSON output, as k8s.ns.name and k8s.pod.name. Falcosidekick reads those fields, so the falcosecurity_falcosidekick_falco_events_total metric also carries them in its k8s_ns_name and k8s_pod_name labels.

Rules of the k8sAudit type are not affected. They already report ns and pod, taken from the Kubernetes audit event itself.

Both fields can come out empty, and an empty label is dropped by Prometheus. This happens in two cases:

  • The event did not come from a container. Host-level activity has no Pod to name.
  • Falco has not yet resolved the container’s metadata. It is collected asynchronously from the runtime socket, so the earliest events of a short-lived container can be exported before it arrives.

Do not build a PromQL selector on k8s_ns_name or k8s_pod_name: it silently skips every series where the label is absent, including all k8sAudit events. Use the fields to read an event, not to select one.

If you need enrichment that does not depend on this timing, process events through the security-events-manager module. It enriches events in the event-building pipeline rather than inside Falco.

Emulating a Falco event

You can use the event-generator CLI utility to generate Falco events.

event-generator can generate a variety of suspicious actions (such as syscalls, Kubernetes audit events, etc.).

Use the following command to run a testing set of events in the Kubernetes cluster:

d8 k run falco-event-generator --image=falcosecurity/event-generator run

The event-generator image is included with the platform.
To determine its path, run the following command:

kubectl -n d8-system exec svc/deckhouse-leader -- bash -c "echo \$(deckhouse-controller module values runtime-audit-engine | yq '.registry.base')/runtime-audit-engine@\$(cat /deckhouse/downloaded/modules/900-runtime-audit-engine/images_digests.json | jq -r '.eventGenerator')"

For security reasons, this image is created without additional dependencies and is designed to extract the binary file and execute it on the host. To extract the binary, run the following commands:

kubectl run falco-event-generator --image=$(kubectl -n d8-system exec svc/deckhouse-leader -- bash -c "echo \$(deckhouse-controller module values runtime-audit-engine | yq '.registry.base')/runtime-audit-engine@\$(cat /deckhouse/downloaded/modules/900-runtime-audit-engine/images_digests.json | jq -r '.eventGenerator')") --command -- /event-generator run
kubectl wait --for=jsonpath='{.status.phase}'=Running pod/falco-event-generator
kubectl cp falco-event-generator:event-generator ./event-generator 
kubectl delete pod falco-event-generator

To correctly run all types of event tests, the following packages or their equivalents, appropriate for the specific operating system, must be installed on the host:

policykit-1 libcap-dev e2fsprogs openssh-client openssh-server nmap netcat-openbsd wget

If you need to implement an action, use this guide.

Emulating a Falcosidekick event

You can use the Falcosidekick /test HTTP endpoint to send a test event.

  • Create a debug event by executing a command:

    nsenter -t $(pidof falcosidekick) curl -X POST -H "Content-Type: application/json" -H "Accept: application/json" http://localhost:2801/test

  • Check a debug event metric:

    d8 k -n d8-monitoring exec -it prometheus-main-0 prometheus -- \
  curl -s "http://127.0.0.1:9090/api/v1/query?query=falcosecurity_falcosidekick_falco_events_total" \
  | jq '.data.result.[] | select (.metric.priority_raw == "debug")'

  • Example of the output part:

    {
  "metric": {
    "__name__": "falcosecurity_falcosidekick_falco_events_total",
    "container": "kube-rbac-proxy",
    "hostname": "falcosidekick",
    "instance": "192.168.208.7:4212",
    "job": "runtime-audit-engine",
    "node": "dev-master-0",
    "priority": "1",
    "priority_raw": "debug",
    "rule": "Test rule",
    "source": "internal",
    "tier": "cluster"
  },
  "value": [
    1744234729.799,
    "1"
  ]
}