The module lifecycle stage: Experimental

The module has requirements for installation

ClusterSecurityEventConfig with Loki (explicit allowlist)

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventConfig
metadata:
  name: default
spec:
  defaultSeverityThreshold: High
  enabledSources:
    - clusterSecurityEventShipper/kube-audit/kube-apiserver
  destinations:
    - cluster-loki

ClusterSecurityEventConfig with glob masks

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventConfig
metadata:
  name: default
spec:
  defaultSeverityThreshold: High
  enabledSourcesMasks:
    - clusterSecurityEventShipper/kube-audit/*
    - podSecurityEventShipper/*
  destinations:
    - cluster-loki

ClusterSecurityEventDestination (Loki)

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: cluster-loki
spec:
  type: Loki
  loki:
    endpoint: https://loki.d8-monitoring:3100
    auth:
      strategy: Bearer
      token: EXAMPLE
    tls:
      verifyCertificate: false
      verifyHostname: false

Credentials (token, password) can also be referenced from a Secret using tokenSecretRef/passwordSecretRef instead of inline values. The Secret must be in the d8-security-events-manager namespace with the key value in its data field and labeled security-events-manager.deckhouse.io/credential-secret: "true". Inline credential fields and their *SecretRef counterparts are mutually exclusive.

SecurityEventDefinition

apiVersion: security.deckhouse.io/v1alpha1
kind: SecurityEventDefinition
metadata:
  name: k8s-privilege-escalation
spec:
  code: K8S_PRIV_ESC
  category: Rbac
  severity: High
  description: "Attempt to create privileged pod or escalate permissions"
  source: kube-apiserver
  fields:
    - name: metadata.extra.privileges
      required: true

PodSecurityEventShipper (KubernetesPods with inline parser)

apiVersion: security.deckhouse.io/v1alpha1
kind: PodSecurityEventShipper
metadata:
  name: my-audit
  namespace: my-namespace
spec:
  - source: my-audit-app
    input:
      type: KubernetesPods
      kubernetesPods:
        labelSelector:
          matchLabels:
            app: audit
    parser:
      - name: app
        parser:
          type: Regex
          regex:
            patterns:
              - '^(?P<level>\w+)\s+(?P<msg>.+)$'
        fields:
          - name: level
            type: String
    produces:
      - eventCode: K8S_PRIV_ESC
        extract:
          field: message
          operator: Regex
          values:
            - '.*'
        transform:
          - key: event.severity
            value: level

ClusterSecurityEventShipper (File with parserRef)

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventShipper
metadata:
  name: kube-audit
spec:
  - source: kube-apiserver
    input:
      type: File
      files:
        - /var/log/kube-apiserver/audit.log
    parserRef: audit-json
    produces:
      - eventCode: K8S_AUDIT_FAIL

With a ClusterSecurityEventLoggingTransformationRules:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventLoggingTransformationRules
metadata:
  name: audit-json
spec:
  type: File
  file:
    paths:
      - /var/log/kube-apiserver/audit.log
    transform:
      parser:
        type: JSON
      fields:
        - name: stage
          type: String
        - name: responseStatus
          type: Int
      drop_raw: true

Multiple destinations (Loki + Splunk)

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: cluster-loki
spec:
  type: Loki
  loki:
    endpoint: https://loki.example:3100
---
apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: splunk-prod
spec:
  type: SplunkHEC
  splunkHEC:
    endpoint: https://splunk.example:8088
    token: YOUR_TOKEN
---
apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventConfig
metadata:
  name: default
spec:
  defaultSeverityThreshold: Medium
  enabledSourcesMasks:
    - "*"
  destinations:
    - cluster-loki
    - splunk-prod

SecurityEventLoggingTransformationRules (reusable parser)

apiVersion: security.deckhouse.io/v1alpha1
kind: SecurityEventLoggingTransformationRules
metadata:
  name: falco
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: runtime-audit-engine
  containers:
    - name: falco
      parser:
        type: JSON
      fields:
        - name: priority
          type: String
        - name: output
          type: String
      drop_raw: true

Enrichment via Plugin (k8s-pod-info)

A shipper can resolve a Pod field absent from the raw log by using enrich.source: Plugin. For the plugin reference (required input parameters, available output values, and the example enrich element template), see Enrichment plugins.

apiVersion: security.deckhouse.io/v1alpha1
kind: PodSecurityEventShipper
metadata:
  name: app-audit
  namespace: my-namespace
spec:
  - source: my-audit-app
    input:
      type: KubernetesPods
      kubernetesPods:
        labelSelector:
          matchLabels:
            app: audit
    parser:
      - name: app
        parser:
          type: JSON
    producesDefaults:
      # k8s.pod.name / k8s.namespace.name are populated by the KubernetesPods input.
      enrich:
        - target: actor.id
          source: Plugin
          plugin: k8s-pod-info
          value: serviceAccountName
          args:
            - key: pod_name
              value: k8s.pod.name
            - key: namespace
              value: k8s.namespace.name
    produces:
      - eventCode: K8S_PRIV_ESC
        extract:
          field: message
          operator: Regex
          values:
            - '.*'

Container-ID mode

When the raw event carries a container runtime ID instead of pod name/namespace (common in syscall/falco events), use the k8s-container-info plugin to resolve the serviceAccountName, name, or namespace in one lookup:

apiVersion: security.deckhouse.io/v1alpha1
kind: PodSecurityEventShipper
metadata:
  name: falco-audit
  namespace: kube-system
spec:
  - source: falco
    input:
      type: KubernetesPods
      kubernetesPods:
        labelSelector:
          matchLabels:
            app: falco
    parser:
      - name: falco
        parser:
          type: JSON
    producesDefaults:
      # container.id is populated by the parser from the falco JSON output.
      enrich:
        - target: actor.id
          source: Plugin
          plugin: k8s-container-info
          value: serviceAccountName
          args:
            - key: container_id
              value: container.id
    produces:
      - eventCode: K8S_PRIV_ESC
        extract:
          field: message
          operator: Regex
          values:
            - '.*'

Enrichment via Plugin (k8s-nodeuser-info)

A shipper can resolve a static-user username (nodeusers.deckhouse.io metadata.name) from the system UID carried in the event by using enrich.source: Plugin with the k8s-nodeuser-info plugin. For the plugin reference (required input parameters, available output values, and the example enrich element template), see Enrichment plugins.

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventShipper
metadata:
  name: falco-nodeuser-enrich
spec:
  - source: runtime-audit-engine
    input:
      type: KubernetesPods
      kubernetesPods:
        labelSelector:
          matchLabels:
            app: falco
    parser:
      - name: falco
        parser:
          type: JSON
    producesDefaults:
      # output_fields.user.uid is populated by the parser from the falco JSON output.
      enrich:
        - target: actor.name
          source: Plugin
          plugin: k8s-nodeuser-info
          value: username
          args:
            - key: uid
              value: output_fields.user.uid
    produces:
      - eventCode: K8S_SSH_LOGIN
        extract:
          field: message
          operator: Regex
          values:
            - '.*'

Enrichment via custom External plugin

Users can create External ClusterSecurityEventEnrichmentPlugin resources to register custom enrichment endpoints. Below is an example of an External plugin that resolves employee info by badge ID.

First, create the CSEP resource:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventEnrichmentPlugin
metadata:
  name: employee-info
spec:
  type: External
  description: "Resolve employee info by badge ID"
  endpoint:
    url: http://employee-enricher.hr-namespace.svc:8080/lookup
  args:
    - name: badge_id
      required: true
      description: "Event field path containing the employee badge ID"
  returns:
    fields:
      - name: username
        type: String
        description: "Employee username"
      - name: department
        type: String
        description: "Employee department"

Then, reference it in a shipper’s enrich rule:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventShipper
metadata:
  name: badge-enrich-example
spec:
  - source: runtime-audit-engine
    input:
      type: KubernetesPods
      kubernetesPods:
        labelSelector:
          matchLabels:
            app: falco
    parser:
      - name: falco
        parser:
          type: JSON
    producesDefaults:
      # output_fields.user.badge_id is populated by the parser.
      enrich:
        - target: actor.name
          source: Plugin
          plugin: employee-info
          value: username
          args:
            - key: badge_id
              value: output_fields.user.badge_id
    produces:
      - eventCode: K8S_BADGE_LOGIN
        extract:
          field: message
          operator: Regex
          values:
            - '.*'

Buffer configuration

Production setup: Disk + Block (default)

The default buffer settings are designed for production — Disk buffer with Block overflow behavior ensures zero data loss of security events during temporary destination outages.

Module config (ModuleConfig):

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: security-events-manager
spec:
  version: 1
  settings:
    gateway:
      buffer:
        type: Disk
        whenFull: Block
        maxSize: 512Mi
        maxEvents: 500
      logShipperBuffer:
        type: Disk
        whenFull: Block
        maxSize: 257Mi    # per node; must be > 256Mi, log-shipper's disk buffer minimum
        maxEvents: 500

Per-destination override in ClusterSecurityEventDestination (e.g., larger buffer for a remote Elasticsearch):

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: remote-elasticsearch
spec:
  type: Elasticsearch
  elasticsearch:
    endpoint: https://es.example.com:9200
    index: security-events
    auth:
      strategy: Bearer
      tokenSecretRef:
        name: es-token
  buffer:
    type: Disk
    whenFull: Block
    maxSize: 1Gi    # larger buffer for unreliable remote destination
    maxEvents: 500

Test setup: Memory + DropNewest

For test/dev environments where data loss is acceptable and you want maximum speed without backpressure:

Module config:

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: security-events-manager
spec:
  version: 1
  settings:
    gateway:
      buffer:
        type: Memory
        whenFull: DropNewest
        maxSize: 50Mi
        maxEvents: 500
      logShipperBuffer:
        type: Memory
        whenFull: DropNewest
        maxSize: 50Mi
        maxEvents: 500

Alerting on security events

An alert rule turns matching events into a Prometheus alert. The aggregator counts matching events and, when their number reaches the threshold, exports the security_events_alert_last_firing_timestamp_seconds metric. The D8SecurityEventAlertFiring rule shipped with the module keeps the alert firing while that timestamp is recent.

This is an ordinary Prometheus alert: it becomes a ClusterAlert object (to list these objects, use the d8 k get clusteralerts command), appears in the console alongside other cluster alerts, and is delivered to every channel the cluster already sends alerts to. You do not need to configure delivery.

Example of a rule configuration manifest:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventAlertRule
metadata:
  name: mass-secret-access
spec:
  match:
    eventCodes: [K8S_SECRET_ACCESSED]
    excludeSystemActors: true  # Enabled by default: the platform reads secrets continuously.
  aggregation:
    window: 5m                 # How long a group keeps counting.
    groupBy: [ActorID]        # A separate counter for each actor.
    threshold: 20              # The alert fires on the 20th event.
  alert:
    severityLevel: 6
    summary: "One actor read an unusual number of secrets"
    description: "Check who it was in the actor_id label and whether the access was expected"

Without the aggregation block, the rule fires an alert on every matching event — this suits events that are rare and serious on their own. Example:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventAlertRule
metadata:
  name: cluster-admin-bound
spec:
  match:
    eventCodes: [K8S_CLUSTER_ADMIN_BOUND]
  alert:
    severityLevel: 5
    summary: "A subject was bound to cluster-admin"

Examples of rule configurations for different events

Example of a rule that fires an alert when one actor runs exec into several pods:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventAlertRule
metadata:
  name: repeated-pod-exec
spec:
  match:
    eventCodes: [K8S_POD_EXEC_ATTACH]
  aggregation:
    window: 10m
    groupBy: [ActorID]
    threshold: 5
  alert:
    severityLevel: 6
    summary: "One actor opened shells in several pods"

Example of a rule that fires an alert on a burst of RBAC edits in one namespace (privilege escalation):

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventAlertRule
metadata:
  name: rbac-edit-burst
spec:
  match:
    eventCodes: [K8S_RBAC_RESOURCES_MODIFIED]
  aggregation:
    window: 5m
    groupBy: [ActorID, ObjectNamespace]   # Counted per actor–namespace pair.
    threshold: 5
  alert:
    severityLevel: 5
    summary: "Several RBAC objects changed in a short window"
    description: "Compare actor_id and object_namespace against what was planned"

Example of a rule that fires an alert on brute force (repeated rejections from one address). Group by the client address rather than the actor, because an unauthorized request has no identity:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventAlertRule
metadata:
  name: unauthorized-burst
spec:
  match:
    eventCodes: [K8S_UNAUTHORIZED_REQUEST]
  aggregation:
    window: 5m
    groupBy: [ActorSourceIP]
    threshold: 10
  alert:
    severityLevel: 5
    summary: "Repeated unauthorized requests from one address"

Example of a rule without aggregation that fires an alert for an event rare enough that a single occurrence deserves attention:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventAlertRule
metadata:
  name: containerd-integrity
spec:
  match:
    eventCodes: [D8_CONTAINERD_INTEGRITY_VIOLATION]
  alert:
    severityLevel: 4     # Deliberately low: this event should also stop platform updates.
    summary: "containerd integrity check failed on a node"

Example of a rule that fires an alert when RBAC changes are initiated by subjects other than a ServiceAccount:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventAlertRule
metadata:
  name: person-touched-rbac
spec:
  match:
    categories: [Rbac]           # Any RBAC event, regardless of its code.
    actorType: NonServiceAccounts
  alert:
    severityLevel: 5
    summary: "RBAC changed by a non-ServiceAccount identity"

NonServiceAccounts is not quite “a human”: a client certificate or an external identity is also not a ServiceAccount. To watch a specific identity, specify it explicitly:

  match:
    eventCodes: [K8S_SECRET_ACCESSED]
    actors: ["system:serviceaccount:my-app:*", "kubernetes-admin"]

Specifics of writing rules

When writing rules, consider the following:

  • The rule applies only to events recorded by the cluster audit policy. The default DKP policy logs list of secrets but not a get of one secret by name, so a rule on K8S_SECRET_ACCESSED fires on listing secrets, not on reading a single known secret. The policy also drops most requests from the system:authenticated group, so human activity reaches the event stream only where an explicit rule records it: mutations in platform namespaces, RBAC changes, exec/attach, and similar. To confirm that the rule is not working, check the policy: an event that was never recorded cannot become an alert.
  • Platform actors are filtered out by default. During routine cluster operation, secrets are read continuously: on an idle cluster, one DKP ServiceAccount produced about 2600 reads in an hour versus two from a person. excludeSystemActors hides the same set as the “Exclude system” filter in the platform web interface, so the interface and the rule treat platform noise the same way. Set it to false to also alert on the platform itself, and use excludeActors for everything else — for example, to silence a noisy application ServiceAccount.
  • The alert fires on the event that reaches the threshold, not at the end of the window. The window determines how long a group keeps counting: a group that never reaches the threshold is forgotten when the window expires.
  • groupBy values become alert labels (ActorID appears as actor_id), so alerts that concern different actors are not mixed. Only fields with bounded cardinality are accepted: an object name would create a group per event. After 500 distinct values for one key, the label is dropped, the alert continues to fire without that dimension, and the rule cannot overload Prometheus with series.
  • Counters live in memory. Restarting the aggregator starts the windows again, and a partially reached threshold is counted from zero.
  • severityLevel 4 or below blocks DKP updates on clusters where update.blockOnAlerts is enabled in the deckhouse module. For a critical event this can be a deliberate decision; for a routine one it is unexpected.
  • summary and description are ordinary text. Context reaches the alert through labels: the group keys, rule, and severity_level.
  • All rules fire an alert with the same name — D8SecurityEventAlertFiring. The object name goes into the rule label, which is what distinguishes the rules. Search by the rule, not by the alert name.
  • The alert disappears 5–7 minutes after the last firing. While the rule continues to fire, the alert stays active. Once the events stop, it disappears. The condition is the age of the last firing, so the alert cannot remain stuck even if the rule is deleted while it is active.

HTTP destination (webhook or HTTP collector)

Send security events with a POST request. Both framing modes exist because receivers interpret a batch of events differently: some read one JSON array per request, while most collectors read a stream of JSON records.

Example of sending security events from a DKP cluster to an external system over HTTP:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: siem-http
spec:
  type: Http
  http:
    endpoint: "https://collector.example.com/api/ingest"
    framing: NewlineDelimited      # One JSON object per line (NDJSON).
    auth:
      strategy: Bearer
      tokenSecretRef:
        name: siem-http-token
        key: token
    tls:
      verifyCertificate: true
    headers:
      X-Scope-OrgID: security
---
apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventConfig
metadata:
  name: default
spec:
  destinations:
    - siem-http
  enabledSourcesMasks:
    - "clusterSecurityEventShipper/*"
    - "podSecurityEventShipper/*"
  defaultSeverityThreshold: Low

Do not put an Authorization header in headers together with auth — the two conflict and the destination is rejected on admission.

Socket destination (CEF over syslog to SIEM)

Send security events in CEF format wrapped in RFC5424 syslog over TCP/TLS to a SIEM receiver:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: siem-syslog
spec:
  type: Socket
  socket:
    address: "siem.example.com:6514"
    mode: TCP
    tls:
      verifyCertificate: true
    encoding:
      codec: CEF
      syslogWrapper: RFC5424
      cef:
        deviceVendor: MyCompany
        deviceProduct: k8s-security
---
apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventConfig
metadata:
  name: default
spec:
  destinations:
    - siem-syslog
  enabledSourcesMasks:
    - "clusterSecurityEventShipper/*"
    - "podSecurityEventShipper/*"
  defaultSeverityThreshold: Low

Socket destination (raw JSON over UDP)

Send events as JSON over UDP (fire-and-forget, no TLS):

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: udp-collector
spec:
  type: Socket
  socket:
    address: "collector.example.com:514"
    mode: UDP

Socket destination (Unix socket for sidecar SIEM agent)

Send events to a local Unix domain socket (e.g. for a sidecar SIEM agent):

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: sidecar-siem
spec:
  type: Socket
  socket:
    address: "/var/run/siem.sock"
    mode: Unix

Per-destination override:

apiVersion: security.deckhouse.io/v1alpha1
kind: ClusterSecurityEventDestination
metadata:
  name: test-loki
spec:
  type: Loki
  loki:
    endpoint: http://loki:3100
  buffer:
    type: Memory
    whenFull: DropNewest
    maxSize: 50Mi
    maxEvents: 500