Audit

Audit

To diagnose API operations, for example, in case of unexpected behavior of control plane components, Kubernetes provides an API operation logging mode. You can configure this mode by creating Audit Policy rules, and the result of the audit will be the /var/log/kube-audit/audit.log log file with all the operations of interest. For more details, see the Auditing section of the Kubernetes documentation.

Basic audit policies

Deckhouse clusters have the following basic audit policies created by default:

• logging of resource creation, deletion, and modification operations;
• logging of actions performed on behalf of service accounts from the system namespaces: kube-system, d8-*;
• logging of actions performed on resources in the system namespaces: kube-system, d8-*.

Disabling basic policies

You can disable log collection based on the basic policies by setting the basicAuditPolicyEnabled flag to false.

Example of enabling audit in kube-apiserver without the basic Deckhouse policies:

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: control-plane-manager
spec:
  version: 1
  settings:
    apiserver:
      auditPolicyEnabled: true
      basicAuditPolicyEnabled: false

You can also use a patch:

d8 k patch mc control-plane-manager --type=strategic -p '{"settings":{"apiserver":{"auditPolicyEnabled":true, "basicAuditPolicyEnabled": false}}}'

Custom audit policies

The control-plane-manager module automates the kube-apiserver configuration for adding custom audit policies. For such additional policies to work, make sure that audit is enabled in the apiserver parameter section and create a secret with the audit policy:

1. Enable the auditPolicyEnabled parameter in the module settings:

   apiVersion: deckhouse.io/v1alpha1
   kind: ModuleConfig
   metadata:
     name: control-plane-manager
   spec:
     version: 1
     settings:
       apiserver:
         auditPolicyEnabled: true

   You can enable it by editing the resource or by using a patch:

   d8 k patch mc control-plane-manager --type=strategic -p '{"settings":{"apiserver":{"auditPolicyEnabled":true}}}'

2. Create the kube-system/audit-policy Secret with the Base64-encoded policy YAML file:

   apiVersion: v1
   kind: Secret
   metadata:
     name: audit-policy
     namespace: kube-system
   data:
     audit-policy.yaml: <BASE64_AUDIT_POLICY>

   As an example of audit-policy.yaml, here is a rule for logging all metadata changes:

   apiVersion: audit.k8s.io/v1
   kind: Policy
   rules:
   - level: Metadata
     omitStages:
     - RequestReceived

   Examples and information about audit policy rules are available in:

   • Official Kubernetes documentation.
   • Our article on Habr (in Russian).
   • Code of the generator script used in GCE.

In this case, to recover, manually remove the --audit-log-* parameters from the /etc/kubernetes/manifests/kube-apiserver.yaml manifest and restart apiserver with the following command:

crictl stopp $(crictl pods --name=kube-apiserver -q)

After the restart, you will have enough time to delete the erroneous secret:

d8 k -n kube-system delete secret audit-policy

How to work with the audit log?

It is assumed that master nodes have a log collector (for example, log-shipper, promtail, filebeat) that sends records from the file to centralized storage:

/var/log/kube-audit/audit.log

The log file rotation parameters are preset and cannot be changed:

• Maximum disk space used: 1000 MB.
• Maximum retention depth: 30 days.

Keep in mind that “maximum retention depth” does not mean “guaranteed”. The log write rate depends on the additional policy settings and the number of requests to apiserver, so the actual retention depth may be much less than 7 days, for example, 30 minutes. Take this into account when configuring the log collector and writing audit policies.

Sending the audit log to standard output

If a pod log collector is configured in the cluster, you can collect the audit log by sending it to standard output. To do this, set the apiserver.auditLog.output parameter to Stdout in the module settings.

Example of enabling audit with output to stdout:

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: control-plane-manager
spec:
  version: 1
  settings:
    apiserver:
      auditPolicyEnabled: true
      auditLog:
        output: Stdout

You can also use a patch:

d8 k patch mc control-plane-manager --type=strategic -p '{"settings":{"apiserver":{"auditPolicyEnabled":true, "auditLog":{"output":"Stdout"}}}}'

After kube-apiserver restarts, you can see audit events in its log:

d8 k -n kube-system logs $(d8 k -n kube-system get po -l component=kube-apiserver -oname | head -n1)

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"38a26239-7f3e-402f-8c56-2fb57a3fe49d","stage":"ResponseComplete","requestURI": ...