The Kubernetes auditing feature allows you to track requests to the API server and analyze events occurring in the cluster. Auditing can be useful for troubleshooting unexpected behavior and for meeting security requirements.

Kubernetes supports audit configuration via the Audit policy mechanism, which allows you to define logging rules for target operations. By default, audit results are written to the /var/log/kube-audit/audit.log file.

Built-in audit policies

Deckhouse Virtualization Platform (DVP) configures a basic audit policy by default, which can be supplemented with custom rules.

This policy implements the following rules:

  • Do not log frequent updates for Endpoints, EndpointSlices, and Events.
  • Do not log leader election operations on Lease resources.
  • Do not log cert-manager leader election ConfigMaps.
  • Do not log VerticalPodAutoscalerCheckpoints resources.
  • Do not log PATCH operations on VerticalPodAutoscaler from recommender.
  • Do not log UpmeterHookProbes resources.
  • Do not log any operations in d8-upmeter namespace.
  • Do not log ingress-nginx leader election updates in ConfigMaps.
  • Do not log dex health-check create/delete operations on AuthRequest resources.
  • Log create and delete operations for Node resources with request/response payload.
  • Log kubectl logs requests (pods/log) at Metadata level.
  • Log create/update/patch/delete operations from system service accounts (kube-system, d8-*).
  • Log create/update/patch/delete operations for Pod resources.
  • Log create/update/patch/delete operations in system namespaces (kube-system, d8-*).
  • Log all LIST operations in all namespaces.
  • Log create and delete operations for ServiceAccount resources.
  • Log create/update/delete/patch operations for Role and ClusterRole resources.
  • Log create/update/delete operations for ClusterRoleBinding resources.
  • Log attach and ephemeral container related pod subresource operations.
  • Log creation of VirtualMachineOperation resources with request/response payload.
  • Log create/update/patch/delete operations for virtualization.deckhouse.io resources.
  • Log update/patch operations for internal virtualization subresources.
  • Log GET operations for subresources.virtualization.deckhouse.io API group.
  • Log create/update/patch/delete operations for Pod resources.
  • Log create/update/patch/delete operations in d8-virtualization namespace.
  • Log create/update/patch/delete operations for ModuleConfig resources.
  • Do not log requests from authenticated users.
  • Log all remaining (unauthenticated) requests at Metadata level.

Detailed information about the rules and an example of the final policy are available in the control-plane-manager module documentation

These rules from the basic policy are enabled by default. To disable them, set the basicAuditPolicyEnabled parameter to false.

Example:

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: control-plane-manager
spec:
  version: 1
  settings:
    apiserver:
      auditPolicyEnabled: true
      basicAuditPolicyEnabled: false

Configuring a custom audit policy

To create an advanced Kubernetes API audit policy, follow these steps:

  1. Enable the auditPolicyEnabled parameter in the control-plane-manager module configuration:

    apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: control-plane-manager
spec:
  version: 1
  settings:
    apiserver:
      auditPolicyEnabled: true

  2. Create the kube-system/audit-policy Secret containing the policy YAML file encoded in Base64:

    apiVersion: v1
kind: Secret
metadata:
  name: audit-policy
  namespace: kube-system
data:
  audit-policy.yaml: <Base64>

    Example audit-policy.yaml content with a minimal working configuration:

    apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
  omitStages:
  - RequestReceived

    For more information on possible contents of audit-policy.yaml, refer to the following sources:

Working with the audit log file

On DVP master nodes, it is assumed that a log collection tool (log-shipper, promtail, or filebeat) is installed to monitor the /var/log/kube-audit/audit.log file.

The log rotation settings for this file are predefined and cannot be changed:

  • Maximum file size: 1000 MB.
  • Maximum retention period: 30 days.

Depending on the policy configuration and the volume of requests to the API server, the number of log entries can be very large. In such cases, the retention period may be reduced to less than 30 minutes.

Unsupported options or typos in the configuration file may cause the API server to fail to start.

If the API server fails to start, take the following steps:

  1. Manually remove the --audit-log-* parameters from the /etc/kubernetes/manifests/kube-apiserver.yaml manifest.

  2. Restart the API server with the following command:

    docker stop $(docker ps | grep kube-apiserver- | awk '{print $1}')
   
# Alternative option (depending on the CRI in use).
crictl stopp $(crictl pods --name=kube-apiserver -q)

  3. After restarting, fix the Secret or delete it with the following command:

    d8 k -n kube-system delete secret audit-policy

Redirecting the audit log file to stdout

By default, the audit log is saved to the /var/log/kube-audit/audit.log file on master nodes. If necessary, you can redirect its output to the kube-apiserver process stdout instead of a file by setting the apiserver.auditLog.output parameter in the control-plane-manager module to Stdout:

apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
  name: control-plane-manager
spec:
  version: 1
  settings:
    apiserver:
      auditPolicyEnabled: true
      auditLog:
        output: Stdout

In this case, the log will be available in the kube-apiserver container stdout.

Then, using the built-in DVP logging mechanism, you can configure log collection and forwarding to your own security system.

Additional resources