Module ansible: user guide

This guide explains how to create, configure and troubleshoot runs of the ansible module. A run executes an Ansible playbook against Deckhouse Platform (DP) virtual machines (VMs) or against hosts specified by address.

Module resources

The module adds two custom resources to the cluster, which are listed in the table below.

This guide uses the following terms:

• Run: An AnsibleRun object.
• Playbook: The YAML file with tasks that the run executes.
• Runner Pod: A temporary Pod where DP executes the ansible-playbook utility.

The Secret with credentials, the ConfigMap with the playbook text and the target VirtualMachine resources have to be in the namespace of the run. The only object a run can use from outside its namespace is a cluster-wide ClusterNetwork.

Quick start

The steps below create a first run. It checks connectivity to a virtual machine (VM) and records the result in its status.

1. Prepare the VM. To connect, it needs an SSH server, a login user and an address DP knows about. The following cloud-init fragment installs the SSH server and creates such a user:

   #cloud-config
   packages:
     - openssh-server
   users:
     - name: ansible
       lock_passwd: false
       sudo: ALL=(ALL) NOPASSWD:ALL
       ssh_authorized_keys:
         - ssh-ed25519 <SSH_PUBLIC_KEY> ansible@example
   runcmd:
     - systemctl enable --now ssh

   Here <SSH_PUBLIC_KEY> is the public key of the pair whose private key you put into the Secret.

2. Wait until the VM reaches the Running phase and receives an address:

   d8 k get vm demo-vm-01 -n demo \
     -o jsonpath='{.status.phase}{"  "}{.status.ipAddress}{"\n"}'

   Here demo-vm-01 is the name of your VM, and demo is its namespace.

3. Create a Secret with SSH credentials in the namespace of the VM:

   d8 k create secret generic ssh-creds -n demo \
     --from-literal=username=ansible \
     --from-file=ssh-privatekey=./id_ed25519

   Here ./id_ed25519 is the path to the private key file of the same pair as <SSH_PUBLIC_KEY>.

4. Create a run that checks connectivity to the VM:

   apiVersion: ansible.deckhouse.io/v1alpha1
   kind: AnsibleRun
   metadata:
     name: configure-vm
     namespace: demo
   spec:
     target:
       type: VirtualMachines
       virtualMachines:
         names: [demo-vm-01]
     connection:
       secretRef:
         name: ssh-creds
     playbook:
       type: Inline
       inline: |
         ---
         - name: Configure VM
           hosts: all
           tasks:
             - name: Check SSH connectivity
               ansible.builtin.ping:

   The names field holds the name of the VM from the previous steps. Selecting VMs by labels is described in Virtual machines.

5. Check the result of the run:

   d8 k get ansibleruns -n demo

   Example output:

   NAME           PHASE               REASON              HOSTS   OK   FAILED   UNREACHABLE   SKIPPED   AGE
   configure-vm   PlaybookSucceeded   PlaybookSucceeded   1       1    0        0             0         42s
   

   The detailed result, including per-host task counters, is recorded in the status of the run:

   d8 k get ansiblerun configure-vm -n demo -o yaml

Run structure

A run manifest is made of four blocks, which are listed in the table below. Each block is described in a section of its own.

The manifest below has every block of a run filled in:

apiVersion: ansible.deckhouse.io/v1alpha1
kind: AnsibleRun
metadata:
  name: configure-db
  namespace: demo
spec:
  target:
    type: VirtualMachines            # VirtualMachines | Hosts
    virtualMachines:
      selector:                      # Either selector or names.
        matchLabels:
          role: db
  connection:
    secretRef:
      name: ssh-creds                # SSH credentials from the namespace of the run.
    network:                         # Optional. The main network by default.
      type: ClusterNetwork           # Main | Network | ClusterNetwork
      name: vlan-64
  playbook:
    type: Inline                     # Inline | ConfigMap | Git
    inline: |
      ---
      - hosts: all
        tasks:
          - ansible.builtin.ping:
    tags: [deploy]                   # Optional. Execute only the tasks with these tags.
    skipTags: [migrations]           # Optional. Leave out the tasks with these tags.
    vars:                            # Optional. Passed as --extra-vars.
      - name: app_version
        value: "1.4.2"               # Any YAML value: string, number, list, mapping.
      - name: db_password
        valueFrom:
          secretKeyRef:
            name: app-secrets
            key: db-password
    varsFiles:                       # Optional. YAML documents with variables.
      - configMapRef:
          name: app-config           # The key defaults to vars.yaml.
  runner:
    dryRun: false                    # Ansible check mode.
    diff: false
    verbosity: 0                     # From 0 to 4.

Targets

The target block defines the hosts the playbook is executed on, that is, DP virtual machines (VMs) or hosts specified by address. One run works with hosts of a single kind only, so the virtualMachines and hosts fields cannot be set in the same object.

This block replaces the -i parameter of the Ansible command line. From it, DP builds the inventory file that you would otherwise have to write and keep up to date by hand. Host addresses are resolved when the run starts, so a VM recreated with another address is configured without any change to the run manifest.

Virtual machines

VMs are selected by labels or named explicitly.

The example below selects VMs by label:

target:
  type: VirtualMachines
  virtualMachines:
    selector:
      matchLabels:
        role: demo

An example of selecting VMs by a label expression:

target:
  type: VirtualMachines
  virtualMachines:
    selector:
      matchExpressions:
        - key: role
          operator: In
          values: [web, db]

An example of selecting VMs by name:

target:
  type: VirtualMachines
  virtualMachines:
    names: [demo-vm-01, demo-vm-02]

The selector and names fields are mutually exclusive. The API server rejects a manifest with an empty selector. Such a selector would match every VM in the namespace, and the action of a run cannot be undone.

If a VM matches the selection but cannot be configured when the run starts, it is listed in status.skippedHosts with a reason, and the playbook is executed on the remaining VMs. The skip reasons are listed in Troubleshooting.

Hosts specified by address

To configure a host whose address DP does not know, use a target of type Hosts and specify the address in the run manifest. DP does not know addresses configured by hand inside the guest OS or assigned by an external DHCP server, nor the addresses of physical servers, network appliances and hosts from other clusters.

An example of a run with hosts specified by address:

target:
  type: Hosts
  hosts:
    - address: 192.168.55.10
      vars:
        - name: app_role
          value: frontend
      groups: [web]
    - address: db.example.com

A host specified by address has no VirtualMachine resource, so the run treats it differently from a DP VM. The differences are listed in the table below.

DP does not check whether another run uses a host specified by address, so two runs can work on the same address at the same time. Do not start runs that configure the same host concurrently.

The vars field of such a host is declared like host variables, valueFrom sources included. Group names follow the rules described in Ansible inventory.

Ansible inventory

DP builds the inventory file from the selected targets and from the annotations of the VMs. DP sets the ansible_host and ansible_user variables and the credential paths itself, and you cannot override them, while every other variable is set in VM annotations and in the run manifest.

For a VM, host groups and variables come from its annotations, which are listed in the table below.

An example of a VM with such annotations:

apiVersion: virtualization.deckhouse.io/v1alpha2
kind: VirtualMachine
metadata:
  name: demo-vm-01
  namespace: demo
  annotations:
    ansible.deckhouse.io/groups: "web,production"
    vars.ansible.deckhouse.io/app_role: "frontend"

The annotations of a VM play the same role as the host_vars/<HOST_NAME> file in an ordinary Ansible project. Unlike that file, they are stored on the VM itself. For a host specified by address, the vars and groups fields in the run manifest set the same data.

In both cases DP builds an inventory file of the following form:

all:
  hosts:
    demo-vm-01:
      ansible_host: 10.66.10.2
      ansible_user: ansible
      ansible_ssh_private_key_file: /home/runner/.ssh/ssh-privatekey
      app_role: frontend
web:
  hosts:
    demo-vm-01: {}
production:
  hosts:
    demo-vm-01: {}

Variable and group names in annotations follow the same rules as in the run manifest (they are described in Refused variable names). In addition, an annotation may not set variables with the ansible_ prefix, except ansible_port. The reasons for this restriction are described in Permissions and security.

An SSH port belongs to a particular VM, so set a non-standard port with an annotation on that VM:

metadata:
  annotations:
    vars.ansible.deckhouse.io/ansible_port: "2222"

The port of a host specified by address is set in the hosts[].vars field. The spec.playbook.vars field sets the port for every host of the run at once and overrides both of these values, because it has a higher precedence.

Do not set ansible_port in the spec.playbook.vars field if the VMs of the run use different SSH ports.

DP ignores annotations whose names do not start with the ansible.deckhouse.io/groups or vars.ansible.deckhouse.io/ prefix and skips empty group entries. If an annotation breaks a naming rule, the run ends in the Error phase with the InvalidInventoryAnnotations reason before DP creates the runner Pod. The status message names the VM and the annotation that broke the rule.

Connection

The connection block defines over which transport, with which credentials and through which network a run connects to its hosts.

Transport

The transport is set in the connection.type field. SSH is currently the only available value and is used by default, so the field can be omitted. An example with the transport set explicitly:

connection:
  type: SSH
  secretRef:
    name: ssh-creds

SSH credentials

A run takes the credentials for the connection from the Secret whose name is set in the connection.secretRef field:

connection:
  secretRef:
    name: ssh-creds

The Secret has to be in the namespace of the run and contain the keys listed in the table below. At least one of the ssh-privatekey and password keys is required.

The key names follow the built-in kubernetes.io/ssh-auth and kubernetes.io/basic-auth Secret types.

The command below creates a Secret that holds a password instead of a private key:

d8 k create secret generic ssh-creds-password -n demo \
  --from-literal=username=ansible \
  --from-literal=password=<PASSWORD>

Here <PASSWORD> is the password of the SSH user.

Networks

By default a run is executed in the main network. The runner Pod works from the cluster pod network. It connects to a virtual machine (VM) at the address from the status.ipAddress field of the VirtualMachine resource.

A VM can also be attached to additional networks of the sdn module, a project one (Network) or a cluster-wide one (ClusterNetwork). Each such network forms a separate L2 domain, and the pod network has no route into it. So if the SSH server of the VM is reachable only in an additional network, the address of the VM alone is not enough, and the run has to be executed in the same network.

Requirements for an additional network

A run in an additional network works under the following conditions:

• The sdn module is enabled in the cluster. Otherwise the run fails with the DependencyMissing reason.
• The network has an address pool (spec.ipam.ipAddressPoolRef), because both the VM and the runner Pod take their address from the pool. A run in a network without a pool ends in the Error phase with the NetworkWithoutIPAM reason before DP creates the runner Pod. The status message names the network.
• A DHCP client runs on the interface of this network in the guest OS. Without it, DP reports an address the VM does not answer at, and the host is unreachable.
• The interface configuration in the guest OS is bound to a MAC address or a predictable name rather than to ethX. Attaching a network or reordering spec.networks can renumber the ethX interfaces, and the DHCP client then requests an address on the wrong interface.
• The MTU of the network does not exceed the MTU of the node interfaces the network is built on. An MTU mismatch is hard to spot, because the SSH connection succeeds, but copying a file or installing a package hangs.

Configuring the network of a run

The network is named in the connection.network block. DP attaches the runner Pod to that network and takes the address of every target in it as well:

connection:
  secretRef:
    name: ssh-creds
  network:
    type: ClusterNetwork   # Main (default) | Network | ClusterNetwork
    name: vlan-64

The network block mirrors an entry of the spec.networks list of the VM, so take its values from the manifest of the VM:

# VirtualMachine
spec:
  networks:
    - type: Main
    - type: ClusterNetwork
      name: vlan-64        # Name this network in the run.
    - type: Network
      name: storage-net

A Network is visible only in its own namespace, while a ClusterNetwork is available from any. A run works in a single network only, so use labels to split VMs from different networks into several runs.

Set the network block for hosts specified by address as well, because a host address does not tell DP which network to attach the runner Pod to. This way you can configure hardware outside the cluster, such as physical servers, network appliances or hosts of other clusters, in a management network.

Runner Pod behaviour in an additional network

The additional interface does not replace the main one, so the runner Pod stays attached to the cluster pod network as well. The init containers still reach the cluster DNS and the cluster services, so a playbook from a Git repository inside the cluster and Galaxy content from a cluster mirror are fetched the same way as without an additional network.

An address pool that publishes routes is an exception. These routes apply inside the Pod, and a default route (0.0.0.0/0) sends all outbound traffic through the additional network. Requests to addresses inside the cluster keep working, because they have their own routes. But downloads from outside the cluster, for example from a public Git repository or from Galaxy, go through the additional network and may become unreachable. Before you start a run in such a network, check the list of routes in the spec.pools[].routes field of the pool.

Size of the address pool

The pool has to cover both the VMs and the concurrent runs, because a run holds an address while its runner Pod exists. DP deletes the Pod of a successful run at once but keeps the Pod of a failed run until the AnsibleRun object is deleted. Delete failed runs once you investigate them, otherwise they keep holding addresses.

Playbook

The playbook block defines where the playbook comes from and which part of it to execute. The playbook can be set as text in the run manifest or taken from a ConfigMap resource or a Git repository. A single run uses one source only.

Inline playbook

The playbook text is set directly in the manifest of the run:

playbook:
  type: Inline
  inline: |
    ---
    - name: Configure VM
      hosts: all
      tasks:
        - ansible.builtin.ping:

The size of an inline playbook is limited to 65,536 characters. Place a larger playbook in a ConfigMap resource or in a Git repository.

Playbook from a ConfigMap

To reuse a playbook in several runs or to get around the size limit of an inline playbook, store it in a ConfigMap in the namespace of the run. The run points at the key of that resource:

playbook:
  type: ConfigMap
  configMapRef:
    name: demo-playbook
    key: playbook.yaml     # Default value.

Playbook from a Git repository

The Git source fetches the whole repository tree, so a project with roles, a group_vars directory and templates is executed unchanged. An example of a run with a playbook from a Git repository:

playbook:
  type: Git
  git:
    url: https://github.com/example-org/infra
    revision: v1.4.0             # Branch, tag or SHA. The default branch if omitted.
    path: playbooks/site.yml     # Defaults to playbook.yaml.
    secretRef:
      name: git-token            # For private repositories.

These settings correspond to the following commands:

git clone --recurse-submodules https://github.com/example-org/infra && cd infra
git checkout v1.4.0
ansible-galaxy install -r requirements.yml     # If the project declares dependencies.
ansible-playbook -i inventory.yml playbooks/site.yml

DP fetches the repository together with its submodules. If the repository contains a requirements.yml, roles/requirements.yml or collections/requirements.yml file, the Galaxy roles and collections declared in it are installed before the playbook starts. The commit that was actually executed is recorded in the status.playbookCommit field.

The repository and the Galaxy content are fetched through the proxy configured for the cluster, and the module has no proxy settings of its own. These settings do not apply to the tasks of the playbook, because Ansible executes the tasks on the target hosts over SSH. If a task needs a proxy, declare it in the environment parameter of that task.

Private repositories

The credentials of a private repository are stored in a separate Secret whose name is set in the git.secretRef field. They are unrelated to the SSH credentials for the target hosts. DP passes them to the checkout step only, and the tasks of the playbook never see them.

For URLs of the form ssh://git@… the Secret is required and has to contain the ssh-privatekey and known_hosts keys. The known_hosts key is needed because SSH checks the key of the Git server strictly, and its contents can be obtained with the ssh-keyscan <GIT_SERVER> command, for example ssh-keyscan github.com. An example of such a Secret:

apiVersion: v1
kind: Secret
metadata:
  name: git-deploy-key
  namespace: demo
type: kubernetes.io/ssh-auth
stringData:
  ssh-privatekey: |
    -----BEGIN OPENSSH PRIVATE KEY-----
    ...
  known_hosts: |
    github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5...

For HTTP(S) URLs the Secret has to contain the username and password keys, where the access token serves as the password. For GitLab, set the username to oauth2. If the Git server uses a certificate of its own certificate authority, add the ca.crt key to the Secret as well. An example of a Secret with a token:

apiVersion: v1
kind: Secret
metadata:
  name: git-token
  namespace: demo
type: kubernetes.io/basic-auth
stringData:
  username: oauth2
  password: glpat-xxxxxxxxxxxx

Executing part of a playbook by tag

To execute only part of the playbook, set tags in the tags and skipTags fields. They work the same way as the --tags and --skip-tags parameters of the Ansible command line. An example of a run that executes only part of the playbook:

playbook:
  type: Git
  git:
    url: https://github.com/example-org/infra
    path: site.yml
  tags: [deploy, config]     # Execute only the tasks with these tags.
  skipTags: [migrations]     # Leave out the tasks with these tags.

The settings above correspond to the following command:

ansible-playbook -i inventory.yml site.yml --tags deploy,config --skip-tags migrations

If both fields are set, the run executes the tasks selected by tags minus those excluded by skipTags, the same way as the ansible-playbook utility.

A tag name can be any string without whitespace or commas. The comma separates tags in the command-line argument, and Ansible strips whitespace itself. Other notations used in playbooks are allowed, for example role:install and db/migrate.

The tags reserved by Ansible keep their standard meaning, which is listed in the table below.

Filter with no matches

A tag filter that matches no task is not an error. The ansible-playbook utility exits successfully without executing a single task, and the run ends in the PlaybookSucceeded phase with zero counters. A typo in a tag name produces the same result, so the run reports this case in its status:

$ d8 k get ansiblerun configure-db -n demo -o jsonpath='{.status.message}'
No task matched the tags of the run.

Runner options

To check a playbook without changes on the hosts or to see detailed output, use the runner block. It changes how the playbook is executed and how much it prints, but does not affect which tasks are executed:

runner:
  dryRun: true      # Ansible check mode.
  diff: true
  verbosity: 2      # From 0 to 4.

The settings above correspond to the following command:

ansible-playbook -i inventory.yml site.yml --check --diff -vv

With the dryRun parameter the run reports which changes would be made, and with the diff parameter it also prints the changes themselves. In check mode Ansible skips the tasks of the command and shell modules.

If the connection Secret contains an ansible-vault-password key, DP passes the --vault-password-file parameter to the ansible-playbook utility, so a project with files encrypted by Ansible Vault is executed unchanged.

Variables

Variables set the values a playbook is executed with, so the playbook itself does not have to change. Thanks to that, one Git project serves several environments, and a task receives a password that is not stored in the run manifest.

In the simplest case a variable holds a value the playbook reads as {{ app_version }}:

spec:
  playbook:
    vars:
      - name: app_version
        value: "1.4.2"

The notation matches the env block of a Pod. The value of a variable is either set directly in the value field or taken from another object through the valueFrom field. The value field accepts a string, a number, a list or a mapping.

A value can be supplied in three ways, which are listed in the table below.

The manifest below uses all three ways:

spec:
  playbook:
    vars:
      - name: app_version
        value: "1.4.2"
      - name: packages
        value: [nginx, curl, git]
      - name: users
        value:
          - name: deploy
            groups: [wheel]
      - name: db_password
        valueFrom:
          secretKeyRef:
            name: app-secrets
            key: db-password
    varsFiles:
      - configMapRef:
          name: app-config

The settings above correspond to the following command:

ansible-playbook -i inventory.yml site.yml \
  -e @app-config-vars.yml \
  -e '{"app_version":"1.4.2","packages":["nginx","curl","git"],"users":[{"name":"deploy","groups":["wheel"]}]}' \
  -e db_password="$DB_PASSWORD"

The spec.playbook.vars field corresponds to the Extra Variables field of AWX and to the -e parameter of the command line and has the same highest precedence. The full order of precedence is given in Precedence of values.

A variable name has to be an Ansible identifier. The naming rules and the list of refused names are given in Refused variable names.

Values from a Secret or a ConfigMap

The valueFrom field takes the value of a variable from one key of a Secret or a ConfigMap, so you do not have to copy a password or another stored value into the run manifest. The playbook sees the variable under the name set in the run:

spec:
  playbook:
    vars:
      - name: db_password
        valueFrom:
          secretKeyRef:
            name: app-secrets
            key: db-password

The kubelet substitutes the value when the runner Pod starts. DP reads the Secret and the ConfigMap only to check the key names, so the value reaches no object of the module and does not appear in the process arguments that ps shows. The variables file holds only a reference to the environment variable:

db_password: "{{ lookup('env', 'ANSIBLE_VAR_db_password') }}"

This way of passing values has the following properties:

• Permissions on the Secret: Kubernetes does not check them when the Pod starts, so the right to create runs lets a user read any Secret or ConfigMap in the namespace. Details are in Permissions and security.
• Name of the environment variable: The ANSIBLE_VAR_ prefix followed by the variable name, case preserved. Ansible treats db_password and DB_PASSWORD as different variables, and the environment of the Pod holds different entries for them as well.
• One name, one value: If one variable takes its values from two different keys (for example, two hosts name different Secrets), the run fails with the InvalidVariables reason, because the runner Pod has a single environment.
• A key holding binary data (binaryData of a ConfigMap): It never becomes a variable. If such a key is named explicitly, the run fails with a message that says so.

The playbook always receives a value from a single key as a string, even if the key holds a list. Pass a set of values with structure as a file. How to parse the string in the playbook is described in Type of a value from a single key.

Set of variables in a file

A set of variables can be described in a file, the same way as in the group_vars/all.yml file of an ordinary Ansible project. Place such a file in a ConfigMap or a Secret resource and name it in the run. This is equivalent to the -e @vars.yml parameter of the command line. An example of a ConfigMap with a set of variables:

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  vars.yaml: |                  # An ordinary YAML document.
    app_tier: backend
    app_version: "1.10"         # Quoted, so it stays a string.
    packages: [nginx, curl]     # A list stays a list.
    limits:
      cpu: 2
      mem: 4Gi

An example of a run that uses two variables files:

spec:
  playbook:
    varsFiles:
      - configMapRef:
          name: app-config        # The key defaults to vars.yaml.
      - secretRef:
          name: app-secrets
          key: prod.yaml
          optional: true

If the key field is not set, DP uses the vars.yaml key. Set the key field if a single object contains several sets of variables, such as dev.yaml and prod.yaml.

The main advantage of a file is that it keeps the types of values. The author of the file states what is a string and what is a number, while a single key of a Secret or a ConfigMap cannot carry this information. For example, the value 1.10 in a single key stays text, and turning it into a number would change a version into 1.1.

The kubelet mounts the key into the runner Pod, and the ansible-playbook utility reads the file, not DP. The values from the file therefore show up neither in the manifest of the run nor in the process list on the node. For the same reason DP does not check the variable names inside the file. Why this matters is described in Permissions and security.

Host variables

Host variables describe the host itself rather than the run, so they are set next to the host they belong to.

The hosts[].vars field is declared the same way as a variable of a run, valueFrom sources included. An example:

target:
  type: Hosts
  hosts:
    - address: 192.168.55.10
      vars:
        - name: app_role
          value: frontend
        - name: ansible_port
          value: 2222
      groups: [web]

Such variables land in the inventory file next to the host they describe and are meant for information about it, for example datacenter, app_role or a non-standard SSH port. For a virtual machine (VM), annotations set the same values.

A variable of the same name in the spec.playbook.vars field takes precedence over both the annotation and the hosts[].vars field, because DP passes this field as --extra-vars.

The exception is the case where both values come through valueFrom from different keys of a Secret or a ConfigMap. There is a single environment per runner Pod, so precedence does not apply in this case, and the run fails with the InvalidVariables reason before the playbook starts. To avoid that, give such variables different names. This restriction does not affect two values from the value field, or a value from value together with a value from valueFrom.

Missing source of variables

If a source of variables is missing, the behaviour of the run depends on what is missing (the object or a key in it) and on whether the optional parameter is set. The possible cases are listed in the table below.

For a missing key with optional: true, DP does not create the variable, so the when: x is defined construct and the default() filter work in the playbook.

Only variable sources have the optional field. The connection.secretRef and playbook.configMapRef fields do not have it, because a run without credentials or without a playbook cannot be executed.

Precedence of values

The variables of a run take precedence over every value a playbook or its project sets. A host variable comes from an annotation of a VM or from the hosts[].vars field. It takes precedence over the group_vars/ directory of the project but loses to a vars block inside a play.

Every value from the vars and varsFiles fields is passed as --extra-vars, that is level 22 of the Ansible precedence list, the highest one. The order of precedence, from the lowest to the highest, is listed in the table below.

A play in the table is a - hosts: … tasks: … block, and a playbook may hold several of them.

The spec.playbook.vars and spec.playbook.varsFiles fields sit on the same level of precedence, and the order of the arguments decides between them. The files are passed first, in list order, and the spec.playbook.vars field is passed last. The ansible-playbook utility applies -e parameters from left to right, and the entry further right wins. The resulting order is visible in the arguments of the runner Pod:

d8 k -n <NAMESPACE> get pod -l ansible.deckhouse.io/owner=<RUN_NAME> \
  -o jsonpath='{.spec.containers[0].args}'
# … -e @/ansible/vars.d/0/vars.yaml -e @/ansible/vars.d/1/prod.yaml -e @/ansible/vars.yml

Here <NAMESPACE> is the namespace of the run, and <RUN_NAME> is the name of the run.

A value with a higher precedence replaces the variable as a whole, and mappings do not merge. For example, if one file sets limits: {cpu: 2, mem: 4Gi} and the next one sets limits: {cpu: 4}, the playbook receives limits: {cpu: 4}, and the mem value is lost. Keep a whole mapping in one file, and override individual variables with the spec.playbook.vars field, whose value no file overrides.

It follows that a shared Git project cannot protect its variables from being overridden by a run. Keep values that relate to the project in its group_vars/ directory, and use the vars field for values that differ between runs.

Resolving a name conflict

In the example below, the playbook sets the app_version variable, the VM has an annotation with the same name, and the run passes its own value:

# VirtualMachine
metadata:
  annotations:
    vars.ansible.deckhouse.io/app_version: "from-annotation"   # Level 8.
---
# AnsibleRun
spec:
  playbook:
    type: Inline
    inline: |
      - hosts: all
        vars:
          app_version: "from-the-playbook"                       # Level 12.
        tasks:
          - ansible.builtin.debug:
              msg: "{{ app_version }}"
    vars:
      - name: app_version
        value: "from-the-run"                                    # Level 22.

The task prints the value from-the-run. Without the spec.playbook.vars field, the task prints the value from-the-playbook rather than the value of the annotation, because level 12 is higher than level 8. The value of an annotation applies only when the playbook does not set that variable.

Types of values

The way a value is written in the manifest determines its type. The resource schema checks variable names but not their values, so a typo inside a nested structure shows up only when the playbook is executed.

Keep in mind the YAML pitfall where the value 1.10 is read as the number 1.1 and the value no as the boolean false. Quote versions so that they stay strings:

vars:
  - name: app_version
    value: "1.10"      # Unquoted, it becomes 1.1.

Integers keep their type, so the value value: 1000000 reaches the playbook as the integer 1,000,000 without losing precision.

Type of a value from a single key

A key of a Secret or a ConfigMap holds a string, and the playbook receives exactly that string. The value field and a document from the varsFiles field keep the type of a value, while a single key does not. An example:

vars:
  - name: packages
    value: [nginx, curl]          # The playbook receives a list.
  - name: extra_packages
    valueFrom:
      configMapKeyRef:            # The playbook receives the string "[nginx, curl]".
        name: app-config
        key: extra_packages

If the playbook needs a list, parse the string in the playbook itself:

- hosts: all
  tasks:
    - ansible.builtin.package:
        name: "{{ extra_packages | from_yaml }}"
        state: present

This is why a task fails with an error such as “expected a list, got a string”. Pass a set of values with structure as a document in the varsFiles field. The document already sets the types of the values, so no string has to be parsed.

Value with curly braces

Whether a Jinja expression inside a value is evaluated depends on the source of that value. The possible cases are listed in the table below.

The author of the run writes the value field, so Ansible templates its value the same way as a variable in any variables file. A value from the valueFrom field arrives through the lookup('env', ...) expression, and Ansible marks the result of such an expression as unsafe. Because of that, the contents of an object the run does not own cannot become an expression executed inside the runner Pod.

To keep braces literal in the value field, escape them the way Ansible expects:

vars:
  - name: template_source
    value: "{{ '{{' }} not a template {{ '}}' }}"   # The playbook receives {{ not a template }}.

Refused variable names

A variable name has to be an Ansible identifier that matches the [A-Za-z_][A-Za-z0-9_]* pattern. For example, Ansible treats the {{ app-role }} expression not as a variable reference but as a subtraction. The API server rejects a name that does not match the pattern, and the d8 k apply command returns an error with the field name right away:

The AnsibleRun "pattern-check" is invalid:
* spec.target.hosts[0].vars[0].name: Invalid value: "app-role": spec.target.hosts[0].vars[0].name in body should match '^[A-Za-z_][A-Za-z0-9_]*$'

Group names follow the same rule and are rejected the same way.

In addition, two groups of names are refused.

The first group includes the names whose values DP sets itself, for example ansible_host with the address of a VM. Such a name cannot be used in any field where a variable is declared, because the API has a separate field for each of them. Two sources of one value lead to authentication errors that are hard to diagnose. The table below maps the names to the fields.

The ansible_connection variable with the local value would execute the playbook inside the runner Pod, and the run would report success, although not a single task was executed on the target VMs.

Every other variable with the ansible_ prefix is allowed in the run manifest, for example ansible_port, ansible_python_interpreter, ansible_become_user, ansible_become_method, ansible_shell_type and ansible_ssh_common_args.

The second group includes Ansible special variables (magic variables), for example groups, hostvars, inventory_hostname, playbook_dir, role_path and omit. They cannot be used in any field of a run. Ansible fills the values of such variables itself, so setting one either has no effect or breaks the evaluation of expressions. The omit variable is especially dangerous, because its value means “leave this parameter out”. A variable with that name would silently change what every task using omit does.

The table below shows where and how names are checked.

If a name belongs to one of the refused groups, the run ends before DP creates the runner Pod. The status message names the field and the reason:

status:
  phase: Error
  message: 'The variable "omit" in spec.playbook.vars[0] cannot be used: it shadows an Ansible magic variable, which Ansible fills in itself.'
  conditions:
    - type: Completed
      status: "False"
      reason: InvalidVariables

If DP sets the value of such a variable itself, the message states where DP takes it from:

The variable "ansible_user" in spec.playbook.vars[0] cannot be used: the module sets it itself, from the username key of the connection Secret.

Scheduled runs

To execute a playbook regularly, create an AnsibleRunSchedule resource. It works the same way as a CronJob resource and creates a separate run with a status of its own each time the schedule triggers. An example of a schedule:

apiVersion: ansible.deckhouse.io/v1alpha1
kind: AnsibleRunSchedule
metadata:
  name: nightly-hardening
  namespace: demo
spec:
  schedule: "0 4 * * *"
  timeZone: "Europe/Moscow"        # UTC by default.
  concurrencyPolicy: Forbid        # Forbid (default) | Allow | Replace
  successfulRunsHistoryLimit: 3
  failedRunsHistoryLimit: 1
  startingDeadlineSeconds: 300     # A run delayed by more than this past its scheduled time is skipped.
  suspend: false
  template:                        # The template of the run created each time the schedule triggers.
    spec:
      target: { type: VirtualMachines, virtualMachines: { selector: { matchLabels: { role: worker } } } }
      connection: { secretRef: { name: ssh-creds } }
      playbook: { type: Git, git: { url: https://github.com/example-org/infra, path: playbooks/site.yml } }

A schedule works by the following rules:

• A run is named after its schedule plus the scheduled time in Unix time format, for example nightly-hardening-1735660800. Old runs are pruned by the history limits, and deleting a schedule deletes all of its runs.
• The schedule spec, run template included, can be changed. Changes apply from the next scheduled time, while a run already started continues with the previous template.
• With concurrencyPolicy: Forbid, DP skips a scheduled time while the previous run has not finished. Skipped runs are never replayed.
• Setting suspend: true pauses the creation of new runs, while the runs already created continue to completion.
• If the schedule or the time zone is invalid, the Ready condition has the InvalidSchedule reason, and DP creates no runs until the error is fixed.

Run results

A run records in its own status the phase of the execution, the per-host task counters and the list of failed tasks.

The following commands show the result:

d8 k get ansibleruns -n demo
d8 k get ansiblerun configure-vm -n demo -o yaml

An example of the status of a successful run:

status:
  phase: PlaybookSucceeded
  message: The playbook run completed successfully.
  hosts:                                # One entry per host, as in PLAY RECAP.
    - name: demo-vm-01
      address: 10.66.10.2
      ok: 3
      changed: 1
      rescued: 1                        # Errors handled by a rescue block.
      ignored: 0                        # Errors skipped by ignore_errors.
  summary: { total: 1, successful: 1, failed: 0, unreachable: 0, skipped: 0 }
  failures: []                          # Failed tasks with their output, up to 20 entries.
  podRef: { name: d8a5e-configure-vm-x7k2p, namespace: demo }
  playbookCommit: 82b6d90a…             # Git source only.
  conditions:
    - type: Completed
      status: "True"
      reason: PlaybookSucceeded

A run can be in one of the phases listed in the table below.

The terminal phase tells you where to look for the cause of an error. With PlaybookFailed the playbook was executed, so look for the cause in its tasks. The exception is the RunFailed reason, where the playbook never started. With Error the run never reached the playbook, so look for the cause in the run setup, for example in dependencies, targets or the network.

Errors a playbook handles itself, through a rescue block or the ignore_errors parameter, are counted in rescued and ignored. The playbook is still considered successful, and the status.failures list stays empty. Ansible still prints such errors with the fatal: prefix in the log, so track them by the rescued and ignored counters.

Troubleshooting

Start troubleshooting a run with the phase of the execution and the reason in the Completed condition. The details are in the status.message field, and the status.podRef.name field holds the name of the runner Pod whose logs contain the full output.

The reasons why a run ends or waits in the Pending phase are listed in the table below. The reasons why individual virtual machines (VMs) are skipped follow after it.

Here <POD_NAME> is the name of the runner Pod from the status.podRef.name field.

The VMs a run skipped are listed in status.skippedHosts. The skip reasons are listed in the table below.

Some problems have no reason of their own in the status of a run. Such symptoms and the way to investigate them are listed in the table below.

The following command shows the logs of the runner Pod:

d8 k logs -n demo -l ansible.deckhouse.io/owner=configure-vm

If the name of a run is longer than 63 characters, DP shortens the value of the ansible.deckhouse.io/owner label and appends a hash to it. A selector with the full run name then does not find such a Pod. In this case, take the Pod name from the status.podRef.name field.

The status.failures list holds up to 20 task errors, while the complete output stays in the logs of the Pod. A failed run keeps its Pod until the AnsibleRun object is deleted, but DP deletes the Pod of a successful run right after the run finishes, and the full Ansible output disappears with it. To keep that output, configure log collection into log storage. In a playbook with several tasks, the task field may point to a wrong task, so check it against the Pod logs.

Collecting logs into log storage

To keep the complete output of the ansible-playbook utility available after the runner Pod is deleted, configure its collection with the log-shipper module. The output stays only in the logs of the Pod, and DP deletes the Pod of a successful run right after the run finishes.

The ready-made DP log collection configurations cover only namespaces labeled heritage: deckhouse, while runner Pods are created in the namespace of the run, so collecting the logs of runs requires a configuration of your own.

The collector reads the log file to its end even after the Pod is gone. The log of a run that finishes in seconds therefore reaches log storage whole, the PLAY RECAP line included. The tail of the output can be lost only if the log destination is unavailable during exactly those seconds.

Collection configured in a project

The owner of a project can collect the logs of runs in its namespace with a PodLoggingConfig resource, because this resource is namespaced. The configuration selects Pods by the label that DP puts on every runner Pod. The name of a Pod is not suitable for selection, because the API server appends a random suffix to it, for example d8a5e-configure-vm-x7k2p. An example of a configuration that sends the logs of every run in the namespace to Loki:

apiVersion: deckhouse.io/v1alpha1
kind: PodLoggingConfig
metadata:
  name: ansible-runners
  namespace: demo
spec:
  labelSelector:
    matchLabels:
      app.kubernetes.io/managed-by: ansible-controller
  clusterDestinationRefs:
    - d8-loki

The clusterDestinationRefs field points at a log destination that already exists in the cluster. The available destinations can be listed with the following command:

d8 k get clusterlogdestinations

Collection configured cluster-wide

The destination (ClusterLogDestination) and log collection across every project (ClusterLoggingConfig) resources are cluster-scoped, so only a cluster administrator can create them. The upside is that the configuration is set up once for the whole cluster, and projects need no configurations of their own:

apiVersion: deckhouse.io/v1alpha2
kind: ClusterLoggingConfig
metadata:
  name: ansible-runner-logs
spec:
  type: KubernetesPods
  kubernetesPods:
    labelSelector:
      matchLabels:
        app.kubernetes.io/managed-by: ansible-controller
  destinationRefs:
    - d8-loki

The Loki that ships with DP is meant for looking into logs right away rather than for archiving them. The retentionPeriodHours parameter of the loki module sets its retention period. But Loki also frees space as the disk fills up, so logs may be kept for less than that period.

To keep the logs of runs for months, create a separate ClusterLogDestination resource for external log storage. The log-shipper module supports the Loki, Elasticsearch, Logstash, Vector, Kafka, Splunk and Socket destinations. The same collection can feed several destinations at once if you list them in the destinationRefs field.

Searching for the log of a run in Loki

Once log collection is configured, the output of a run lands in a stream labeled with the namespace, the Pod and the container. The container with the playbook output is named ansible, and the name of the runner Pod starts with d8a5e- followed by the name of the run. If the name of the run is longer than 51 characters, the Pod name contains only its beginning. The LogQL queries below cover the common cases:

# Every run in a namespace.
{namespace="demo", container="ansible"}

# A single run.
{namespace="demo", pod=~"d8a5e-configure-vm-.+"}

# Every run created by one schedule.
{namespace="demo", pod=~"d8a5e-nightly-hardening-.+"}

# Recap only. The PLAY RECAP line holds ok, changed and failed per host.
{namespace="demo", container="ansible"} |= "PLAY RECAP"

# Failed tasks and unreachable hosts.
{namespace="demo", container="ansible"} |~ "fatal:|UNREACHABLE"

# Number of task errors per Pod over a day.
sum by (pod) (count_over_time({container="ansible"} |= "fatal:" [24h]))

Permissions and security

This section is meant for a cluster administrator who grants users the right to create runs. This right gives wider access to data and networks than it may seem, so take the properties below into account before you grant it.

Reading Secrets and ConfigMaps of a namespace

Kubernetes does not check permissions on a Secret when a Pod references it, and the runner Pod is no exception. The author of a run writes the playbook and can name any Secret or ConfigMap of the namespace in the valueFrom field, so the right to create runs is enough to read their contents. Keep that in mind when granting this right in a namespace whose Secrets not everyone should read.

Additional networks

A ClusterNetwork is visible from any namespace, so the right to create an AnsibleRun also means the right to attach a runner Pod to any cluster-wide L2 domain. The sdn module behaves this way for every workload, not only for runs.

A NetworkPolicy of the project constrains the pod network only, so it does not limit what a run does inside a VLAN.

Variables files

The kubelet mounts the document from the varsFiles field into the runner Pod, and the ansible-playbook utility reads the file, not DP. DP therefore does not check the variable names inside the file, and refused names take effect from it. The ansible_connection variable with the local value is especially dangerous, because with it the playbook is executed inside the runner Pod and succeeds without touching a single target host.

The missing check does not extend the rights of the run author, because a variables file has the same trust level as the playbook. The author of a run already chooses the playbook, the host addresses and the connection Secret.

Annotations of virtual machines

The owner of a virtual machine (VM) can change its annotations, and that owner is not necessarily the author of the run. An annotation therefore may not set variables with the ansible_ prefix. For example, an ansible_host variable in an annotation would redirect someone else’s run to another VM and give that VM the SSH credentials, while an ansible_ssh_common_args variable with a ProxyCommand value would execute an arbitrary command inside the runner Pod.

The only exception is ansible_port, because it does not reduce the security of a run. The host address stays the one DP resolved, so the connection reaches the same VM, and its owner already decides which service listens on each of its ports.

Numeric limits

The numeric limits of a single run are listed in the table below.