User guide

This guide is intended for users of the ansible module and describes how to create runs that execute Ansible playbooks against virtual machines and against hosts named by address. It covers every block of a run manifest, the way targets are resolved, the playbook sources, the format of the result and the troubleshooting steps.

Module resources

The module adds two custom resources to the cluster, which are listed in the table below.

From here on, a run means an AnsibleRun object, and a playbook means the YAML file with tasks that the run executes.

All objects a run references have to be in its namespace. Such objects are the Secret with credentials, the ConfigMap with the playbook text and the target VirtualMachine resources.

Quick start

An example of configuring a virtual machine, in which the run checks connectivity to the machine and records the result in its status.

1. Prepare the virtual machine. The run needs an SSH server on it, a login user and an address the platform knows about. The cloud-init fragment below creates such a user:

   #cloud-config
   packages:
     - openssh-server
   users:
     - name: ansible
       lock_passwd: false
       sudo: ALL=(ALL) NOPASSWD:ALL
       ssh_authorized_keys:
         - ssh-ed25519 <SSH_PUBLIC_KEY> ansible@example
   runcmd:
     - systemctl enable --now ssh

   <SSH_PUBLIC_KEY> is the public key whose private half goes into the Secret in the next step.

2. Put the credentials into a Secret next to the run:

   apiVersion: v1
   kind: Secret
   metadata:
     name: ssh-creds
     namespace: demo
   type: Opaque
   stringData:
     username: ansible
     ssh-privatekey: |
       -----BEGIN OPENSSH PRIVATE KEY-----
       ...
       -----END OPENSSH PRIVATE KEY-----

3. Create the run:

   apiVersion: ansible.deckhouse.io/v1alpha1
   kind: AnsibleRun
   metadata:
     name: configure-vm
     namespace: demo
   spec:
     target:
       type: VirtualMachines
       virtualMachines:
         selector:
           matchLabels:
             vm: vm-01
     connection:
       secretRef:
         name: ssh-creds
     playbook:
       type: Inline
       inline: |
         ---
         - name: Configure VM
           hosts: all
           tasks:
             - name: Check SSH connectivity
               ansible.builtin.ping:

4. Check the result of the run:

   d8 k get ansibleruns -n demo

   Example output:

   NAME           PHASE               REASON              HOSTS   OK   FAILED   UNREACHABLE   SKIPPED   AGE
   configure-vm   PlaybookSucceeded   PlaybookSucceeded   1       1    0        0             0         42s

   The detailed result, including per-host task counters, is available in the status of the run:

   d8 k get ansiblerun configure-vm -n demo -o yaml

Run structure

A run manifest is made of four blocks, which are listed in the table below. Each block is described in a section of its own.

The manifest below has every block of a run filled in:

apiVersion: ansible.deckhouse.io/v1alpha1
kind: AnsibleRun
metadata:
  name: configure-db
  namespace: demo
spec:
  target:
    type: VirtualMachines            # VirtualMachines | Hosts
    virtualMachines:
      selector:                      # selector OR names, never both
        matchLabels:
          role: db
  connection:
    secretRef:
      name: ssh-creds                # SSH credentials, same namespace
    network:                         # optional; the main network by default
      type: ClusterNetwork           # Main | Network | ClusterNetwork
      name: vlan-64
  playbook:
    type: Inline                     # Inline | ConfigMap | Git
    inline: |
      ---
      - hosts: all
        tasks:
          - ansible.builtin.ping:
    tags: [deploy]                   # optional; run only these tags
    skipTags: [migrations]           # optional; leave these tags out
    vars:                            # optional; --extra-vars for this run
      - name: app_version
        value: "1.4.2"               # any YAML: string, number, list, mapping
      - name: db_password
        valueFrom:
          secretKeyRef:
            name: app-secrets
            key: db-password
    varsFiles:                       # optional; a YAML document of variables
      - configMapRef:
          name: app-config           # key defaults to vars.yaml
  runner:
    dryRun: false                    # Ansible check mode
    diff: false
    verbosity: 0                     # 0..4

Targets

The target block defines the hosts the playbook is executed on. Such hosts are either virtual machines of the platform or hosts named by address. One run works with hosts of a single kind only, so the virtualMachines and hosts fields cannot be set in the same object.

The target block is used instead of the -i parameter of the Ansible command line. From this block the controller builds the inventory file that would otherwise have to be written and kept up to date by hand. Host addresses are resolved when the run starts, from the data of the platform, so a virtual machine that was recreated with another address is configured without any change to the run manifest.

Virtual machines

Virtual machines are selected by labels or named explicitly. The addresses of the selected machines are resolved by the controller when the run starts.

The example below selects a machine by label:

target:
  type: VirtualMachines
  virtualMachines:
    selector:
      matchLabels:
        vm: vm-01

An example of selecting machines by a label expression:

target:
  type: VirtualMachines
  virtualMachines:
    selector:
      matchExpressions:
        - key: vm
          operator: In
          values: [vm-01, vm-02]

An example of selecting machines by name:

target:
  type: VirtualMachines
  virtualMachines:
    names: [demo-vm-01, demo-vm-02]

The selector and names fields are mutually exclusive, and a manifest with an empty selector is refused by the cluster. A run is executed once, so its targets have to be stated explicitly.

If a machine matches the selection but cannot be configured when the run starts, it is listed in status.skippedHosts with a reason, and the playbook is executed on the remaining machines. The possible skip reasons are listed in Troubleshooting.

Hosts named by address

The addresses of some hosts are unknown to the platform. Such hosts are machines whose address is configured inside the guest OS by hand or handed out by an external DHCP server, as well as physical servers, network appliances and machines from other clusters. Those hosts are configured with targets of type Hosts, where the addresses are named in the run manifest.

An example of a run with hosts named by address:

target:
  type: Hosts
  hosts:
    - address: 192.168.55.10
      vars:
        - name: app_role
          value: frontend
      groups: [web]
    - address: db.example.com

A host named by address is not a VirtualMachine resource, so the run treats it differently from a virtual machine of the platform. The differences are listed in the table below.

Because a host named by address is not checked for another active run, two runs can work on the same address at the same time. Separating such runs is the responsibility of the user who creates them.

The vars field of such a host is declared like the host variables of a run, valueFrom sources included. Group names follow the rules described in Ansible inventory.

Ansible inventory

An inventory file does not have to be written. The controller builds it from the selected targets and from the annotations of the virtual machines. The ansible_host and ansible_user variables and the credential paths are set by the controller and cannot be overridden. Every other variable is set by the user.

For a virtual machine, host groups and variables come from its annotations, which are listed in the table below.

apiVersion: virtualization.deckhouse.io/v1alpha2
kind: VirtualMachine
metadata:
  name: demo-vm-01
  namespace: demo
  annotations:
    ansible.deckhouse.io/groups: "web,production"
    vars.ansible.deckhouse.io/app_role: "frontend"

An annotation is host_vars/<HOST_NAME> written on the machine itself. For a host named by address, the same thing lives in the spec (vars, groups).

In both cases the controller builds an inventory file of the following form:

all:
  hosts:
    demo-vm-01:
      ansible_host: 10.66.10.2
      ansible_user: ansible
      ansible_ssh_private_key_file: /home/runner/.ssh/ssh-privatekey
      app_role: frontend
web:
  hosts:
    demo-vm-01: {}
production:
  hosts:
    demo-vm-01: {}

Variable names in annotations follow the same rules as the refused variable names. One more rule applies to annotations only. An annotation may not set variables with the ansible_ prefix, with the ansible_port variable as the single exception.

This restriction exists because an annotation can be changed by the owner of the virtual machine, who is not necessarily the author of the run. For example, an ansible_host variable in an annotation would redirect someone else’s run to another machine and hand it the SSH credentials, while an ansible_ssh_common_args variable with a ProxyCommand value would execute an arbitrary command inside the runner Pod.

The ansible_port variable is an exception because an SSH port belongs to a particular machine. The spec.playbook.vars field sets one port for the whole run, and machines are selected by label, so a non-standard port is known to the machine alone:

metadata:
  annotations:
    vars.ansible.deckhouse.io/ansible_port: "2222"

Such an exception does not reduce the security of a run. The address of the host stays the one the controller resolved, so the connection lands on that same machine, and the owner of the machine already decides which service listens on each of its ports.

A port therefore comes from three places, from the weakest to the strongest:

• spec.playbook.vars sets the port for every host of the run.
• hosts[].vars sets the port for a single host named by address.
• The annotation of a machine sets the port for that machine only and overrides both values above.

Annotations whose names do not start with the two listed prefixes are ignored by the controller, and empty group entries are skipped. If a naming rule is violated, the run ends in the Error phase with the InvalidInventoryAnnotations reason before a Pod is created. The status message then names the machine and the annotation that violated the rule.

Connection

The connection block defines over which transport, with which credentials and through which network a run connects to its hosts.

Transport

The transport defines how a run connects to a host and is named in the connection.type field. In the current version of the module SSH is the only available value and is used by default, so the field can be omitted:

connection:
  type: SSH
  secretRef:
    name: ssh-creds

SSH credentials

A run takes the credentials for the connection from the Secret whose name is set in the connection.secretRef field:

connection:
  secretRef:
    name: ssh-creds

Such a Secret has to be in the namespace of the run and hold the keys listed in the table below. At least one of the ssh-privatekey or password keys is required.

The key names follow the built-in kubernetes.io/ssh-auth and kubernetes.io/basic-auth Secret types.

The example below shows a Secret that holds a password instead of a private key:

apiVersion: v1
kind: Secret
metadata:
  name: ssh-creds-password
  namespace: demo
type: Opaque
stringData:
  username: ansible
  password: ansible

Networks

By default a run is executed in the main network. The runner Pod works from the cluster pod network and connects to a machine at the address from the status.ipAddress field of the VirtualMachine resource.

A virtual machine can also be attached to additional networks of the sdn module, either a project one (Network) or a cluster-wide one (ClusterNetwork). The SSH server in the guest OS may listen in such a network only.

An additional network is a separate L2 domain, and the pod network has no route into it, so the address of a machine alone is not enough to connect. A run has to be executed in the same network the machine is in.

The network is named in the connection.network block. The runner Pod is attached to the named network, and the controller takes the address of every target from it as well:

connection:
  secretRef:
    name: ssh-creds
  network:
    type: ClusterNetwork   # Main (default) | Network | ClusterNetwork
    name: vlan-64

The network block mirrors an entry of the spec.networks list of the virtual machine, so its values are copied from the manifest of the machine:

# VirtualMachine
spec:
  networks:
    - type: Main
    - type: ClusterNetwork
      name: vlan-64        # the network to name in the run
    - type: Network
      name: storage-net

Take the following properties into account before configuring an additional network:

• Address pool of the network (spec.ipam.ipAddressPoolRef): Both the machine and the runner take their address from it, so a network without a pool is L2-only. The run ends in Error with NetworkWithoutIPAM, naming the network to fix, and no Pod is created.
• Interface setup in the guest OS: The address is delivered over DHCP, so the guest needs a DHCP client on that interface. Otherwise the platform reports an address the machine does not answer at, and the host is unreachable. Match the interface by MAC or by a predictable name rather than by ethX, because attaching a network or reordering spec.networks can renumber ethX inside the guest, and the DHCP client then works on the wrong interface.
• One network per run: Machines listening in different networks are split by labels into several runs. Every manifest then states where its run goes.
• Visibility of the network: A Network is visible only in its own namespace, while a ClusterNetwork is available from any.
• Size of the pool: The addresses have to cover the machines plus the concurrent runs, because a run holds an address while its Pod lives. The Pod of a successful run is deleted at once, while the Pod of a failed one stays until the AnsibleRun is deleted, so failed runs left behind keep their addresses.
• Hosts named by address: They need network too, because the address alone does not put the runner there. That is how a management VLAN gets configured at all, for physical servers, network appliances and machines of other clusters, none of which exist in this cluster.

The runner Pod stays attached to the cluster pod network as well, because an additional interface is added to the main one rather than replacing it. A playbook from a Git repository inside the cluster and Galaxy content from a mirror in the cluster are therefore fetched the same way as in a run without an additional network. The init containers still see the cluster DNS and the cluster services.

An address pool that publishes routes is an exception. Those routes are applied inside the Pod, and a default route (0.0.0.0/0) sends all outbound traffic through the additional network. Requests to cluster-internal addresses keep working, because they have their own routes. Data the run fetches from outside the cluster, for example from a public Git repository or from Galaxy, is sent through the additional network and may become unreachable. Check the list of routes in the spec.pools[].routes field of the pool before executing a run in such a network.

Additional networks have three more properties that follow from how the platform builds them, and a cluster administrator should take them into account before granting the right to create runs:

• Network policies: A NetworkPolicy of the project constrains the pod network only, so it does not limit what a run does inside a VLAN.
• Availability of a ClusterNetwork: It is visible from any namespace, so the right to create an AnsibleRun is the right to put a Pod into any cluster-wide L2 domain. This is how the sdn module works for every workload, not something this module adds. It is worth knowing before handing out that right.
• MTU: It comes from the network and must not exceed the MTU of the node interfaces it is built on. A mismatch shows up in the worst way. SSH connects, and then copying a file or installing a package hangs.

Playbook

The playbook block defines where the playbook comes from and which part of it to execute. The source of a playbook is text in the run manifest, a ConfigMap resource or a Git repository. A single run uses one source only.

Inline playbook

The playbook text is set directly in the manifest of the run:

playbook:
  type: Inline
  inline: |
    ---
    - name: Configure VM
      hosts: all
      tasks:
        - ansible.builtin.ping:

The size of an inline playbook is limited to 64KB. For a larger playbook, use a ConfigMap resource or a Git repository.

Playbook from a ConfigMap

The playbook is stored in a ConfigMap resource in the namespace of the run, and the run points at the key of that resource:

playbook:
  type: ConfigMap
  configMapRef:
    name: demo-playbook
    key: playbook.yaml     # default

Playbook project from Git

For projects with roles, group_vars and templates the whole tree is checked out, so roles next to the playbook work as they are:

playbook:
  type: Git
  git:
    url: https://github.com/example-org/infra
    revision: v1.4.0             # branch, tag or SHA; default branch if omitted
    path: playbooks/site.yml     # default playbook.yaml
    secretRef:
      name: git-token            # private repositories

The settings above correspond to the following Ansible commands:

git clone --recurse-submodules https://github.com/example-org/infra && cd infra
git checkout v1.4.0
ansible-galaxy install -r requirements.yml     # when the project declares any
ansible-playbook -i inventory.yml playbooks/site.yml

If the checkout contains a requirements.yml, roles/requirements.yml or collections/requirements.yml file, the Galaxy roles and collections declared in it are installed before the playbook starts. Submodules of the repository are fetched as well. The commit that was actually executed is recorded by the controller in the status.playbookCommit field.

Fetching the repository and the Galaxy content is the only traffic of a run that leaves the cluster, and it goes through the proxy configured for the cluster. The module has no proxy settings of its own. The tasks of the playbook are not part of that traffic, because they reach the targets over SSH. If a task needs a proxy, declare it in the environment parameter of that task.

Private repositories

Access to a private repository requires separate credentials, unrelated to the SSH credentials for the target hosts. The controller takes them from the Secret named in the git.secretRef field and mounts them into the checkout step only, so the tasks of the playbook never see them.

For URLs of the form ssh://git@… the Secret is required and has to carry the ssh-privatekey and known_hosts keys, because host keys are verified strictly. The contents of known_hosts can be obtained with the ssh-keyscan github.com command.

apiVersion: v1
kind: Secret
metadata:
  name: git-deploy-key
  namespace: demo
type: kubernetes.io/ssh-auth
stringData:
  ssh-privatekey: |
    -----BEGIN OPENSSH PRIVATE KEY-----
    ...
  known_hosts: |
    github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5...

For HTTP(S) URLs with a token, the username and password keys are required, where the token is the password. GitLab expects the username oauth2. If the repository is hosted on a server with its own certificate authority, add the ca.crt key to the Secret:

apiVersion: v1
kind: Secret
metadata:
  name: git-token
  namespace: demo
type: kubernetes.io/basic-auth
stringData:
  username: oauth2
  password: glpat-xxxxxxxxxxxx

Selective runs by tag

Tags define which part of the playbook is executed and work the same way as the --tags and --skip-tags parameters of the Ansible command line.

playbook:
  type: Git
  git:
    url: https://github.com/example-org/infra
    path: site.yml
  tags: [deploy, config]     # run only the tasks marked with these
  skipTags: [migrations]     # and leave these out

The settings above correspond to the following command:

ansible-playbook -i inventory.yml site.yml --tags deploy,config --skip-tags migrations

If both fields are set, the run executes the tasks selected by tags minus the tasks excluded by skipTags. The ansible-playbook utility behaves the same way.

A tag is any string without whitespace or a comma. The comma is the separator of tags in the argument, and Ansible strips whitespace itself. Any other notation used in playbooks is allowed, including role:install and db/migrate.

The tags reserved by Ansible keep their standard meaning, which is listed in the table below.

Filter with no matches

A filter that matches no task is not an error. The ansible-playbook utility exits successfully without having executed a single task, and the run ends in the PlaybookSucceeded phase with zero counters. A typo in a tag name produces the same result, so the run reports the case in its status:

$ d8 k get ansiblerun configure-db -o jsonpath='{.status.message}'
No task matched the tags of the run.

Tags are unrelated to the runner options. The dryRun and verbosity parameters change the execution mode and the amount of output, but do not affect the set of tasks that are executed.

Runner options

The runner block changes how the playbook is executed and how much it prints, without affecting the set of tasks:

runner:
  dryRun: true      # Ansible check mode
  diff: true
  verbosity: 2      # 0..4

The settings above correspond to the following command:

ansible-playbook -i inventory.yml site.yml --check --diff -vv

With the dryRun parameter the run reports which changes would be made, and together with the diff parameter it prints those changes in detail. In check mode Ansible skips the tasks of the command and shell modules.

If the connection Secret holds an ansible-vault-password key, the controller adds the --vault-password-file parameter to the invocation, so a project with vaulted files is executed unchanged.

Variables

Variables set the values a playbook is executed with and make changes to the playbook itself unnecessary. Thanks to that, one Git project serves several environments, and a task receives a password that is not stored in the run manifest.

In the simplest case a variable holds a value the playbook reads as {{ app_version }}:

spec:
  playbook:
    vars:
      - name: app_version
        value: "1.4.2"

The notation matches the env block of a Pod. A variable has a name, and its value is either set directly in the value field or taken from another object through the valueFrom field. The value field accepts a string, a number, a list or a mapping.

There are three ways to supply a value, which are listed in the table below.

The manifest below uses all three ways:

spec:
  playbook:
    vars:
      - name: app_version
        value: "1.4.2"
      - name: packages
        value: [nginx, curl, git]
      - name: users
        value:
          - name: deploy
            groups: [wheel]
      - name: db_password
        valueFrom:
          secretKeyRef:
            name: app-secrets
            key: db-password
    varsFiles:
      - configMapRef:
          name: app-config

The command line this stands for:

ansible-playbook -i inventory.yml site.yml \
  -e @app-config-vars.yml \
  -e '{"app_version":"1.4.2","packages":["nginx","curl","git"],"users":[{"name":"deploy","groups":["wheel"]}]}' \
  -e db_password="$DB_PASSWORD"

The spec.playbook.vars field corresponds to the Extra Variables field of AWX and to the -e parameter of the command line, and it has the same maximum precedence.

A variable name is an Ansible identifier, that is a letter or an underscore followed by letters, digits and underscores. A name that does not match this rule is refused by the API server. Some names are refused in addition to that, and they are listed in Refused variable names.

Values from a Secret or a ConfigMap

A password or another value that already exists does not have to be copied into the manifest of a run. The valueFrom field takes the value from one key of a Secret or a ConfigMap, and the playbook sees the variable under the name set in the run:

spec:
  playbook:
    vars:
      - name: db_password
        valueFrom:
          secretKeyRef:
            name: app-secrets
            key: db-password

The value is substituted by the kubelet when the runner Pod starts. The controller accesses the Secret and the ConfigMap only to verify the key names, so the value is not copied into any object of the module and does not appear in the arguments of the process, which are visible in the output of ps. The variables file holds only a reference to the environment variable:

db_password: "{{ lookup('env', 'ANSIBLE_VAR_db_password') }}"

This way of passing values has the properties listed below.

• Permissions on that Secret: They are not checked along the way, the same as for any Pod that references one. Creating a run is therefore enough to read what the objects it names contain, because the owner of the run writes the playbook. Keep that in mind when granting the right to create runs in a namespace whose Secrets not everyone should read.
• Name of the environment variable: ANSIBLE_VAR_ followed by the variable name as written, case included. Ansible treats db_password and DB_PASSWORD as two variables, and the runner’s environment holds two entries. A Secret whose keys are UPPER_SNAKE therefore sits next to a variable named the Ansible way without either having to be renamed.
• One name, one value: If the same name is fed from two different keys (two hosts each naming their own Secret for it), the run is refused with InvalidVariables, because there is a single environment per runner Pod.
• A key holding binary data (binaryData of a ConfigMap): It never becomes a variable, and naming such a key fails the run and says so.

Type of a value from a single key

A key of a Secret or a ConfigMap holds a string, and that string is what reaches the playbook. The value field and a document named in varsFiles keep the shape of a value, while a single key does not.

vars:
  - name: packages
    value: [nginx, curl]          # a list, arrives as a list
  - name: extra_packages
    valueFrom:
      configMapKeyRef:            # "[nginx, curl]" — a string
        name: app-config
        key: extra_packages

If the playbook needs a list, it parses the string itself:

- hosts: all
  tasks:
    - ansible.builtin.package:
        name: "{{ extra_packages | from_yaml }}"
        state: present

This is exactly why a task fails with an error such as “expected a list, got a string”. If a set of values has structure, pass it as a document in the varsFiles field, where the types of the values are already set and nothing has to be parsed.

Set of variables in a file

A set of variables is stored in a file the same way as in the group_vars/all.yml file of an ordinary Ansible project. Place such a file in a ConfigMap or a Secret resource and name it in the run. This way corresponds to the -e @vars.yml parameter of the command line.

apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  vars.yaml: |                  # an ordinary YAML document
    app_tier: backend
    app_version: "1.10"         # quoted, so it stays a string
    packages: [nginx, curl]     # a list stays a list
    limits:
      cpu: 2
      mem: 4Gi

spec:
  playbook:
    varsFiles:
      - configMapRef:
          name: app-config        # key defaults to vars.yaml
      - secretRef:
          name: app-secrets
          key: prod.yaml
          optional: true

The default key is vars.yaml, just as the default key of a ConfigMap with a playbook is playbook.yaml. Set the key field if a single object carries several sets of variables, such as dev.yaml and prod.yaml.

The main purpose of a file is that the types of the values in it survive. Whoever wrote the file already stated what is a string and what is a number, while a single key of a Secret or a ConfigMap cannot carry that. For example, the value 1.10 in a single key is text, and turning it into a number would change a version into 1.1.

The file is read by the ansible-playbook utility rather than by the controller, because the kubelet mounts the key into the runner Pod. Two conclusions follow from that:

• The contents pass through no object of the module and appear in no process argument. They show up neither in the manifest of the run nor in the process list on the node.
• The variable names inside are not checked: The names refused everywhere else, listed in refused variable names, take effect from a file. One of them is worth knowing by name. With ansible_connection: local the playbook runs inside the runner and reports a green run without touching a single target.

The second conclusion is a warning about a possible mistake rather than an extension of rights. A variables file is trusted the same way the playbook is, and the author of a run already chooses the playbook, the host addresses and the connection Secret.

Host variables

Host variables describe the machine rather than the run, so they are set next to the host they belong to.

The hosts[].vars field is declared the same way as a variable of a run, valueFrom sources included:

target:
  type: Hosts
  hosts:
    - address: 192.168.55.10
      vars:
        - name: app_role
          value: frontend
        - name: ansible_port
          value: 2222
      groups: [web]

Such variables land in the inventory file next to the machine they describe and are meant for facts about it, for example datacenter, app_role or a non-standard SSH port. For a virtual machine the same values come from annotations.

A variable of the same name in the spec.playbook.vars field takes precedence over both ways, because the annotation was written in advance while the values of a run apply at execution time.

The exception is the case where both sides take their value from a Secret or a ConfigMap and from different keys. Precedence does not apply then, and the run is refused with the InvalidVariables reason before the playbook starts, because there is a single environment per runner Pod. To avoid that, give those values different names. Two literal values, or a literal value together with a valueFrom source, are not affected.

Precedence of values

One rule governs the values: the variables of a run take precedence over every value a playbook or its project sets. A host variable, set by a VM annotation or by the hosts[].vars field, takes precedence over the group_vars/ directory of the project, but loses to a vars block inside a play.

Every value from the vars and varsFiles fields is passed as --extra-vars, that is level 22 of the Ansible precedence list, the highest one. The full order of precedence, from the weakest to the strongest, is listed in the table below.

A play in the table means a - hosts: … tasks: … block, and a playbook may hold several of them.

The spec.playbook.vars and spec.playbook.varsFiles fields sit on the same level of precedence, and the order of the arguments separates them. The files are passed first, in list order, and the spec.playbook.vars field is passed last. The ansible-playbook utility applies -e parameters from left to right, so the resulting order is visible in the arguments of the Pod, where the entry further right wins.

d8 k -n <NAMESPACE> get pod -l ansible.deckhouse.io/owner=<RUN_NAME> \
  -o jsonpath='{.spec.containers[0].args}'
# … -e @/ansible/vars.d/0/vars.yaml -e @/ansible/vars.d/1/prod.yaml -e @/ansible/vars.yml

A stronger value replaces the name of a variable as a whole, and mappings do not merge. For example, the value limits: {cpu: 2, mem: 4Gi} in one file and the value limits: {cpu: 4} in the next one give the playbook limits: {cpu: 4}, and the mem value is lost. Keep one mapping in one file, and override a single name with the spec.playbook.vars field, whose value no file can override back.

The described order of precedence means that a shared Git project cannot protect its own variable names from a run. Keep values that belong to the project in its group_vars/ directory, and use the vars field for values that differ between runs.

Resolving a name conflict

In the example below the playbook sets the app_version variable itself, the machine carries an annotation with the same name, and the run passes its own value:

# VirtualMachine
metadata:
  annotations:
    vars.ansible.deckhouse.io/app_version: "from-annotation"   # level 8
---
# AnsibleRun
spec:
  playbook:
    vars:
      - name: app_version
        value: "from-the-run"                                    # level 22
    playbook:
      type: Inline
      inline: |
        - hosts: all
          vars:
            app_version: "from-the-playbook"                     # level 12
          tasks:
            - ansible.builtin.debug:
                msg: "{{ app_version }}"

The task prints the value from-the-run. If the spec.playbook.vars field is removed, the task prints the value from-the-playbook rather than the value of the annotation, because level 12 is higher than level 8. The value of an annotation applies only when the playbook does not set that variable.

Types of values

The type of a value is determined by the way that value is written in the manifest. The schema of the resource validates variable names only and does not validate the contents of a value, so a typo inside a nested structure shows up when the playbook is executed.

The familiar YAML pitfall applies here as well, where the value 1.10 is the number 1.1 and the value no is the boolean false. A version is a string, so it has to be quoted:

vars:
  - name: app_version
    value: "1.10"      # unquoted it becomes 1.1

Integers keep their type, so the value value: 1000000 reaches the playbook as the integer 1 000 000, and a large integer does not lose precision.

Value with curly braces

Whether a Jinja expression inside a value is evaluated depends on the source of that value. The possible cases are listed in the table below.

The value field is written by the author of the run, so its value is templated the same way as a variable in any variables file. A value from the valueFrom field arrives through the lookup('env', ...) expression, and Ansible marks the result of that lookup as unsafe. Because of that, the contents of an object the run does not own cannot become an expression executed inside the runner Pod.

To keep braces literal in the value field, escape them the way Ansible expects:

vars:
  - name: template_source
    value: "{{ '{{' }} not a template {{ '}}' }}"   # arrives as {{ not a template }}

Missing source of variables

If a declared source of variables is missing, the run neither proceeds nor fails right away. The behaviour differs for a missing object and for a missing key, the same way it does for a ConfigMap with a playbook. The possible cases are listed in the table below.

No reference is emitted for a key that is not there, so the when: x is defined construct and the default() filter keep working in the playbook.

The optional field exists on variable sources and does not exist on the connection.secretRef and playbook.configMapRef fields, because a run without credentials and without a playbook cannot be executed.

Refused variable names

A variable name is an Ansible identifier that matches the [A-Za-z_][A-Za-z0-9_]* pattern. For example, Ansible treats the {{ app-role }} expression not as a variable reference but as a subtraction. A name that does not match the pattern is refused by the API server, so the error arrives in response to the d8 k apply command and names the field:

The AnsibleRun "pattern-check" is invalid:
* spec.target.hosts[0].vars[0].name: Invalid value: "app-role": spec.target.hosts[0].vars[0].name in body should match '^[A-Za-z_][A-Za-z0-9_]*$'

Group names follow the same rule and are refused the same way.

Two groups of names are refused in addition to that.

The first group is the names the module sets itself. Such names are refused in every field where a variable is declared, because the API already provides a separate field for each of them, and two sources of one value lead to authentication errors that are hard to diagnose. The mapping between the names and the fields is listed in the table below.

The ansible_connection variable with the local value would execute the playbook inside the runner Pod and report success, although not a single task ran on the target machines.

Every other variable with the ansible_ prefix is allowed in the run manifest, for example ansible_port, ansible_python_interpreter, ansible_become_user, ansible_become_method, ansible_shell_type and ansible_ssh_common_args.

The second group is the Ansible magic variables listed in the Ansible documentation, for example groups, hostvars, inventory_hostname, playbook_dir, role_path and omit. Such names are refused in every field of a run. Ansible fills the values of magic variables in itself, so setting one either has no effect or breaks the evaluation of expressions. The omit variable is a special case, because its value means “leave this parameter out”. A variable with that name would change what every task using omit does, and no error would be raised.

The places where the name checks are performed are listed in the table below.

If the check in the third column of the table does not pass, the run ends before a Pod is created, and the status message names the field and the reason:

status:
  phase: Error
  message: 'The variable "omit" in spec.playbook.vars[0] cannot be used: it shadows an Ansible magic variable, which Ansible fills in itself.'
  conditions:
    - type: Completed
      status: "False"
      reason: InvalidVariables

For a name the module sets itself, the message points at the field where the value has to be set:

The variable "ansible_user" in spec.playbook.vars[0] cannot be used: the module sets it
itself, from the username key of the connection Secret.

Scheduled runs

A schedule executes a playbook on a timer the same way a CronJob resource starts Job objects. Every cron tick creates a run of its own, with a status of its own. A schedule is described by the AnsibleRunSchedule resource:

apiVersion: ansible.deckhouse.io/v1alpha1
kind: AnsibleRunSchedule
metadata:
  name: nightly-hardening
  namespace: demo
spec:
  schedule: "0 4 * * *"
  timeZone: "Europe/Moscow"        # UTC by default
  concurrencyPolicy: Forbid        # Forbid (default) | Allow | Replace
  successfulRunsHistoryLimit: 3
  failedRunsHistoryLimit: 1
  startingDeadlineSeconds: 300     # a tick missed by more than this is dropped
  suspend: false
  template:                        # the AnsibleRun created on every tick
    spec:
      target: { type: VirtualMachines, virtualMachines: { selector: { matchLabels: { role: worker } } } }
      connection: { secretRef: { name: ssh-creds } }
      playbook: { type: Git, git: { url: https://github.com/example-org/infra, path: playbooks/site.yml } }

Scheduling follows these rules:

• A run is named after its schedule plus the tick time in Unix format, for example nightly-hardening-1735660800. Old runs are pruned by the history limits, and deleting a schedule deletes all of its runs.
• The schedule spec, run template included, is editable, unlike the immutable spec of a single run. Changes apply from the next tick, while the current run continues with the previous template.
• Setting concurrencyPolicy: Forbid skips a tick while the previous run is still active. Missed ticks are never replayed.
• Setting suspend: true pauses the creation of new runs. The runs already created continue to completion.
• An invalid schedule or time zone shows up in the Ready condition with the InvalidSchedule reason. No runs are created from such a schedule until the error is fixed.

Run results

A run records the outcome in its own status, which holds the phase of the execution, the per-host task counters and the list of tasks that failed.

The following commands show the result:

d8 k get ansibleruns -n demo
d8 k get ansiblerun configure-vm -n demo -o yaml

status:
  phase: PlaybookSucceeded
  message: Run completed successfully   # on failure, also the Pod holding the full output
  hosts:                                # one entry per host, mirroring PLAY RECAP
    - name: demo-vm-01
      address: 10.66.10.2
      ok: 3
      changed: 1
      rescued: 1                        # errors handled by a rescue block
      ignored: 0                        # errors swallowed by ignore_errors
  summary: { total: 1, successful: 1, failed: 0, unreachable: 0, skipped: 0 }
  failures: []                          # failed tasks with their output, up to 20
  podRef: { name: d8a5e-configure-vm-x7k2p, namespace: demo }
  playbookCommit: 82b6d90a…             # Git source only
  conditions:
    - type: Completed
      status: "True"
      reason: PlaybookSucceeded

A run can be in one of the phases listed in the table below.

The terminal phase shows whether the playbook ran. That decides where to look for the cause of an error, in the tasks of the playbook or in the preparation of the run.

Errors a playbook handles itself, through a rescue block or the ignore_errors parameter, are counted in rescued and ignored. The playbook stays successful, and the status.failures list stays empty. Ansible still prints such errors with the fatal: prefix in the log, so they have to be tracked by the counters listed above.

Troubleshooting

Troubleshooting a run starts with the phase of the execution and the reason in the Completed condition. The status.message field carries the same text and, on failure, also names the runner Pod whose logs hold the full playbook output.

The reasons why the execution of a run never started are listed in the table below. There is no runner Pod in these cases, so the manifest of the run itself has to be fixed.

The machines a run skipped are listed in status.skippedHosts, and the playbook is executed on the remaining machines. The possible skip reasons are listed in the table below.

The reasons why a run whose execution started ended are listed in the table below.

Some problems have no reason of their own in the status of a run. Such symptoms and the way to investigate them are listed in the table below.

The following command shows the logs of the runner Pod:

d8 k logs -n demo -l ansible.deckhouse.io/owner=configure-vm

The status.failures list holds up to 20 task errors. When more tasks fail, the complete output stays in the logs of the Pod, which is kept until the AnsibleRun object is deleted. A successful run keeps its whole result in the status, so its Pod is deleted right away.

This cleanup order has to be taken into account in two cases:

• The full Ansible output of a successful run is removed together with its Pod. A run whose log has to be kept is collected into a log store.
• A failed run in an additional network holds its address from the pool until the AnsibleRun object is deleted. Failed runs left in a namespace gradually consume the pool.

Collecting logs into a store

The complete output of the ansible-playbook utility stays in the runner Pod, and the Pod of a successful run is deleted seconds after it finishes. To keep that output, configure its collection with the log-shipper module. The log then remains available after the Pod is deleted.

Runner Pods are not system ones and are created in the namespace of the run, so the ready-made log collection configurations of the platform do not select them. Those configurations cover only namespaces labeled heritage: deckhouse, so collecting the logs of a run requires a configuration of its own.

Collection configured in a project

The PodLoggingConfig resource is namespaced, so the owner of the project can create it. The Pods are selected by the label the controller puts on every runner Pod. The name of a Pod is not suitable for selection, because the apiserver appends a suffix of its own to the name of the run, so it is unique for every attempt to start it, for example d8a5e-configure-vm-x7k2p.

apiVersion: deckhouse.io/v1alpha1
kind: PodLoggingConfig
metadata:
  name: ansible-runners
  namespace: demo
spec:
  labelSelector:
    matchLabels:
      app.kubernetes.io/managed-by: ansible-controller
  clusterDestinationRefs:
    - d8-loki

The clusterDestinationRefs field points at a log destination that already exists in the cluster. The following command lists the available destinations:

d8 k get clusterlogdestinations

Collection configured cluster-wide

A destination (ClusterLogDestination) and collection across every namespace (ClusterLoggingConfig) are cluster-scoped resources, so only a cluster administrator can create them. Such a configuration is set up once for the whole cluster and does not require a configuration in every project:

apiVersion: deckhouse.io/v1alpha2
kind: ClusterLoggingConfig
metadata:
  name: ansible-runner-logs
spec:
  type: KubernetesPods
  kubernetesPods:
    labelSelector:
      matchLabels:
        app.kubernetes.io/managed-by: ansible-controller
  destinationRefs:
    - d8-loki

A destination for long-term storage is created by a cluster administrator as well. The Loki that ships with the platform is meant for looking into logs right away rather than for archiving them. Its retention comes from the retentionPeriodHours parameter of the loki module, and space is reclaimed as the disk fills up, so the actual depth of storage depends on how much is written.

To keep the logs of runs for months, create a separate ClusterLogDestination resource for an external store. The log-shipper module supports the Loki, Elasticsearch, Logstash, Vector, Kafka, Splunk and Socket destinations. The same collection can feed several destinations at once if they are all listed in the destinationRefs field.

Searching for the log of a run in Loki

Once collection is configured, the output of a run lands in a stream labeled with the namespace, the Pod and the container, where the container is named ansible. The LogQL queries below cover the common cases:

# every run in a namespace
{namespace="demo", container="ansible"}

# a single run
{namespace="demo", pod=~"d8a5e-configure-vm-.+"}

# every run created by one schedule
{namespace="demo", pod=~"d8a5e-nightly-baseline-.+"}

# recap only: the PLAY RECAP line carries ok/changed/failed per host
{namespace="demo", container="ansible"} |= "PLAY RECAP"

# failed tasks and unreachable hosts
{namespace="demo", container="ansible"} |~ "fatal:|UNREACHABLE"

# how many runs failed over a day
sum by (pod) (count_over_time({container="ansible"} |= "fatal:" [24h]))

A run that finishes in seconds is collected whole, the PLAY RECAP line included, because the collector reads the log file to its end even after the Pod is gone. The tail of the output can be lost only if the log destination is unavailable during exactly those seconds.

Numeric limits

The numeric ceilings of a single run are listed in the table below.