Working with virtual machines
Installing and configuring the operating system
How to install an operating system in a virtual machine from an ISO image?
Below is a typical Windows guest OS installation scenario from an ISO image. Before you begin, host the ISO on an HTTP endpoint reachable from the cluster.
-
Create an empty VirtualDisk for OS installation:
apiVersion: virtualization.deckhouse.io/v1alpha2 kind: VirtualDisk metadata: name: win-disk namespace: default spec: persistentVolumeClaim: size: 100Gi storageClassName: local-path -
Create ClusterVirtualImage resources for the Windows OS ISO and the VirtIO driver ISO:
apiVersion: virtualization.deckhouse.io/v1alpha2 kind: ClusterVirtualImage metadata: name: win-11-iso spec: dataSource: type: HTTP http: url: "http://example.com/win11.iso"apiVersion: virtualization.deckhouse.io/v1alpha2 kind: ClusterVirtualImage metadata: name: win-virtio-iso spec: dataSource: type: HTTP http: url: "https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/stable-virtio/virtio-win.iso" -
Create a virtual machine:
apiVersion: virtualization.deckhouse.io/v1alpha2 kind: VirtualMachine metadata: name: win-vm namespace: default labels: vm: win spec: virtualMachineClassName: generic runPolicy: Manual osType: Windows bootloader: EFI cpu: cores: 6 coreFraction: 50% memory: size: 8Gi enableParavirtualization: true blockDeviceRefs: - kind: VirtualDisk name: win-disk - kind: ClusterVirtualImage name: win-11-iso - kind: ClusterVirtualImage name: win-virtio-iso -
Start the virtual machine:
d8 v start win-vm -
Connect to the VM console and complete the OS installation and VirtIO drivers using the graphical installer.
VNC connection:
d8 v vnc -n default win-vm -
After the installation is complete, restart the virtual machine.
-
For further work, connect via VNC again:
d8 v vnc -n default win-vm
How to provide a Windows answer file (Sysprep)?
Unattended Windows installation uses an answer file (unattend.xml or autounattend.xml).
The example answer file below:
- Sets the English UI language and keyboard layout.
- Connects the
VirtIOdrivers for the setup stage (the order of devices inblockDeviceRefson the VirtualMachine resource must match the paths in the file). - Creates disk layout for installation with EFI.
- Creates a user
cloud(administrator, passwordcloud) and a useruser(passworduser).
-
Save the answer file as
autounattend.xml(use the example above or adjust it to your needs). -
Create a secret with the type
provisioning.virtualization.deckhouse.io/sysprep:d8 k create secret generic sysprep-config --type="provisioning.virtualization.deckhouse.io/sysprep" --from-file=./autounattend.xml -
Create a virtual machine that will use the answer file during installation. Specify
provisioningwith typeSysprepRefin the specification. If necessary, add other Base64-encoded files to the specification required for the answer file scripts to run successfully.apiVersion: virtualization.deckhouse.io/v1alpha2 kind: VirtualMachine metadata: name: win-vm namespace: default labels: vm: win spec: virtualMachineClassName: generic provisioning: type: SysprepRef sysprepRef: kind: Secret name: sysprep-config runPolicy: AlwaysOn osType: Windows bootloader: EFI cpu: cores: 6 coreFraction: 50% memory: size: 8Gi enableParavirtualization: true blockDeviceRefs: - kind: VirtualDisk name: win-disk - kind: ClusterVirtualImage name: win-11-iso - kind: ClusterVirtualImage name: win-virtio-iso
How to create a golden image for Linux?
A golden image is a pre-configured virtual machine image that can be used to quickly create new VMs with pre-installed software and settings.
-
Create a virtual machine, install the required software on it, and perform all necessary configurations.
-
Install and configure qemu-guest-agent (recommended):
-
For RHEL/CentOS:
yum install -y qemu-guest-agent -
For Debian/Ubuntu:
apt-get update apt-get install -y qemu-guest-agent
-
-
Enable and start the service:
systemctl enable qemu-guest-agent systemctl start qemu-guest-agent -
Set the VM run policy to runPolicy: AlwaysOnUnlessStoppedManually — this is required so you can shut down the VM.
-
Prepare the image. Clean unused filesystem blocks:
fstrim -v / fstrim -v /boot -
Clean network settings:
-
For RHEL:
nmcli con delete $(nmcli -t -f NAME,DEVICE con show | grep -v ^lo: | cut -d: -f1) rm -f /etc/sysconfig/network-scripts/ifcfg-eth* -
For Debian/Ubuntu:
rm -f /etc/network/interfaces.d/*
-
-
Clean system identifiers:
echo -n > /etc/machine-id rm -f /var/lib/dbus/machine-id ln -s /etc/machine-id /var/lib/dbus/machine-id -
Remove SSH host keys:
rm -f /etc/ssh/ssh_host_* -
Clean systemd journal:
journalctl --vacuum-size=100M --vacuum-time=7d -
Clean package manager cache:
-
For RHEL:
yum clean all -
For Debian/Ubuntu:
apt-get clean
-
-
Clean temporary files:
rm -rf /tmp/* rm -rf /var/tmp/* -
Clean logs:
find /var/log -name "*.log" -type f -exec truncate -s 0 {} \; -
Clean command history:
history -cFor RHEL: reset and restore SELinux contexts (choose one of the following):
-
Option 1: Check and restore contexts immediately:
restorecon -R / -
Option 2: Schedule relabel on next boot:
touch /.autorelabel
-
-
Verify that
/etc/fstabreferences UUID orLABELrather than names like/dev/sdX:blkid cat /etc/fstab -
Reset cloud-init state (logs and seed):
cloud-init clean --logs --seed -
Perform final synchronization and buffer cleanup:
sync echo 3 > /proc/sys/vm/drop_caches -
Shut down the virtual machine:
poweroff -
Create a VirtualImage resource that references the prepared VM’s VirtualDisk:
d8 k apply -f -<<EOF apiVersion: virtualization.deckhouse.io/v1alpha2 kind: VirtualImage metadata: name: <image-name> namespace: <namespace> spec: dataSource: type: ObjectRef objectRef: kind: VirtualDisk name: <source-disk-name> EOFOr create a ClusterVirtualImage resource so the image is available cluster-wide for all projects:
d8 k apply -f -<<EOF apiVersion: virtualization.deckhouse.io/v1alpha2 kind: ClusterVirtualImage metadata: name: <image-name> spec: dataSource: type: ObjectRef objectRef: kind: VirtualDisk name: <source-disk-name> namespace: <namespace> EOF -
Create a new VirtualDisk from the resulting image:
d8 k apply -f -<<EOF apiVersion: virtualization.deckhouse.io/v1alpha2 kind: VirtualDisk metadata: name: <vm-disk-name> namespace: <namespace> spec: dataSource: type: ObjectRef objectRef: kind: VirtualImage name: <image-name> EOF
After completing these steps, you will have a golden image that can be used to quickly create new virtual machines with pre-installed software and configurations.
Configuring virtual machines
How to use cloud-init to configure virtual machines?
Cloud-init is used for initial guest OS configuration on first boot. The configuration is written in YAML and starts with the #cloud-config directive.
When using cloud images (for example, official distribution images), you must provide a cloud-init configuration. Without it, some distributions do not configure network connectivity, and the virtual machine becomes unreachable on the network, even if the main network (Main) is attached.
In addition, cloud images do not allow login by default — you must either add SSH keys for the default user or create a new user with SSH access. Otherwise, you will not be able to access the virtual machine.
Updating and installing packages
Example cloud-config for updating the system and installing packages from a list:
#cloud-config
# Update package lists
package_update: true
# Upgrade installed packages to latest versions
package_upgrade: true
# List of packages to install
packages:
- nginx
- curl
- htop
# Commands to run after package installation
runcmd:
- systemctl enable --now nginx.service
Creating a user
Example cloud-config for creating a local user with a password and SSH key:
#cloud-config
# List of users to create
users:
- name: cloud # Username
passwd: "$6$rounds=4096$saltsalt$..." # Password hash (SHA-512)
lock_passwd: false # Do not lock the account
sudo: ALL=(ALL) NOPASSWD:ALL # Sudo privileges without password prompt
shell: /bin/bash # Default shell
ssh-authorized-keys: # SSH keys for access
- ssh-ed25519 AAAAC3NzaC... your-public-key ...
# Allow password authentication via SSH
ssh_pwauth: true
To generate a password hash for the passwd field, run:
mkpasswd --method=SHA-512 --rounds=4096
Creating a file with required permissions
Example cloud-config for creating a file with specified access permissions:
#cloud-config
# List of files to create
write_files:
- path: /opt/scripts/start.sh # File path
content: | # File content
#!/bin/bash
echo "Starting application"
owner: cloud:cloud # File owner (user:group)
permissions: '0755' # Access permissions (octal format)
Configuring disk and filesystem
Example cloud-config for disk partitioning, filesystem creation, and mounting:
#cloud-config
# Disk partitioning setup
disk_setup:
/dev/sdb: # Disk device
table_type: gpt # Partition table type (gpt or mbr)
layout: true # Automatically create partitions
overwrite: false # Do not overwrite existing partitions
# Filesystem setup
fs_setup:
- label: data # Filesystem label
filesystem: ext4 # Filesystem type
device: /dev/sdb1 # Partition device
partition: auto # Automatically detect partition
# Filesystem mounting
mounts:
# [device, mount_point, fs_type, options, dump, pass]
- ["/dev/sdb1", "/mnt/data", "ext4", "defaults", "0", "2"]
Configuring network interfaces for additional networks
The settings described in this section apply only to additional networks. The main network (Main) is configured automatically via cloud-init and does not require manual configuration.
If additional networks are connected to a virtual machine, configure them manually via cloud-init: create configuration files in write_files and apply the settings in runcmd.
For more information on connecting additional networks to a virtual machine, see Additional network interfaces.
For systemd-networkd
Example cloud-config for distributions that use systemd-networkd (Debian, CoreOS, and others):
#cloud-config
write_files:
- path: /etc/systemd/network/10-eth1.network
content: |
[Match]
Name=eth1
[Network]
Address=192.168.1.10/24
Gateway=192.168.1.1
DNS=8.8.8.8
runcmd:
- systemctl restart systemd-networkd
For Netplan (Ubuntu)
Example cloud-config for Ubuntu and other systems that use Netplan:
#cloud-config
write_files:
- path: /etc/netplan/99-custom.yaml
content: |
network:
version: 2
ethernets:
eth1:
addresses:
- 10.0.0.5/24
gateway4: 10.0.0.1
nameservers:
addresses: [8.8.8.8]
eth2:
dhcp4: true
runcmd:
- netplan apply
For ifcfg (RHEL/CentOS)
Example cloud-config for RHEL-compatible distributions that use the ifcfg scheme and NetworkManager:
#cloud-config
write_files:
- path: /etc/sysconfig/network-scripts/ifcfg-eth1
content: |
DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes
IPADDR=192.168.1.10
PREFIX=24
GATEWAY=192.168.1.1
DNS1=8.8.8.8
runcmd:
- nmcli connection reload
- nmcli connection up eth1
For Alpine Linux
Example cloud-config for distributions that use the traditional /etc/network/interfaces format (Alpine and similar):
#cloud-config
write_files:
- path: /etc/network/interfaces
append: true
content: |
auto eth1
iface eth1 inet static
address 192.168.1.10
netmask 255.255.255.0
gateway 192.168.1.1
runcmd:
- /etc/init.d/networking restart
How to use Ansible to provision virtual machines?
Ansible is an automation tool for running tasks on remote servers over SSH. This example shows how to use Ansible with virtual machines in the demo-app project.
The example assumes that:
demo-appnamespace contains a VM namedfrontend.- VM has a
clouduser with SSH access. - Private SSH key on the machine where Ansible runs is stored in
/home/user/.ssh/id_rsa.
-
Create an
inventory.yamlfile:--- all: vars: ansible_ssh_common_args: '-o ProxyCommand="d8 v port-forward --stdio=true %h %p"' # Default user for SSH access. ansible_user: cloud # Path to private key. ansible_ssh_private_key_file: /home/user/.ssh/id_rsa hosts: # Host name in the format <VM name>.<namespace>. frontend.demo-app: -
Check the virtual machine
uptime:ansible -m shell -a "uptime" -i inventory.yaml all # frontend.demo-app | CHANGED | rc=0 >> # 12:01:20 up 2 days, 4:59, 0 users, load average: 0.00, 0.00, 0.00
If you do not want to use an inventory file, pass all parameters on the command line:
ansible -m shell -a "uptime" \
-i "frontend.demo-app," \
-e "ansible_ssh_common_args='-o ProxyCommand=\"d8 v port-forward --stdio=true %h %p\"'" \
-e "ansible_user=cloud" \
-e "ansible_ssh_private_key_file=/home/user/.ssh/id_rsa" \
all
How to automatically generate inventory for Ansible?
The d8 v ansible-inventory command requires d8 v0.27.0 or higher.
The command works only for virtual machines that have the main cluster network (Main) connected.
Instead of manually creating an inventory file, you can use the d8 v ansible-inventory command, which automatically generates an Ansible inventory from virtual machines in the specified namespace. The command is compatible with the ansible inventory script interface.
The command includes only virtual machines with assigned IP addresses in the Running state. Host names are formatted as <vmname>.<namespace> (for example, frontend.demo-app).
-
Optionally set host variables via annotations (for example, the SSH user):
d8 k -n demo-app annotate vm frontend provisioning.virtualization.deckhouse.io/ansible_user="cloud" -
Run Ansible with a dynamically generated inventory:
ANSIBLE_INVENTORY_ENABLED=yaml ansible -m shell -a "uptime" all -i <(d8 v ansible-inventory -n demo-app -o yaml)
The <(...) construct is necessary because Ansible expects a file or script as the source of the host list. Simply specifying the command in quotes will not work — Ansible will try to execute the string as a script. The <(...) construct passes the command output as a file that Ansible can read.
-
Or save the inventory to a file and run the check:
d8 v ansible-inventory --list -o yaml -n demo-app > inventory.yaml ansible -m shell -a "uptime" -i inventory.yaml all
How to redirect traffic to a virtual machine?
The virtual machine runs in a Kubernetes cluster, so directing network traffic to it works like routing traffic to pods. To route traffic to a virtual machine, use the standard Kubernetes mechanism — the Service resource, which selects targets using a label selector.
-
Create a service with the required settings.
For example, consider a virtual machine with the label
vm: frontend-0, an HTTP service exposed on ports 80 and 443, and SSH access on port 22:apiVersion: virtualization.deckhouse.io/v1alpha2 kind: VirtualMachine metadata: name: frontend-0 namespace: dev labels: vm: frontend-0 spec: ... -
To route network traffic to the virtual machine’s ports, create the following Service:
-
To route network traffic to the virtual machine’s ports, create the following service:
This service listens on ports 80 and 443 and forwards traffic to the target virtual machine’s ports 80 and 443. SSH access from outside the cluster is provided on port 2211.
apiVersion: v1
kind: Service
metadata:
name: frontend-0-svc
namespace: dev
spec:
type: LoadBalancer
ports:
- name: ssh
port: 2211
protocol: TCP
targetPort: 22
- name: http
port: 80
protocol: TCP
targetPort: 80
- name: https
port: 443
protocol: TCP
targetPort: 443
selector:
vm: frontend-0
Platform management
How to increase the DVCR size?
The DVCR volume size is set in the virtualization module ModuleConfig (spec.settings.dvcr.storage.persistentVolumeClaim.size). The new value must be greater than the current one.
-
Check the current DVCR size:
d8 k get mc virtualization -o jsonpath='{.spec.settings.dvcr.storage.persistentVolumeClaim}'Example output:
{"size":"58G","storageClass":"linstor-thick-data-r1"} -
Increase
sizeusingpatch(set the value you need):d8 k patch mc virtualization \ --type merge -p '{"spec": {"settings": {"dvcr": {"storage": {"persistentVolumeClaim": {"size":"59G"}}}}}}'Example output:
moduleconfig.deckhouse.io/virtualization patched -
Verify that ModuleConfig shows the new size:
d8 k get mc virtualization -o jsonpath='{.spec.settings.dvcr.storage.persistentVolumeClaim}'Example output:
{"size":"59G","storageClass":"linstor-thick-data-r1"} -
Check the current DVCR status:
d8 k get pvc dvcr -n d8-virtualizationExample output:
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE dvcr Bound pvc-6a6cedb8-1292-4440-b789-5cc9d15bbc6b 57617188Ki RWO linstor-thick-data-r1 7d
How to restore the cluster if images from registry.deckhouse.io cannot be pulled after a license change?
After a license change on a cluster with containerd v1 and removal of the outdated license, images from registry.deckhouse.io may stop being pulled. Nodes then retain the outdated configuration file /etc/containerd/conf.d/dvcr.toml, which is not removed automatically. Because of it, the registry module does not start, and without it DVCR does not work.
Applying a NodeGroupConfiguration (NGC) manifest removes the file on the nodes. After the registry module starts, delete the manifest, since this is a one-time fix.
-
Save the manifest to a file (for example,
containerd-dvcr-remove-old-config.yaml):apiVersion: deckhouse.io/v1alpha1 kind: NodeGroupConfiguration metadata: name: containerd-dvcr-remove-old-config.sh spec: weight: 32 # Must be in range 32–90 nodeGroups: ["*"] bundles: ["*"] content: | # Copyright 2023 Flant JSC # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # http://www.apache.org/licenses/LICENSE-2.0 # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. rm -f /etc/containerd/conf.d/dvcr.toml -
Apply the saved manifest:
d8 k apply -f containerd-dvcr-remove-old-config.yaml -
Verify that the
registrymodule is running:d8 k -n d8-system -o yaml get secret registry-state | yq -C -P '.data | del .state | map_values(@base64d) | .conditions = (.conditions | from_yaml)'Example output when the
registrymodule has started successfully:conditions: # ... - lastTransitionTime: "..." message: "" reason: "" status: "True" type: Ready -
Delete the one-time NodeGroupConfiguration manifest:
d8 k delete -f containerd-dvcr-remove-old-config.yaml
For more information on migration, see Migrating container runtime to containerd v2.