How do I find out all Deckhouse parameters?
All the Deskhouse settings (including module parameters) are stored in cluster scoped ModuleConfig resources and custom resources. The global
ModuleConfig contains global Deckhouse settings. Read more in the documentation.
To get list of all ModuleConfigs:
kubectl get mc
To view global Deckhouse settings:
kubectl get mc global -o yaml
How do I find the documentation for the version installed?
Documentation in the cluster is available when the deckhouse-web module is enabled (it is enabled by default except the
Minimal
bundle).
The documentation for the Deckhouse version running in the cluster is available at deckhouse.<cluster_domain>
, where <cluster_domain>
is the DNS name that matches the template defined in the global.modules.publicDomainTemplate
parameter.
How do I set the desired release channel?
Change (set) the releaseChannel
parameter in the deckhouse
module configuration to automatically switch to another release channel.
It will activate the mechanism of automatic stabilization of the release channel.
Here is an example of the deckhouse
module configuration with the Stable
release channel:
apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
name: deckhouse
spec:
version: 1
settings:
releaseChannel: Stable
How do I disable automatic updates?
To completely disable the Deckhouse update mechanism, remove the releaseChannel
parameter in the `deckhouse’ module configuration.
In this case, Deckhouse does not check for updates and even doesn’t apply patch releases.
It is highly not recommended to disable automatic updates! It will block updates to patch releases that may contain critical vulnerabilities and bugs fixes.
How does automatic Deckhouse update work?
Every minute Deckhouse checks a new release appeared in the release channel specified by the releaseChannel
parameter.
When a new release appears on the release channel, Deckhouse downloads it and creates CustomResource DeckhouseRelease
.
After creating a DeckhouseRelease
CR in a cluster, Deckhouse updates the deckhouse
Deployment and sets the image tag to a specified release tag according to selected update mode and update windows (automatic at any time by default).
To get list and status of all releases use the following command:
kubectl get deckhousereleases
Patch releases (e.g., an update from version
1.30.1
to version1.30.2
) ignore update windows settings and apply as soon as they are available.
Change the release channel
- When switching to a more stable release channel (e.g., from
Alpha
toEarlyAccess
), Deckhouse downloads release data from the release channel (theEarlyAccess
release channel in the example) and compares it with the existingDeckhouseReleases
:- Deckhouse deletes later releases (by semver) that have not yet been applied (with the
Pending
status). - if the latest releases have been already Deployed, then Deckhouse will hold the current release until a later release appears on the update channel (on the
EarlyAccess
release channel in the example).
- Deckhouse deletes later releases (by semver) that have not yet been applied (with the
- When switching to a less stable release channel (e.g., from
EarlyAcess
toAlpha
), the following actions take place:- Deckhouse downloads release data from the release channel (the
Alpha
release channel in the example) and compares it with the existingDeckhouseReleases
. - Then Deckhouse performs the update according to the update parameters.
- Deckhouse downloads release data from the release channel (the
How do I run Deckhouse on a particular node?
Set the nodeSelector
parameter of the deckhouse
module and avoid setting tolerations
. The necessary values will be assigned to the tolerations
parameter automatically.
You should also avoid using CloudEphemeral nodes. Otherwise, a situation may occur when the target node is not in the cluster and node ordering for some reason is impossible.
Here is an example of the module configuration:
apiVersion: deckhouse.io/v1alpha1
kind: ModuleConfig
metadata:
name: deckhouse
spec:
version: 1
settings:
nodeSelector:
node-role.deckhouse.io/deckhouse: ""
How do I configure Deckhouse to use a third-party registry?
Deckhouse can be configured to work with a third-party registry (e.g., a proxy registry inside private environments).
Configuring
Define the following parameters in the InitConfiguration
resource:
imagesRepo: <PROXY_REGISTRY>/<DECKHOUSE_REPO_PATH>/<DECKHOUSE_REVISION>
. The path to the Deckhouse image in the third-party registry matching the edition used (CE/EE), for exampleimagesRepo: registry.deckhouse.io/deckhouse/ce
;registryDockerCfg: <BASE64>
. Base64-encoded auth credentials of the third-party registry.
Use the following registryDockerCfg
if anonymous access to Deckhouse images is allowed in the third-party registry:
{"auths": { "<PROXY_REGISTRY>": {}}}
registryDockerCfg
must be Base64-encoded.
Use the following registryDockerCfg
if authentication is required to access Deckhouse images in the third-party registry:
{"auths": { "<PROXY_REGISTRY>": {"username":"<PROXY_USERNAME>","password":"<PROXY_PASSWORD>","auth":"<AUTH_BASE64>"}}}
<PROXY_USERNAME>
— auth username for<PROXY_REGISTRY>
.<PROXY_PASSWORD>
— auth password for<PROXY_REGISTRY>
.<PROXY_REGISTRY>
— registry address:<HOSTNAME>[:PORT]
.<AUTH_BASE64>
— Base64-encoded<PROXY_USERNAME>:<PROXY_PASSWORD>
auth string.
registryDockerCfg
must be Base64-encoded.
The InitConfiguration
resource provides two more parameters for non-standard third-party registry configurations:
registryCA
- root CA certificate to validate the third-party registry’s HTTPS certificate (if self-signed certificates are used);registryScheme
- registry scheme (HTTP
orHTTPS
). The default value isHTTPS
.
Tips for configuring the third-party registry
Note! Deckhouse only supports Bearer authentication for registries.
Nexus
Requirements
The following requirements must be met if the Nexus repository manager is used:
Docker Bearer Token Realm
must be enabled.- Docker proxy repository must be pre-created.
Allow anonymous docker pull
must be enabled.- Access control must be configured as follows:
- The Nexus role with the
nx-repository-view-docker-<repo>-browse
andnx-repository-view-docker-<repo>-read
permissions must be created. - The Nexus user must be created with the above role granted.
- The Nexus role with the
Maximum metadata age
for the created repository must be set to 0.
Configuration
-
Enable
Docker Bearer Token Realm
: -
Create a docker proxy repository pointing to the Deckhouse registry:
- Fill in the fields on the Create page as follows:
Name
must contain the name of the repository you created earlier, e.g.,d8-proxy
.Repository Connectors / HTTP
orRepository Connectors / HTTPS
must contain a dedicated port for the created repository, e.g.,8123
or other.Allow anonymous docker pull
must be enabled for the Bearer token authentication to work. Note, however, that anonymous access won’t work unless it is explicitly enabled in Settings -> Security -> Anonymous Access and theanonymous
user has been granted access rights to the created repository.Remote storage
must be set tohttps://registry.deckhouse.io/
.- You can disable
Auto blocking enabled
andNot found cache enabled
for debugging purposes, otherwise they must be enabled. Maximum Metadata Age
must be set to 0.Authentication
must be enabled if you plan to use Deckhouse Enterprise Edition and the related fields must be set as follows:Authentication Type
must be set toUsername
.Username
must be set tolicense-token
.Password
must contain your license key for Deckhouse Enterprise Edition.
- Configure Nexus access control to allow Nexus access to the created repository:
- Create a Nexus role with the
nx-repository-view-docker-<repo>-browse
andnx-repository-view-docker-<repo>-read
permissions. - Create a Nexus user with the role above granted.
- Create a Nexus role with the
Harbor
You need to use the Proxy Cache feature of a Harbor.
- Create a Registry:
Administration -> Registries -> New Endpoint
.Provider
:Docker Registry
.Name
— specify any of your choice.Endpoint URL
:https://registry.deckhouse.io
.- Specify the
Access ID
andAccess Secret
if you use Deckhouse Enterprise Edition; otherwise, leave them blank.
- Create a new Project:
Projects -> New Project
.Project Name
will be used in the URL. You can choose any name, for example,d8s
.Access Level
:Public
.Proxy Cache
— enable and choose the Registry, created in the previous step.
Thus, Deckhouse images will be available at https://your-harbor.com/d8s/deckhouse/{d8s-edition}:{d8s-version}
.
Manually upload images to an air-gapped registry
-
Download the script on a host that have access to
registry.deckhouse.io
(for the script to work, you need installedDocker
and jq):curl -fsSL -o d8-pull.sh https://raw.githubusercontent.com/deckhouse/deckhouse/main/tools/release/d8-pull.sh chmod 700 d8-pull.sh
Note! Use the following commands if you want to upload images of Deckhouse v1.44 or older:
curl -fsSL -o d8-pull.sh https://raw.githubusercontent.com/deckhouse/deckhouse/v1.44.4/tools/release/d8-pull.sh chmod 700 d8-pull.sh
-
Pull images using the
d8-pull.sh
script.Example of pulling Deckhouse EE v1.45.5 images to the
/your/output-dir/
directory:./d8-pull.sh --license <DECKHOUSE_LICENSE_KEY> --release v1.45.5 --output-dir /your/output-dir/
For Deckhouse CE specify the
--edition ce
parameter and omit the--license
parameter. -
Upload the directory from the previous step to a host with access to an air-gapped registry.
-
Download script on the host:
curl -fsSL -o d8-push.sh https://raw.githubusercontent.com/deckhouse/deckhouse/main/tools/release/d8-push.sh chmod 700 d8-push.sh
Note! Use the following commands if you want to upload images of Deckhouse v1.44 or older:
curl -fsSL -o d8-push.sh https://raw.githubusercontent.com/deckhouse/deckhouse/v1.44.4/tools/release/d8-push.sh chmod 700 d8-push.sh
-
Push the images using the
d8-push.sh
script to an air-gapped registry.Example of pushing images from the
/your/source-dir/
directory:./d8-push.sh --source-dir /your/source-dir/ --path your.private.registry.com/deckhouse --username <USERNAME> --password <PASSWORD>
Note! Please, refer to the documentation of your registry to properly specify the
--path
value. For example, forHarbor
it can beharbor.registry.com/deckhouse/deckhouse
, but notharbor.registry.com/deckhouse
.
How to bootstrap a cluster and run Deckhouse without the usage of release channels?
This case is only valid if you don’t have release channel images in your air-gapped registry.
- If you want to install Deckhouse with automatic updates disabled:
- Use the tag of the installer image of the corresponding version. For example, use the image
your.private.registry.com/deckhouse/install:v1.44.3
, if you want to install releasev1.44.3
. - Set the corresponding version number in the deckhouse.devBranch parameter of the
InitConfiguration
resource. - Do not set the deckhouse.releaseChannel parameter of the
InitConfiguration
resource.
- Use the tag of the installer image of the corresponding version. For example, use the image
- If you want to disable automatic updates for an already installed Deckhouse (including patch release updates), then delete the releaseChannel parameter from the
deckhouse
module configuration.
Using a proxy server
Setting up a proxy server
- Prepare the VM for setting up the proxy. The machine must be accessible to the nodes that will use it as a proxy and be connected to the Internet.
-
Install Squid on the server (here and further examples for Ubuntu):
apt-get install squid
-
Create a config file:
cat <<EOF > /etc/squid/squid.conf auth_param basic program /usr/lib/squid3/basic_ncsa_auth /etc/squid/passwords auth_param basic realm proxy acl authenticated proxy_auth REQUIRED http_access allow authenticated # Choose the port you want. Below we set it to default 3128. http_port 3128
-
Create a user for proxy-server authentication:
Example for the user
test
with the passwordtest
(be sure to change):echo "test:$(openssl passwd -crypt test)" >> /etc/squid/passwords
-
Start squid and enable the system to start it up automatically:
systemctl restart squid systemctl enable squid
Configuring proxy usage in Deckhouse
Use the proxy parameter of the ClusterConfiguration
resource to configure proxy usage.
An example:
apiVersion: deckhouse.io/v1
kind: ClusterConfiguration
clusterType: Cloud
cloud:
provider: OpenStack
prefix: main
podSubnetCIDR: 10.111.0.0/16
serviceSubnetCIDR: 10.222.0.0/16
kubernetesVersion: "1.23"
cri: "Containerd"
clusterDomain: "cluster.local"
proxy:
httpProxy: "http://user:password@proxy.company.my:3128"
httpsProxy: "https://user:password@proxy.company.my:8443"
How do I switch a running Deckhouse cluster to use a third-party registry?
To switch the Deckhouse cluster to using a third-party registry, follow these steps:
- Update the
image
field in thed8-system/deckhouse
deployment to contain the address of the Deckhouse image in the third-party-registry; - Download script to the master node and run it with parameters for a new registry.
- Example:
curl -fsSL -o change-registry.sh https://raw.githubusercontent.com/deckhouse/deckhouse/main/tools/change-registry.sh chmod 700 change-registry.sh ./change-registry.sh --registry-url https://my-new-registry/deckhouse --user my-user --password my-password
- If the registry uses a self-signed certificate, put the root CA certificate that validates the registry’s HTTPS certificate to file
ca.crt
near the script and add the--ca-file ca.crt
option to the script.
- Wait for the Deckhouse Pod to become
Ready
. Restart Deckhouse Pod if it will be inImagePullBackoff
state. - Wait for bashible to apply the new settings on the master node. The bashible log on the master node (
journalctl -u bashible
) should contain the messageConfiguration is in sync, nothing to do
. - If you want to disable Deckhouse automatic updates, remove the
releaseChannel
parameter from thedeckhouse
module configuration. -
Check if there are Pods with original registry in cluster (if there are — restart them):
kubectl get pods -A -o json | jq '.items[] | select(.spec.containers[] | select((.image | contains("deckhouse.io")))) | .metadata.namespace + "\t" + .metadata.name' -r
How to switch Deckhouse EE to CE?
The instruction implies using the public address of the container registry:
registry.deckhouse.io
. If you use a different container registry address, change the commands or use the instruction for switching Deckhouse to using a third-party registry.
To switch Deckhouse Enterprise Edition to Community Edition, follow these steps:
-
Make sure that the modules used in the cluster are supported in Deckhouse CE. Disable modules that are not supported in Deckhouse CE.
Please note that Deckhouse CE does not support cloud clusters on OpenStack and VMware vSphere.
-
Run the following command:
bash -c "$(curl -Ls https://raw.githubusercontent.com/deckhouse/deckhouse/main/tools/change-registry.sh)" -- --registry-url https://registry.deckhouse.io/deckhouse/ce && \ kubectl -n d8-system set image deployment/deckhouse deckhouse=$(kubectl -n d8-system get deployment deckhouse -o jsonpath='{.spec.template.spec.containers[?(@.name=="deckhouse")].image}' | awk -F: '{print "registry.deckhouse.io/deckhouse/ce:" $2}')
-
Wait for the Deckhouse Pod to become
Ready
:kubectl -n d8-system get po -l app=deckhouse
-
Restart Deckhouse Pod if it will be in
ImagePullBackoff
state:kubectl -n d8-system delete po -l app=deckhouse
-
Wait for Deckhouse to restart and to complete all tasks in the queue:
kubectl -n d8-system exec deploy/deckhouse -- deckhouse-controller queue main | grep status:
Example of output when there are still jobs in the queue (
length 38
):# kubectl -n d8-system exec deploy/deckhouse -- deckhouse-controller queue main | grep status: Queue 'main': length 38, status: 'run first task'
Example of output when the queue is empty (
length 0
):# kubectl -n d8-system exec deploy/deckhouse -- deckhouse-controller queue main | grep status: Queue 'main': length 0, status: 'waiting for task 0s'
-
On the master node, check the application of the new settings.
The message
Configuration is in sync, nothing to do
should appear in thebashible
systemd service log on the master node.An example::
# journalctl -u bashible -n 5 Jan 12 12:38:20 demo-master-0 bashible.sh[868379]: Configuration is in sync, nothing to do. Jan 12 12:38:20 demo-master-0 systemd[1]: bashible.service: Deactivated successfully. Jan 12 12:39:18 demo-master-0 systemd[1]: Started Bashible service. Jan 12 12:39:19 demo-master-0 bashible.sh[869714]: Configuration is in sync, nothing to do. Jan 12 12:39:19 demo-master-0 systemd[1]: bashible.service: Deactivated successfully.
-
Check if there are any Pods left in the cluster with the Deckhouse EE registry address:
kubectl get pods -A -o json | jq '.items[] | select(.spec.containers[] | select((.image | contains("deckhouse.io/deckhouse/ee")))) | .metadata.namespace + "\t" + .metadata.name' -r | sort | uniq
Sometimes, some static Pods may remain running (for example,
kubernetes-api-proxy-*
). This is due to the fact that kubelet does not restart the Pod despite changing the corresponding manifest, because the image used is the same for the Deckhouse CE and EE editions. To make sure that the corresponding manifests have also been changed, run the following command on any master node:grep -ri 'deckhouse.io/deckhouse/ee' /etc/kubernetes | grep -v backup
The output of the command should be empty.
How to switch Deckhouse CE to EE?
You will need a valid license key (you can request a trial license key if necessary).
The instruction implies using the public address of the container registry:
registry.deckhouse.io
. If you use a different container registry address, change the commands or use the instruction for switching Deckhouse to using a third-party registry.
To switch Deckhouse Community Edition to Enterprise Edition, follow these steps:
-
Run the following command:
LICENSE_TOKEN=<PUT_YOUR_LICENSE_TOKEN_HERE> bash -c "$(curl -Ls https://raw.githubusercontent.com/deckhouse/deckhouse/main/tools/change-registry.sh)" -- --user license-token --password $LICENSE_TOKEN --registry-url https://registry.deckhouse.io/deckhouse/ee && \ kubectl -n d8-system set image deployment/deckhouse deckhouse=$(kubectl -n d8-system get deployment deckhouse -o jsonpath='{.spec.template.spec.containers[?(@.name=="deckhouse")].image}' | awk -F: '{print "registry.deckhouse.io/deckhouse/ee:" $2}')
-
Wait for the Deckhouse Pod to become
Ready
:kubectl -n d8-system get po -l app=deckhouse
-
Restart Deckhouse Pod if it will be in
ImagePullBackoff
state:kubectl -n d8-system delete po -l app=deckhouse
-
Wait for Deckhouse to restart and to complete all tasks in the queue:
kubectl -n d8-system exec deploy/deckhouse -- deckhouse-controller queue main | grep status:
Example of output when there are still jobs in the queue (
length 38
):# kubectl -n d8-system exec deploy/deckhouse -- deckhouse-controller queue main | grep status: Queue 'main': length 38, status: 'run first task'
Example of output when the queue is empty (
length 0
):# kubectl -n d8-system exec deploy/deckhouse -- deckhouse-controller queue main | grep status: Queue 'main': length 0, status: 'waiting for task 0s'
-
On the master node, check the application of the new settings.
The message
Configuration is in sync, nothing to do
should appear in thebashible
systemd service log on the master node.An example:
# journalctl -u bashible -n 5 Jan 12 12:38:20 demo-master-0 bashible.sh[868379]: Configuration is in sync, nothing to do. Jan 12 12:38:20 demo-master-0 systemd[1]: bashible.service: Deactivated successfully. Jan 12 12:39:18 demo-master-0 systemd[1]: Started Bashible service. Jan 12 12:39:19 demo-master-0 bashible.sh[869714]: Configuration is in sync, nothing to do. Jan 12 12:39:19 demo-master-0 systemd[1]: bashible.service: Deactivated successfully.
-
Check if there are any Pods left in the cluster with the Deckhouse CE registry address:
kubectl get pods -A -o json | jq '.items[] | select(.spec.containers[] | select((.image | contains("deckhouse.io/deckhouse/ce")))) | .metadata.namespace + "\t" + .metadata.name' -r | sort | uniq
Sometimes, some static Pods may remain running (for example,
kubernetes-api-proxy-*
). This is due to the fact that kubelet does not restart the Pod despite changing the corresponding manifest, because the image used is the same for the Deckhouse CE and EE editions. To make sure that the corresponding manifests have also been changed, run the following command on any master node:grep -ri 'deckhouse.io/deckhouse/ce' /etc/kubernetes | grep -v backup
The output of the command should be empty.
How do I change the configuration of a cluster?
The general cluster parameters are stored in the ClusterConfiguration structure.
To change the general cluster parameters, run the command:
kubectl -n d8-system exec -ti deploy/deckhouse -- deckhouse-controller edit cluster-configuration
After saving the changes, Deckhouse will bring the cluster configuration to the state according to the changed configuration. Depending on the size of the cluster, this may take some time.
How do I change the configuration of a cloud provider in a cluster?
Cloud provider setting of a cloud of hybrid cluster are stored in the <PROVIDER_NAME>ClusterConfiguration
structure, where <PROVIDER_NAME>
— name/code of the cloud provider. E.g., for an OpenStack provider, the structure will be called OpenStackClusterConfiguration.
Regardless of the cloud provider used, its settings can be changed using the command:
kubectl -n d8-system exec -ti deploy/deckhouse -- deckhouse-controller edit provider-cluster-configuration
How do I change the configuration of a static cluster?
Settings of a static cluster are stored in the StaticClusterConfiguration structure.
To change the settings of a static cluster, run the command:
kubectl -n d8-system exec -ti deploy/deckhouse -- deckhouse-controller edit static-cluster-configuration
How do I upgrade the Kubernetes version in a cluster?
To upgrade the Kubernetes version in a cluster change the kubernetesVersion parameter in the ClusterConfiguration structure by making the following steps:
-
Run the command:
kubectl -n d8-system exec -ti deploy/deckhouse -- deckhouse-controller edit cluster-configuration
- Change the
kubernetesVersion
field. - Save the changes. Cluster nodes will start updating sequentially.
- Wait for the update to finish. You can track the progress of the update using the
kubectl get no
command. The update is completed when the new version appears in the command’s output for each cluster node in theVERSION
column.