Application templates are rendered and deployed by Nelm. Nelm extends the standard Helm behavior with annotations that control deployment order, resource lifecycle, readiness tracking, and log output. This page covers the most commonly used annotations.
Deployment stages
Nelm processes a deployment in three stages. Different annotation groups affect different stages.
Render
Templates are evaluated with the current values. Deckhouse Platform (DP) renders the templates with access to the cluster: .Capabilities reflects the API server, and lookup returns live objects. At this stage:
- Template functions (
include,tpl,lookup, and others) are executed. - All resources are rendered and stored in the release, regardless of
werf.io/deploy-on.
Plan
Nelm connects to the cluster, reads current resource state, and runs a dry-run Server-Side Apply to compute the exact diff. It then builds a DAG of operations — which resources to create, update, or delete, and in what order.
At this stage:
werf.io/deploy-ondetermines for which operations (install,upgrade, and so on) and in which stage the resource is deployed.- Order and dependency annotations (
werf.io/weight,werf.io/deploy-dependency-*,*.external-dependency.werf.io/*) shape the DAG. - Lifecycle annotations (
werf.io/ownership,werf.io/delete-policy,werf.io/delete-propagation) determine which operations enter the plan.
Dry-run SSA means the diff is computed by the API server — defaulting, admission plugins, and mutating webhooks are all applied.
Apply
The DAG is executed: resources are created, updated, or deleted in dependency order, with parallelism where no dependencies exist. Readiness tracking and log streaming run concurrently.
At this stage:
- Tracking annotations (
werf.io/track-termination-mode,werf.io/fail-mode,werf.io/failures-allowed-per-replica,werf.io/no-activity-timeout) control waiting behavior. - Log annotations control what is printed during deployment.
Common use cases
Job that must run before the main application
Database migration before the application starts:
apiVersion: batch/v1
kind: Job
metadata:
name: d8a-{{ .Application.Instance.Name }}-db-migrate
annotations:
werf.io/delete-policy: before-creation
spec: ...
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: d8a-{{ .Application.Instance.Name }}-app
annotations:
werf.io/deploy-dependency-migrate: state=ready,kind=Job,name=d8a-{{ .Application.Instance.Name }}-db-migrate
spec: ...
The annotation werf.io/delete-policy: before-creation makes Nelm recreate the Job on every deployment, so the migration runs each time. By default, Nelm recreates a Job only when an immutable field of the Job changes. The Deployment waits for ready, that is, for the Job to complete successfully.
Preserve a resource across uninstall or chart removal
A PVC with database data that must not be deleted even when the Application is uninstalled:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: d8a-{{ .Application.Instance.Name }}-postgres-data
annotations:
helm.sh/resource-policy: keep
spec:
accessModes: [ReadWriteOnce]
resources:
requests:
storage: 50Gi
storageClassName: gp3
The annotation helm.sh/resource-policy: keep prevents the PVC from being deleted on uninstall or when removed from the chart. It doesn’t protect against d8 k delete pvc. To keep the data in this case, use a StorageClass with reclaimPolicy: Retain: then the PersistentVolume outlives the PVC.
Resource shared between releases
A TLS Secret used by multiple charts should not disappear when any one of them is uninstalled:
apiVersion: v1
kind: Secret
metadata:
name: d8a-{{ .Application.Instance.Name }}-shared-tls
annotations:
werf.io/ownership: anyone
type: kubernetes.io/tls
data: ...
The annotation werf.io/ownership: anyone tells Nelm it is not the sole owner, so it skips deletion on uninstall. DP still deletes the Secret together with the Application if its kind is listed in orphanResources. To keep the Secret, add helm.sh/resource-policy: keep.
Wait for a resource created by an operator
Deploy only after cert-manager creates the Certificate object:
apiVersion: apps/v1
kind: Deployment
metadata:
name: d8a-{{ .Application.Instance.Name }}-app
annotations:
cert.external-dependency.werf.io/resource: certificates.v1.cert-manager.io/d8a-{{ .Application.Instance.Name }}-tls
spec: ...
Nelm waits until the Certificate object exists before creating the Deployment. Nelm doesn’t wait until the certificate is issued.
Non-critical component that must not fail the deploy
A metrics DaemonSet whose unavailability should not block the release:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: d8a-{{ .Application.Instance.Name }}-metrics-agent
annotations:
werf.io/fail-mode: IgnoreAndContinueDeployProcess
werf.io/track-termination-mode: NonBlocking
spec: ...
Nelm will not wait for readiness and will not fail on timeout.
Resource rendered only on first install
An init Job that runs only during install, not upgrade:
apiVersion: batch/v1
kind: Job
metadata:
name: d8a-{{ .Application.Instance.Name }}-init-data
annotations:
werf.io/deploy-on: install
werf.io/ownership: anyone # CRITICAL
spec: ...
Without werf.io/ownership: anyone, on upgrade the resource renders as absent and the owning release removes it. anyone prevents that. Nelm doesn’t delete such a Job on uninstall either. To make DP delete it together with the Application, list Job in orphanResources.
Slow-starting resource with a large image
StatefulSet with a large Docker image (ML models, Elasticsearch with pre-loaded indices):
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: d8a-{{ .Application.Instance.Name }}-elasticsearch
annotations:
werf.io/no-activity-timeout: 20m
werf.io/failures-allowed-per-replica: "3"
werf.io/show-service-messages: "true"
spec: ...
Used annotations:
no-activity-timeout: 20m: 20 minutes without events before Nelm considers it timed out. Use this for slow image pulls or long initialization.failures-allowed-per-replica: "3": Allows up to 3 tracking errors per replica, for example, container restarts or image pull errors, before the deployment fails. Use for flapping init dependencies.show-service-messages: "true": Emit Kubernetes Events in deploy output (useful for diagnosingImagePullBackOff, scheduling failures, OOM events).
Ordering and dependencies
Weights
The annotation werf.io/weight groups resources: equal weights deploy in parallel; different weights deploy sequentially in ascending order. The weight is ignored for a resource with a werf.io/deploy-dependency-* annotation.
metadata:
annotations:
# Deploys before default-weight (0) resources.
werf.io/weight: "-10"
Direct dependencies
The annotation werf.io/deploy-dependency-<id> waits for a specific resource in the same release to reach a specific state before deploying the annotated resource.
metadata:
annotations:
werf.io/deploy-dependency-db: state=ready,kind=StatefulSet,name=d8a-{{ .Application.Instance.Name }}-postgres
werf.io/deploy-dependency-migrations: state=present,kind=Job,name=d8a-{{ .Application.Instance.Name }}-db-migrate
Dependency states:
ready: Resource is in a ready state (for example,availableReplicas == replicasfor a Deployment).present: Resource exists in the cluster.
Full format:
werf.io/deploy-dependency-<id>: state=ready|present[,name=<name>][,namespace=<namespace>][,kind=<kind>][,group=<group>][,version=<version>]
This annotation has no effect if the dependency resource is in a different deploy stage (pre/main/post). Cross-stage ordering is already enforced by the stage sequence itself.
External dependencies
The annotation <id>.external-dependency.werf.io/resource waits until a resource outside the release (created by an operator or another release) exists. Nelm doesn’t wait for the readiness of the resource:
metadata:
annotations:
cert.external-dependency.werf.io/resource: certificates.v1.cert-manager.io/d8a-{{ .Application.Instance.Name }}-tls
cert.external-dependency.werf.io/namespace: myapp-production # Namespace of the external resource
Full format:
<id>.external-dependency.werf.io/resource: <kind>[.<version>.<group>]/<name>
Delete dependencies
The annotation werf.io/delete-dependency-<id> means that the annotated resource will be deleted only after the referenced resource becomes absent:
metadata:
annotations:
werf.io/delete-dependency-app: state=absent,kind=Deployment,name=d8a-{{ .Application.Instance.Name }}-app
Lifecycle annotations
helm.sh/resource-policy
The keep value instructs Nelm not to delete the resource on uninstall or when removed from the chart. The resource continues to be updated on install/upgrade as long as it renders. If werf.io/resource-policy is set, Nelm ignores helm.sh/resource-policy. DP doesn’t delete a resource with helm.sh/resource-policy: keep as an orphan resource, even if werf.io/resource-policy is set: only this annotation protects a resource from such deletion.
werf.io/resource-policy
A list of policies:
skip-create: Do not create the resource if it doesn’t exist in the cluster.skip-update: Do not update the resource if it already exists.skip-recreate: Do not recreate the resource when a recreation would otherwise be required.skip-delete(alias:keep): Do not delete the resource.
DP deletes a resource with skip-delete or keep together with the Application if its kind is listed in orphanResources. To keep the resource, also set helm.sh/resource-policy: keep.
werf.io/ownership
Allowed values:
release(default for regular resources): Resource is deleted on uninstall and when absent from the chart. Release metadata annotations are applied.anyone(default for hooks): Resource is not deleted on uninstall. Release metadata annotations are not applied.
Use anyone for resources shared between releases or for resources that should survive their release (init Jobs with werf.io/deploy-on: install).
Resources that stay after uninstall, that is, hooks and resources with werf.io/ownership: anyone, are deleted by DP together with the Application if their kind is listed in orphanResources. To keep such a resource, set helm.sh/resource-policy: keep.
werf.io/deploy-on
Controls on which lifecycle operations the resource is rendered.
werf.io/deploy-on: pre-install,upgrade,post-install
Allowed values:
pre-installinstallpost-installpre-upgradeupgradepost-upgradepre-rollbackrollbackpost-rollbackpre-deletedeletepost-delete
Default for regular resources: install,upgrade,rollback. Hooks are deployed on the events listed in helm.sh/hook.
If a resource is rendered for install but not upgrade, and werf.io/ownership: release is set, the resource will be deleted on upgrade because it is absent from the upgrade render. Set werf.io/ownership: anyone to prevent this.
werf.io/delete-policy
Controls when a resource is deleted relative to the apply operation.
| Value | When |
|---|---|
before-creation |
Always recreate before apply (default for hooks) |
before-creation-if-immutable |
Recreate only on field is immutable error (default for Jobs) |
succeeded |
Delete after successful deploy |
failed |
Delete on readiness-check failure |
Values can be combined with a comma: before-creation,succeeded.
werf.io/delete-propagation
Kubernetes deletion propagation strategy.
| Value | Behavior |
|---|---|
Foreground (default) |
Wait for dependent objects to be deleted |
Background |
Delete resource immediately; dependents deleted asynchronously |
Orphan |
Delete resource; leave dependents |
Tracking annotations
| Annotation | Default | Description |
|---|---|---|
werf.io/track-termination-mode |
WaitUntilResourceReady |
WaitUntilResourceReady or NonBlocking |
werf.io/fail-mode |
FailWholeDeployProcessImmediately |
FailWholeDeployProcessImmediately or IgnoreAndContinueDeployProcess |
werf.io/failures-allowed-per-replica |
1 for Deployment, StatefulSet, and DaemonSet, 0 for other kinds |
Number of tracking errors per replica, for example, container restarts or image pull errors, allowed before the deployment fails. Ignored for Jobs |
werf.io/no-activity-timeout |
4m |
Go duration; timeout on absence of events or status changes |
werf.io/show-service-messages |
false |
Show Kubernetes Events in deploy output |
Log annotations
DP doesn’t collect pod logs when it deploys Applications, so these annotations don’t affect the deployment.
| Annotation | Default | Description |
|---|---|---|
werf.io/skip-logs |
false |
Suppress all pod logs |
werf.io/skip-logs-for-containers |
— | Comma-separated list of containers to suppress |
werf.io/show-logs-only-for-containers |
— | Comma-separated list; only show logs for these containers |
werf.io/show-logs-only-for-number-of-replicas |
1 |
Show logs only for the first N replicas |
werf.io/log-regex |
— | RE2 pattern; show only matching lines |
werf.io/log-regex-skip |
— | RE2 pattern; suppress matching lines |
werf.io/log-regex-for-<container> |
— | Per-container RE2 filter for showing |
werf.io/log-regex-skip-for-<container> |
— | Per-container RE2 filter for suppressing |
Template functions
werf_secret_file
Embed the decrypted content of a file from the secret/ directory:
data:
config.yaml: {{ werf_secret_file "config.yaml" | b64enc }}
DP doesn’t provide a decryption key for Nelm secrets, so Application packages can’t use werf_secret_file, the secret/ directory, or secret-values.yaml.
dump_debug, printf_debug, include_debug, tpl_debug
The functions dump_debug and printf_debug render nothing, and include_debug and tpl_debug render the same output as include and tpl. The functions write to the log only when Nelm debug logging is enabled. DP doesn’t enable it, so in Application packages these functions log nothing.
{{ dump_debug $ }}
{{ printf_debug "replicaCount: %d" .Values.replicaCount }}
{{ include_debug "myapp.labels" . | nindent 4 }}
{{ tpl_debug "{{ .Values.template }}" . }}
Known issues and troubleshooting
-
nullvalues and SSA. Server-Side Apply frequently fails on fields with anullvalue. If.Values.fooisnil, the manifest getsfoo: null. Fix with{{ if .Values.foo }}guards ordefault. -
lookuptiming. If cluster resources change between the Plan and Apply stages, the rendered plan may be stale. Avoidlookupfor critical logic; pass data through values instead. -
Non-deterministic functions.
now,randAlphaNum,uuidv4, and the unsorted output ofkeysproduce different results on every render. DP applies the templates when the rendered manifests change, so such functions cause a redeployment in every reconciliation run. -
werf.io/deploy-dependency-*and stages. The annotation has no effect across deploy stages (pre/main/post). Cross-stage ordering is implicit in the stage sequence. -
Resources that disappear after deployment. DP checks every 4–5 minutes that all regular (non-hook) resources of the release exist and applies the templates again if one is missing (for details, refer to “Resource restoration”). Therefore, a regular Job that is deleted after it completes, for example, because of
werf.io/delete-policy: succeededorttlSecondsAfterFinished, runs again after every check, and a deleted resource that is rendered only on install causes a redeployment after every check. Make such Jobs Helm hooks: DP doesn’t check hooks.