Troubleshooting

Each entry below describes one symptom, its cause and the action that resolves it. The commands assume the release code in the namespace code.

Install fails on a value check

helm upgrade exits before creating any object, and the output ends with a message from the chart’s value checks.

Cause. The values contradict each other or miss a required field. The chart checks them first and prints every finding at once.

Action. Change the values the message names and run the command again. The messages and their fixes:

Load balancer Service has no external address

d8 k -n code get service code-code-loadbalancer shows <pending> in the EXTERNAL-IP column.

Cause. The cluster has no implementation that assigns addresses to Services of type LoadBalancer, or the implementation needs an annotation to pick the address.

Action. Read the Service events:

d8 k -n code describe service code-code-loadbalancer

Add the annotation the implementation expects to global.codeHelm.loadbalancer.annotations, for example metallb.universe.tf/loadBalancerIPs for MetalLB. When the cluster has no such implementation, disable the load balancer with global.codeHelm.loadbalancer.enabled: false, point DNS at the ingress controller and expose SSH with gitlab.gitlab-shell.service.type: NodePort.

Pods stay in ImagePullBackOff

d8 k -n code get pods shows pods in the ImagePullBackOff or ErrImagePull state.

Cause. The image pull Secret code-image-pull-secret, built from global.codeHelm.externalSecrets.registryDockerConfig, holds wrong or expired credentials, or the nodes cannot reach the image registry.

Action. Read the pull error from the pod events:

d8 k -n code describe pod <POD_NAME>

An unauthorized response points to the credentials: encode the .dockerconfigjson again with base64 -w0 and upgrade the release. A connection error points to the network path from the nodes to the registry.

Pods stay in Pending

d8 k -n code get pods shows the Gitaly or Praefect pods in the Pending state.

Cause. No StorageClass satisfies gitlab.gitaly.persistence.storageClass, the cluster has no default StorageClass, or no node has enough CPU and memory for the pod’s requests.

Action. Read the pod events:

d8 k -n code describe pod <POD_NAME>

A FailedScheduling event that names a PersistentVolumeClaim points to the StorageClass; list the classes with d8 k get storageclass. An event that names CPU or memory points to the node resources.

Certificate is not issued

d8 k -n code get certificate shows False in the READY column for code-tls, code-registry-tls or code-pages-tls, and the browser reports an invalid certificate.

Cause. The ClusterIssuer from global.codeHelm.ingress.tls.clusterIssuer does not exist, or the issuer cannot complete the challenge because the DNS name does not resolve to the load balancer yet.

Action. Read the Certificate status and the issuer:

d8 k -n code describe certificate code-tls
d8 k get clusterissuer

Create the ClusterIssuer or name an existing one, and confirm that the DNS records point at the load balancer address. cert-manager retries the issuance on its own once the cause is removed.

Migrations job fails

d8 k -n code get jobs shows the job code-migrations-<suffix> with 0/1 completions, and the web service and Sidekiq pods wait for it.

Cause. PostgreSQL is not reachable at global.psql.host, the password in global.codeHelm.externalSecrets.psqlPassword or in the Secret from global.psql.password.secret is wrong, the role has no rights on the database, or the release skipped a Deckhouse Code version a later migration depends on.

Action. Read the job log:

d8 k -n code logs job/code-migrations-<suffix>

Resolve the connection or the credentials first. When the log names a missing earlier migration, follow the version order in Upgrade.

Praefect pods restart

The code-praefect pods restart, and their logs report a failed connection to PostgreSQL.

Cause. The Praefect database in global.praefect.psql.host, dbName and user does not exist or the password from global.codeHelm.externalSecrets.praefectPassword is wrong. Praefect has its own database and does not use the application one.

Action. Read the pod log:

d8 k -n code logs statefulset/code-praefect

Create the database and the role on the PostgreSQL server, set the password and upgrade the release.

Object storage rejects requests

Uploads and CI/CD artifacts fail, and the web service or Sidekiq logs show an AccessDenied or NoSuchBucket response from the storage.

Cause. The connection in global.codeHelm.externalSecrets.s3 or in the Secret from global.appConfig.object_store.connection.secret holds a wrong endpoint, region or key, or a bucket from global.appConfig.<store>.bucket does not exist.

Action. Compare the connection with the storage console and confirm that every bucket from Requirements exists and the keys have write access to it. For a storage that addresses buckets in the URL path, add path_style: true to the connection.

Helm upgrade times out

helm upgrade exits with Error: UPGRADE FAILED: timed out waiting for the condition.

Cause. A pod or the migrations job did not become ready within the wait time of Helm, for one of the causes above.

Action. Inspect the pods and jobs of the release:

d8 k -n code get pods,jobs

Resolve the cause, then run helm upgrade again; add --timeout when the release needs more time than the default.

Additional resources

• Quick start
• Advanced configuration