Operate

Troubleshoot

Find what the dashboard shows, or the condition kubectl prints, in the table below and go to its fix. Examples use the Rollout hello-world in namespace hello-world, environment production.

On this page
  1. What the dashboard shows
    1. Read the controller logs
  2. A new release is blocked
    1. No release source
    2. ReleasesUpdated is False with reason ReleaseCatalogueUnavailable
    3. ReleaseSourceReady is False
    4. No releases yet
    5. Blocked by a closed schedule or a failing gate
    6. Blocked by an approval or another rule
    7. Blocked by a service dependency
    8. Blocked by an unhealthy HealthCheck
    9. Blocked by a pin or a pending deploy by hand
    10. Ready is False with reason RolloutFailed, “force deploy version … is not in available releases”
    11. A bypass-gates annotation has no effect
    12. Promotion stalls and the log reads “API rate limit exceeded”
    13. The last deploy failed
  3. A deploy is stuck in Deploying
    1. A HealthCheck stays Pending or Unhealthy
    2. The message reads “Kustomization pending (ProgressingWithRetry)”
    3. The deploy never fails
  4. The deployment failed
    1. Roll back to the previous build
    2. Retry the same build
    3. Let a newer release deploy
    4. A HealthCheck failed but the deploy did not
  5. Troubleshoot other parts
  6. Check the result
  7. Next steps

What the dashboard shows

The banner at the top of a rollout’s Overview names what holds it. The words below are the dashboard’s own; production, 1.4.1 and payments stand for your environment, build and service.

The dashboard showsCondition on the RolloutGo to
Deployment failed banner
the Retry and Roll back buttons
Ready=False, BakeTimeFailedThe deployment failed
The stuck chip
“Stuck — deploying for 2h”
bakeStatus DeployingA deploy is stuck in Deploying
Recovery mode, while a deploy runsBakeFailureDisabled=TrueA HealthCheck failed but the deploy did not
Rolled backstatus.history[0] is older than the build before it (a pin, or a force-deploy of an older build until its bake succeeds)Roll back to the previous build
Held banner
“HealthCheck ‘…’ in namespace ‘…’ is not healthy”
DeploymentBlocked=True, UnhealthyHealthChecksBlocked by an unhealthy HealthCheck
production is waiting on an approval
Waiting for approval
Needs your approval
GatesPassing=False, NoAllowedVersionsBlocked by an approval
production is waiting for payments ^1.67.0GatesPassing=False, NoAllowedVersionsBlocked by a service dependency
production is waiting on promotion orderGatesPassing=False, NoAllowedVersionsPromote Across Environments; if staging already baked, Promotion stalls (rate limit)
Lockstep is set on … onlyLockstepConsistent=False on the Environment; holds nothingPromote Across Environments
Automatic deploys are paused
production is held
GatesPassing=False, SomeGatesBlockingBlocked by a closed schedule
production is pinned to 1.4.1
the Clear pin button
DeploymentBlocked reason ManualDeploymentBlocked by a pin
No release sourceReleasesUpdated=False, ReleaseSourceRequiredNo release source
No releases yetReady=False, NoReleasesNo releases yet

To read the same facts with kubectl, print the conditions and the Rollout’s events (HealthCheckBlocking, BakeTimeFailed).

Terminal
kubectl get rollout hello-world -n hello-world \
  -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'
kubectl events -n hello-world --for rollout/hello-world

Read the controller logs

The controllers’ logs hold the rest, such as “Retry annotation ignored”. The Helm chart runs them in kuberik-system.

Terminal
kubectl logs -n kuberik-system deployment/rollout-controller
kubectl logs -n kuberik-system deployment/release-controller
kubectl logs -n kuberik-system deployment/environment-controller-manager

The integrations log as datadog-controller-controller-manager and openkruise-controller-manager. Installed with kubectl, the first two are rollout-controller-controller-manager and release-controller-controller-manager, the latter in release-controller-system.

A new release is blocked

No release source

ReleasesUpdated is False with reason ReleaseSourceRequired: the Rollout has no spec.releaseSourceRef. Point it at the app’s ReleaseSource.

Terminal
kubectl patch rollout hello-world -n hello-world --type merge \
  -p '{"spec":{"releaseSourceRef":{"name":"hello-world"}}}'

ReleasesUpdated is False with reason ReleaseCatalogueUnavailable

The release-controller CRDs are not installed. Once they are, the Rollout picks them up without a restart.

Terminal
kubectl get crd releasesources.kuberik.com releaseslices.kuberik.com

ReleaseSourceReady is False

With reason ReleaseSourceNotFound, the name in spec.releaseSourceRef does not exist in the Rollout’s namespace. With ReleaseSourceNotReady, the ReleaseSource exists but is not Ready; its conditions are explained in Publish Releases from CI.

Terminal
kubectl get releasesource -n hello-world

No releases yet

Ready is False with reason NoReleases. The ReleaseSource admitted no tag: the image has no tags yet, or spec.oci.filterTags and spec.oci.policy match none of them. Compare the registry’s tags with the filter and fix the ReleaseSource.

Terminal
kubectl get releasesource hello-world -n hello-world -o jsonpath='{.status.count}{"\n"}{.spec.oci}{"\n"}'
Terminal
kubectl edit releasesource hello-world -n hello-world

Blocked by a closed schedule or a failing gate

GatesPassing is False with reason SomeGatesBlocking. A release is waiting and a gate has passing: false, often a closed window from a RolloutSchedule or ClusterRolloutSchedule. With nothing waiting, the condition stays True and the closed gate is still listed in status.gates.

Terminal
kubectl get rollout hello-world -n hello-world -o jsonpath='{.status.gates}'
kubectl get rolloutschedules -n hello-world
kubectl get clusterrolloutschedules

To let one build through now, select Skip once on the gate’s tile (Skip a gate once). With kubectl, bypass the gates for that build; the bypass takes effect on the next reconcile.

Terminal
kubectl annotate rollout hello-world -n hello-world rollout.kuberik.com/bypass-gates=1.4.1 --overwrite

Blocked by an approval or another rule

GatesPassing is False with reason NoAllowedVersions. Newer releases are waiting, but none is in every gate’s allowedVersions. blockedBy names the gates that hold each one.

Terminal
kubectl get rollout hello-world -n hello-world -o jsonpath='{range .status.candidates[*]}{.tag}{"\t"}{.blockedBy}{"\n"}{end}'

An approval-gate-… lets a release through once a RolloutApproval names it. In the dashboard, select Approve on the gate’s tile (Approve a build). With kubectl, apply one to approve 1.4.1.

Terminal
kubectl apply -f - <<EOF
apiVersion: kuberik.com/v1alpha1
kind: RolloutApproval
metadata:
  name: hello-world-1-4-1
  namespace: hello-world
spec:
  rolloutRef:
    name: hello-world
  version: "1.4.1"
EOF

Blocked by a service dependency

A dependency-… gate in blockedBy means the provider has not deployed the contract version yet. Ship the provider, or see Wait for Service Dependencies.

Terminal
kubectl get rolloutdependencies -n hello-world

Blocked by an unhealthy HealthCheck

DeploymentBlocked is True with reason UnhealthyHealthChecks. A selected HealthCheck is Unhealthy; the message names it. Fix the workload, or deploy a known build by hand, which skips this hold.

Terminal
kubectl get healthchecks -n hello-world

Blocked by a pin or a pending deploy by hand

DeploymentBlocked has reason ManualDeployment. A deploy by hand is pending or pinned: spec.wantedVersion is set, or a rollout.kuberik.com/force-deploy annotation names a tag that has not deployed yet. New releases wait. Remove whichever is set, or select Clear pin in the dashboard.

Terminal
kubectl get rollout hello-world -n hello-world -o jsonpath='{.spec.wantedVersion}{"\t"}{.metadata.annotations.rollout\.kuberik\.com/force-deploy}{"\n"}'
Terminal
kubectl patch rollout hello-world -n hello-world --type json \
  -p '[{"op":"remove","path":"/spec/wantedVersion"}]'
kubectl annotate rollout hello-world -n hello-world rollout.kuberik.com/force-deploy-

Ready is False with reason RolloutFailed, “force deploy version … is not in available releases”

The force-deploy annotation names a tag the ReleaseSource does not list. Set a listed tag, or pin any tag with spec.wantedVersion.

Terminal
kubectl annotate rollout hello-world -n hello-world rollout.kuberik.com/force-deploy=1.4.1 --overwrite

A bypass-gates annotation has no effect

A bypass for a build that is not newer than the deployed one is ignored (log: “Gate bypass requested for version not in candidates”). With bypass-gates-only, unnamed gates still hold the build, and the Event BypassGatesUnknown names gates the Rollout does not have. A bypass never skips an Unhealthy HealthCheck or a failed bake.

Terminal
kubectl get rollout hello-world -n hello-world -o jsonpath='{.status.gates[*].name}'

Promotion stalls and the log reads “API rate limit exceeded”

The environment-controller calls GitHub with each Environment’s token on every Rollout change and every spec.requeueInterval (default 1 minute). Once the token’s hourly limit is spent, reconciles fail before the Environment’s status and gate update. Staging shows the release baked, yet production still reads production is waiting on promotion order (GatesPassing=False, NoAllowedVersions): the gate keeps its last allowedVersions, the Environment its last WaveReady. Only the log names the error.

Terminal
kubectl logs -n kuberik-system deployment/environment-controller-manager --since=1h | grep "rate limit"
curl -s -H "Authorization: Bearer <your-token>" https://api.github.com/rate_limit

The error ends in API rate limit exceeded for … and [rate reset in …]. Promotion resumes after the reset. To spend fewer calls, reconcile each Environment less often; it then sees the earlier tier’s bake later.

Terminal
kubectl patch environment hello-world-production -n hello-world --type merge -p '{"spec":{"requeueInterval":"5m"}}'

The last deploy failed

A failed deploy stops automatic deploys until someone acts: see The deployment failed.

A deploy is stuck in Deploying

bakeStatus stays Deploying until every selected HealthCheck is Healthy and has changed since the deploy. Without deployTimeout, it waits forever.

A HealthCheck stays Pending or Unhealthy

For a Kustomization HealthCheck, the message names the resources Flux has not finished (Pending resources: …).

Terminal
kubectl get healthchecks -n hello-world -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.status}{"\t"}{.status.message}{"\n"}{end}'

The message reads “Kustomization pending (ProgressingWithRetry)”

With wait: true, Kuberik reads the Kustomization’s Ready condition, which needs kustomize-controller’s CancelHealthCheckOnNewRevision feature gate (Flux 2.7 or later). Without it, a failing revision holds the next one until spec.timeout. Enable the gate, or set wait: false so Kuberik checks each managed resource.

Terminal
kubectl patch kustomization hello-world -n hello-world --type merge -p '{"spec":{"wait":false}}'

The deploy never fails

Set deployTimeout so a deploy that never turns healthy fails.

Terminal
kubectl patch rollout hello-world -n hello-world --type merge -p '{"spec":{"deployTimeout":"15m"}}'

The deployment failed

The dashboard shows Deployment failed; Ready is False with reason BakeTimeFailed. The message says whether a HealthCheck reported an error or the deploy timeout passed; failedHealthChecks records each failing check. To ship a fix past the bad build, see Deploy a specific version.

Terminal
kubectl get rollout hello-world -n hello-world -o jsonpath='{.status.history[0].bakeStatusMessage}{"\n"}{.status.history[0].failedHealthChecks}'

Roll back to the previous build

This works for any build, failed or baked; Kuberik never rolls back on its own. In the dashboard, Roll back pins the Rollout to an earlier build. With kubectl, print the previous build’s tag.

Terminal
kubectl get rollout hello-world -n hello-world -o jsonpath='{.status.history[1].version.tag}{"\n"}'

Then set spec.wantedVersion to the tag it printed.

Terminal
kubectl patch rollout hello-world -n hello-world --type merge -p '{"spec":{"wantedVersion":"<previous-tag>"}}'

The pin deploys past gates and any failed bake, and the dashboard shows Rolled back. New releases wait until you clear the pin, as in Blocked by a pin. Do not roll back with force-deploy: once the earlier build’s bake succeeds, the Rollout deploys the newest release the gates allow, the failed one included.

Retry the same build

Retry in the dashboard sets this annotation. It resets the bake to Deploying and ignores errors from before the retry. The controller removes it; on a bake that is not Failed it logs “Retry annotation ignored”.

Terminal
kubectl annotate rollout hello-world -n hello-world rollout.kuberik.com/retry=true --overwrite

Let a newer release deploy

unblock-failed lets automatic deploys continue past the failed build once a newer release passes the gates and no selected HealthCheck is Unhealthy. Right after a failed bake a check is often still Unhealthy, and then nothing deploys. To roll forward during the incident, deploy by hand with spec.wantedVersion or force-deploy.

Terminal
kubectl annotate rollout hello-world -n hello-world rollout.kuberik.com/unblock-failed=true --overwrite
Terminal
kubectl patch rollout hello-world -n hello-world --type merge -p '{"spec":{"wantedVersion":"1.4.2"}}'

The next deploy runs with BakeFailureDisabled=True, so health check errors do not fail it.

A HealthCheck failed but the deploy did not

While the deploy runs, the dashboard shows Recovery mode; BakeFailureDisabled stays True until the next deploy starts. This deploy followed a bake that failed or did not finish (PreviousBakeFailed), or was made by hand while a HealthCheck was Unhealthy (DeployedWithUnhealthyHealthChecks).

Terminal
kubectl get rollout hello-world -n hello-world -o jsonpath='{.status.conditions[?(@.type=="BakeFailureDisabled")]}'

Troubleshoot other parts

AreaPage
ReleaseSource not Ready, tags missingPublish Releases from CI
Environments do not promote (WaveReady, LockstepConsistent)Promote Across Environments
Canary steps stall, tests never runRoll Out with a Canary
HealthChecks never start a bake or never changeVerify with Health Checks
Approval policies, Needs approval instead of Needs youRequire a Manual Approval
Kustomization substitutes and HealthChecksFlux
GitHub deployments and sign-inGitHub
Datadog monitorsDatadog
Prometheus alertsPrometheus
Install and upgrade errorsInstall with Helm
Dashboard sign-inSet Up Dashboard Sign-in

Check the result

Print the conditions again. A Rollout that deploys automatically shows GatesPassing=True and DeploymentBlocked=False, and after the bake Ready=True with reason BakeTimePassed.

Terminal
kubectl get rollout hello-world -n hello-world \
  -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\n"}{end}'

Next steps