Gate deploys

Verify with Health Checks

A Rollout with health checks marks a deploy succeeded once its checks stay healthy for the bake time, and fails it if a check turns unhealthy or deployTimeout passes first. Without deployTimeout, a build that never turns healthy waits in Deploying and holds newer releases.

On this page
  1. Add a health check
  2. Select health checks from the Rollout
  3. Hold deploys on an unhealthy check
  4. Choose how a Kustomization is checked
  5. Select health checks in other namespaces
  6. Check the result
  7. Troubleshoot
  8. Next steps
One deploy on the clock: deployTimeout bounds the wait for health, bakeTime runs from the moment every check is HealthyA deploy starts: its bakeStatus is Deploying and every selected HealthCheck is reset to Pending. When every check is Healthy again, here four minutes in, the bake starts and bakeStatus is InProgress. deployTimeout, 15 minutes from the deploy, is the limit for that moment. bakeTime, 10 minutes from the bake start, ends in Succeeded. A check error at any point after the deploy started fails it, before or during the bake; without healthCheckSelector the bake starts at once.deploy startsevery check HealthybakeTimepassedbakeStatusDEPLOYINGINPROGRESSSUCCEEDEDHealthChecksPENDINGHEALTHYdeployTimeout: 15m, the limit for the bake to startbakeTime: 10m, no error, then SucceededA check error after the deploy starts:Failed, before or during the bake.NohealthCheckSelector:the bake starts at once. One deploy on the clock: deployTimeout bounds the wait for health, bakeTime runs from the moment every check is HealthyA deploy starts: its bakeStatus is Deploying and every selected HealthCheck is reset to Pending. When every check is Healthy again, here four minutes in, the bake starts and bakeStatus is InProgress. deployTimeout, 15 minutes from the deploy, is the limit for that moment. bakeTime, 10 minutes from the bake start, ends in Succeeded. A check error at any point after the deploy started fails it, before or during the bake; without healthCheckSelector the bake starts at once.DEPLOYINGdeploy startschecks reset to PendingINPROGRESSevery check HealthySUCCEEDEDbakeTimepasseddeployTimeout: 15mbakeTime: 10ma check errorhere or before:Failed
deployTimeout limits the wait for every check to be Healthy, bakeTime runs from that moment, and a check error at any point fails the deploy.

Works with: Flux Kustomizations (built into rollout-controller), Datadog monitors and incidents (datadog-controller), and Prometheus alerts (prometheus-controller).

Add a health check

A HealthCheck reports Healthy, Unhealthy or Pending in status.status; its spec.class names the controller that fills it in. This one follows a Flux Kustomization:

healthcheck.yamlYAML
apiVersion: kuberik.com/v1alpha1
kind: HealthCheck
metadata:
  name: hello-world
  namespace: hello-world
  labels:
    app: hello-world  # the Rollout selects this
  annotations:
    healthcheck.kuberik.com/kustomization: "hello-world"
spec:
  class: kustomization

The annotation takes the Kustomization’s name, or <namespace>/<name> for one in another namespace.

Select health checks from the Rollout

Set healthCheckSelector on the Rollout, and a bake time the checks must hold:

rollout.yamlYAML
apiVersion: kuberik.com/v1alpha1
kind: Rollout
metadata:
  name: hello-world
  namespace: hello-world
spec:
  releaseSourceRef:
    name: hello-world
  bakeTime: 10m  # healthy this long
  deployTimeout: 15m
  healthCheckSelector:
    selector:
      matchLabels:
        app: hello-world

Each deploy, and each retry, then goes through these rules:

MomentWhat happens
Deploy startsEvery selected HealthCheck is reset to Pending and its lastErrorTime is cleared, and the deploy’s bakeStatus is Deploying
Every check is Healthy againBake starts: bakeStatus is InProgress
Any check reports an error after the deploy startedThe deploy fails (bakeStatus Failed, Ready reason BakeTimeFailed), before or during bake
deployTimeout passes before bake startsThe deploy fails (“Deploy timeout reached before bake could start”)
bakeTime passes with no errorThe deploy succeeds. Without bakeTime, it succeeds as soon as bake starts

deployTimeout fails the deploy if bake has not started by then; without it, the deploy waits. A Rollout without healthCheckSelector has no checks: bake starts at once.

Hold deploys on an unhealthy check

While a check is unhealthy, new automatic deploys wait. You do not configure this: while any selected HealthCheck is Unhealthy, the Rollout reports DeploymentBlocked=True with reason UnhealthyHealthChecks; Pending does not block. The very first deploy and manual deploys (a pin in spec.wantedVersion or rollout.kuberik.com/force-deploy) are not held.

After a deploy fails, the Rollout starts no new automatic deploy until you retry it, set rollout.kuberik.com/unblock-failed: "true", or deploy a version by hand (Operate with kubectl).

A manual deploy made while a check is Unhealthy, and any deploy that follows one that did not succeed, sets BakeFailureDisabled=True (reasons DeployedWithUnhealthyHealthChecks and PreviousBakeFailed). For that deploy, health-check errors and deployTimeout do not fail it, so a fix can land during an incident.

Terminal
kubectl -n hello-world get rollout hello-world -o jsonpath='{range .status.conditions[*]}{.type}={.status} ({.reason}){"\n"}{end}'

Choose how a Kustomization is checked

With spec.wait: false (the default) on the Kustomization, the check reads the Kustomization and every resource it manages with kstatus. With spec.wait: true, it reads only the Kustomization’s Ready condition and assumes kustomize-controller runs with the CancelHealthCheckOnNewRevision feature gate (Flux 2.7 or later); without it, a failing revision holds the next one until spec.timeout.

kustomization.yamlYAML
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: hello-world
  namespace: hello-world
spec:
  interval: 10m
  prune: true
  wait: true  # health from Ready only
  timeout: 5m
  sourceRef:
    kind: OCIRepository
    name: hello-world

Flux lists which Ready reasons count as unhealthy.

Select health checks in other namespaces

Add namespaceSelector to read HealthChecks from every namespace whose labels match, for example checks a platform team keeps in a shared namespace. It replaces the Rollout’s own namespace, so label that namespace too if its checks should still count.

rollout-cross-namespace.yamlYAML
apiVersion: kuberik.com/v1alpha1
kind: Rollout
metadata:
  name: hello-world
  namespace: hello-world
spec:
  releaseSourceRef:
    name: hello-world
  healthCheckSelector:
    selector:
      matchLabels:
        app: hello-world
    namespaceSelector:
      matchLabels:
        kuberik.com/health-checks: "true"

A deploy resets each selected check to Pending and clears its lastErrorTime. A check that several Rollouts select through namespaceSelector resets whenever any of them deploys, so keep the label selector narrow enough that each Rollout selects only its own checks.

Check the result

List the checks and the latest deploy’s bake state:

Terminal
kubectl -n hello-world get healthchecks -o custom-columns=NAME:.metadata.name,STATUS:.status.status,MESSAGE:.status.message
kubectl -n hello-world get rollout hello-world -o jsonpath='{.status.history[0].bakeStatus}: {.status.history[0].bakeStatusMessage}{"\n"}'
OutputText
NAME          STATUS    MESSAGE
hello-world   Healthy   Kustomization and all managed resources are healthy
InProgress: Bake started, monitoring for errors.

A failed deploy lists the checks that failed it in status.history[0].failedHealthChecks.

The rollout page’s Health checks card lists every check the Rollout selects and its state:

The Health checks card: 3/3 healthy, the Flux Deployment check and two alerts, High Error Rate Alert and App Down Alert, all passingThe Health checks card: 3/3 healthy, the Flux Deployment check and two alerts, High Error Rate Alert and App Down Alert, all passing

Troubleshoot

SymptomCauseFix
Bake never startsA selected check stays Pending or UnhealthyRead its status.message; set deployTimeout so the deploy fails instead of waiting
New releases are not deployedDeploymentBlocked=True, reason UnhealthyHealthChecksFix the failing check, or deploy a version by hand
New releases are not deployed, and no check is UnhealthyThe last deploy failed (bakeStatus Failed)Retry it, set rollout.kuberik.com/unblock-failed: "true", or pin a version
A check never changesNo controller handles its spec.classUse kustomization, or install datadog-controller for the datadog classes and prometheus-controller for prometheus-alert

Next steps