Ship releases

Roll Out with a Canary

Move traffic to a new version in steps, and let each step continue only after it has stayed ready for a bake time and its tests have passed. Use it for user-facing services where problems show only under real traffic.

On this page
  1. Define the canary steps
  2. Run tests at a step
  3. Retry or skip a failed test
  4. Check the result
  5. Troubleshoot
  6. Next steps
OpenKruise moves traffic in steps; openkruise-controller unpauses a step once its test passed and its bake time ranStep 1 sends 10% of traffic to one canary replica and pauses. openkruise-controller waits until the RolloutTest smoke-test for step 1 Succeeded, then until the step has stayed ready for step-1-bake-time, and unpauses it. Step 2 sends 50% to five replicas. OpenKruise completes the last step without pausing, and the rollout is Healthy with all traffic on the new version. A failed test or a passed ready-timeout leaves the step paused with Stalled=True.Step 1replicas: 1traffic: 10%Step 2replicas: 5traffic: 50%After the last stepphase: Healthyall trafficopenkruise-controllerRolloutTest smoke-test Succeeded,then ready forstep-1-bake-timepausedunpausesThe test fails, orstep-1-ready-timeoutpasses:the step stays paused,Stalled=True.The last step:OpenKruise completes it without pausing, so its tests never run. OpenKruise moves traffic in steps; openkruise-controller unpauses a step once its test passed and its bake time ranStep 1 sends 10% of traffic to one canary replica and pauses. openkruise-controller waits until the RolloutTest smoke-test for step 1 Succeeded, then until the step has stayed ready for step-1-bake-time, and unpauses it. Step 2 sends 50% to five replicas. OpenKruise completes the last step without pausing, and the rollout is Healthy with all traffic on the new version. A failed test or a passed ready-timeout leaves the step paused with Stalled=True.Step 1replicas: 1traffic: 10%Step 2replicas: 5traffic: 50%openkruise-controllerRolloutTest smoke-test Succeeded,then ready forstep-1-bake-timepausedunpausesAfter the last stepphase: Healthyall trafficTest fails or times out:stays paused.Last step:never pauses.
Each step stays paused until its test passed and its bake time ran; the last step never pauses.

Works with: OpenKruise Rollout and the Kuberik openkruise-controller. Install it with the Helm chart’s integrations.openkruise.enabled, or on its own:

Terminal
kubectl apply --server-side -f https://github.com/kuberik/openkruise-controller/releases/latest/download/install.yaml

Install OpenKruise Rollout itself from the OpenKruise installation guide.

Define the canary steps

An OpenKruise Rollout splits a Deployment into steps; each step sets the canary’s replicas and share of traffic. Set a bake time per step with an annotation:

kruise-rollout.yamlYAML
apiVersion: rollouts.kruise.io/v1beta1
kind: Rollout
metadata:
  name: hello-world
  namespace: hello-world
  annotations:
    rollout.kuberik.io/step-1-bake-time: "5m"
    rollout.kuberik.io/step-1-ready-timeout: "10m"
spec:
  workloadRef:
    apiVersion: apps/v1
    kind: Deployment
    name: hello-world
  strategy:
    canary:
      steps:
        - replicas: 1
          traffic: 10%
        - replicas: 5
          traffic: 50%
      trafficRoutings:
        - service: hello-world
          gateway:
            httpRouteName: hello-world
AnnotationEffect
rollout.kuberik.io/step-<N>-bake-timeHow long step N must stay ready before it continues
rollout.kuberik.io/step-<N>-ready-timeoutHow long step N may take to become ready before the rollout stops and waits for you

N is 1-based and counts the steps in order. A step with neither a bake time nor a RolloutTest is never approved by Kuberik: it stays paused until you approve it in OpenKruise. A step with tests and no bake time continues when its tests pass. When Flux applies this Rollout from a Kustomization that a Kuberik Rollout drives, a failed bake on the Kuberik Rollout also stops the canary at its current step. See the OpenKruise canary strategy for every step option.

Run tests at a step

A RolloutTest runs a Job when the canary reaches a step. The step continues only if the Job succeeds; if it fails, the step stays paused until you retry or skip the test.

smoke-test.yamlYAML
apiVersion: rollout.kuberik.com/v1alpha1
kind: RolloutTest
metadata:
  name: smoke-test
  namespace: hello-world
spec:
  rolloutName: hello-world  # the OpenKruise Rollout
  stepIndex: 1
  jobTemplate:
    backoffLimit: 0
    template:
      spec:
        restartPolicy: Never
        containers:
          - name: test
            image: curlimages/curl
            command: ["curl", "--fail", "http://hello-world-canary"]

jobTemplate is a Kubernetes Job spec, and stepIndex is 1-based like the annotations. Create one RolloutTest per step you want to test.

Tests on the last step never run: OpenKruise completes the last step without pausing. Put tests on an earlier step.

Retry or skip a failed test

A failed test stops the step and sets Stalled=True with reason RolloutTestFailed on the OpenKruise Rollout, but that alone does not fail the Kuberik deploy, and a retry does nothing until the deploy has failed. Retry and skip therefore need two things: the Kuberik Rollout drives the Flux Kustomization that applies the OpenKruise Rollout, and it selects a kustomization HealthCheck for that Kustomization. With wait: false, that check reads the stalled OpenKruise Rollout as failed, so the deploy fails.

Retry the failed deploy from the dashboard’s Retry button, or with the rollout.kuberik.com/retry annotation on the Kuberik Rollout, and the failed tests at the current step run again. To let the step continue without them, also set rollouttest.kuberik.com/retry-mode: skip; the failed tests are marked Skipped and count as passed.

Terminal
kubectl -n hello-world annotate rollouts.kuberik.com hello-world \
  rollouttest.kuberik.com/retry-mode=skip rollout.kuberik.com/retry=true --overwrite

Check the result

Each test reports its phase: WaitingForStep, Pending, Running, Succeeded, Failed, Cancelled or Skipped.

Terminal
kubectl -n hello-world get rollouttests -o custom-columns=NAME:.metadata.name,STEP:.spec.stepIndex,PHASE:.status.phase

In the dashboard, each canary step is a stage of the deploy:

The deployment pipeline of a rollout with two canary stages: Started, Stage 1, Stage 2, then Check, where the health checks are healthy and the check timer is doneThe deployment pipeline of a rollout with two canary stages: Started, Stage 1, Stage 2, then Check, where the health checks are healthy and the check timer is done

Troubleshoot

SymptomCauseFix
The canary stops at a step and the OpenKruise Rollout shows Stalled with reason StepReadyTimeoutExceededThe step was not ready within step-<N>-ready-timeoutFix the canary pods, then retry the failed deploy on the Kuberik Rollout: a retry restarts the step’s timeout
A test never leaves WaitingForStepIts stepIndex is the last stepMove the test to an earlier step
A test failed (Stalled, reason RolloutTestFailed), and Retry or rollout.kuberik.com/retry does nothingThe Kuberik deploy has not failed: no kustomization HealthCheck sees the stalled OpenKruise RolloutSelect a kustomization HealthCheck for the Kustomization that applies the OpenKruise Rollout
A step stays paused with no test and no failureThe step has no step-<N>-bake-time and no RolloutTestAdd a bake time or a test for that step, or approve it in OpenKruise

Next steps