Platform Monkey ctx

Argo Rollouts and Progressive Delivery: Canary Analysis on Kubernetes

· By
Sources

The standard Kubernetes `Deployment` is a blunt instrument. Its `RollingUpdate` strategy is purely focused on availability—ensuring $N$ replicas are up—but it is completely blind to application health. A build that boots successfully but returns 500s for 100% of requests will be promoted across your entire fleet in minutes because the kubelet only cares that the container is "Running."

Progressive Delivery solves this by decoupling deployment (getting code onto a pod) from release (shifting traffic to that code).

Argo Rollouts implements this via a custom controller that manages a `Rollout` resource. Instead of a blind update, it facilitates a controlled shift of traffic, monitored by an automated Analysis Loop that can trigger a sub-second rollback if health markers degrade.

The Architectural Blueprint

The core of the system is the interaction between the Rollout Controller, the Traffic Provider (Ingress/Mesh), and the Metrics Provider (Prometheus).

+---------------------------------------------------------------------------+
|                          Argo Rollouts Controller                         |
|  (Manages Rollout CRD, computes weights, orchestrates AnalysisRuns)        |
+-----------------------+----------------------------+----------------------+
                        |                            |
            (1) Shift Traffic               (2) Query Metrics
                        |                            |
                        v                            v
+---------------------------------------+    +-------------------------------+
|           Traffic Provider            |    |       Metrics Provider        |
|  (Istio / Linkerd / NGINX / ALB)      |    |   (Prometheus / Datadog)      |
|                                       |    |                               |
|  [Stable Service] <--- (90% Traffic)   |    |  [PromQL: error_rate > 1%]    |
|  [Canary Service] <--- (10% Traffic)   |    |  [Result: 0.02 (FAIL)]        |
+---------------------------------------+    +-------------------------------+
                        ^                            |
                        |                            |
                        +-----------(3) ROLLBACK-----+
                                (Abort & Scale to 0)

The Analysis Loop: Template $ ightarrow$ Run $ ightarrow$ Result

Automated rollbacks are not "magic"; they are the result of a structured state machine:

1. `AnalysisTemplate`: A cluster-scoped blueprint. It defines how to check health (e.g., "Query Prometheus for the 99th percentile latency of the canary pods and ensure it is $< 500\text{ms}$").

2. `AnalysisRun`: An instantiation of the template. When a Rollout reaches a specific step (e.g., "Set Weight 20%"), the controller spawns an `AnalysisRun`.

3. The Polling Cycle: The controller polls the metrics provider at a defined `interval`.

4. The Decision: Once the `failureLimit` is reached, the `AnalysisRun` is marked as `Failed`. The Rollout controller immediately shifts 100% of traffic back to the Stable ReplicaSet and marks the rollout as `Degraded`.

Production-Grade Implementation

To implement this, you need three components: the Rollout, the AnalysisTemplate, and the Traffic routing configuration.

1. The AnalysisTemplate (The Health Definition)

Avoid overly sensitive thresholds. In production, a single outlier can trigger a "false-positive" rollback. Use a `failureLimit` to allow for transient blips.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate-check
spec:
  args:

    
  name: service-name


  metrics:

    
  name: error-rate


      interval: 1m
      # We allow up to 3 failed measurements before aborting
      failureLimit: 3
      successCondition: result[0] <= 0.01
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc.cluster.local:9090
          query: |
            sum(rate(http_requests_total{service="NaN", status=~"5.."}[2m])) 
            / 
            sum(rate(http_requests_total{service="NaN"}[2m]))

2. The Rollout Resource

The `Rollout` replaces your `Deployment`. Note the `canary` strategy and the `analysis` blocks.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api-gateway
spec:
  replicas: 10
  selector:
    matchLabels:
      app: api-gateway
  template:
    metadata:
      labels:
        app: api-gateway
    spec:
      containers:

    
  name: api-gateway


          image: my-repo/api-gateway:v2.1.0
          ports:

    
  containerPort: 8080


  strategy:
    canary:
      # Integration with NGINX Ingress
      canaryService: api-gateway-canary
      stableService: api-gateway-stable
      trafficRouting:
        nginx:
          service: api-gateway-service
      steps:

    
  setWeight: 10
  pause: {duration: 5m}
  analysis:


            templates:

    
  templateName: success-rate-check
  setWeight: 50
  pause: {duration: 10m}
  setWeight: 100

Production Realities: Where it Breaks

Progressive delivery is powerful, but it introduces new failure modes that can be more dangerous than a standard outage.

1. The "Ghost Canary" (The Silent Pass)

If your traffic provider (Ingress/Mesh) fails to actually shift traffic to the canary pods, your Prometheus queries for the canary will return `0` errors (because there are 0 requests). If your `successCondition` is `result[0] <= 0.01`, the analysis will Pass, and Argo will promote a potentially broken build to 100% of your users.

Mitigation: Always include a "Minimum Traffic" check in your `AnalysisTemplate` to ensure the canary is actually receiving requests before evaluating health.

2. Metric Lag and Windowing

Using `rate(...[5m])` in PromQL means your analysis is looking at a 5-minute trailing window. If you shift traffic at `T=0`, the metric at `T=1m` is heavily skewed by the "healthy" pods from the previous minute.

Mitigation: Use `irate` for higher resolution on fast-moving canaries, and ensure your `initialDelay` in the `AnalysisTemplate` allows the metrics window to fill with canary-only data.

3. The Resource Leak

Every `AnalysisRun` is a Kubernetes object. In a high-velocity CI/CD environment with 100+ services deploying multiple times a day, you will accumulate thousands of completed `AnalysisRun` objects, bloating the etcd store.

Mitigation: Implement a TTL or a cleanup cronjob to delete completed `AnalysisRun` resources older than 7 days.

Who Should Skip This

Do not implement Argo Rollouts if:

References & Further Reading