Progressive delivery on Kubernetes — Argo Rollouts & Flagger

Progressive delivery = deployment strategies (05_deployment_strategies.md) run as an automated, metric-gated process: shift a slice of traffic, watch the metrics, promote or roll back — no human staring at Grafana. Canary/blue-green describe the shape; progressive delivery is the machinery.

Why a stock k8s Deployment can’t do this

A Deployment gives you rolling updates only — and rolling has three gaps:

• No traffic control: replicas flip old→new; you can’t say “5% of traffic to v2.” With 4 pods, the granularity is 25%, decided by pod count, not intent.
• No metric gate: readiness probes (../17_kubernetes/04_probes_and_hpa.md) check “is the pod up,” not “did the error rate double.” A pod that starts cleanly and corrupts 2% of requests rolls out to 100%.
• No automatic rollback: a bad rollout sits there until a human runs kubectl rollout undo.

Progressive-delivery controllers close all three: traffic shifting via ingress/mesh, promotion gated on real metrics, rollback automatic on failure.

Argo Rollouts

Replaces Deployment with a Rollout CRD — same pod template, plus a strategy:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
spec:
  replicas: 10
  strategy:
    canary:
      steps:
        - setWeight: 5          # 5% of traffic to the canary
        - pause: {duration: 10m}
        - analysis:             # metric gate — promotion stops here on failure
            templates:
              - templateName: success-rate
        - setWeight: 25
        - pause: {duration: 10m}
        - setWeight: 50
        - pause: {}             # manual approval gate (pause forever until promoted)
  # ...template as in a Deployment

The gate is an AnalysisTemplate — typically a Prometheus query with a pass condition:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
spec:
  metrics:
    - name: success-rate
      interval: 1m
      failureLimit: 3
      successCondition: result[0] >= 0.99
      provider:
        prometheus:
          address: http://prometheus:9090
          query: |
            sum(rate(http_requests_total{service="checkout",status!~"5.."}[5m]))
            / sum(rate(http_requests_total{service="checkout"}[5m]))

Analysis fails → Rollout aborts and traffic snaps back to stable. It also supports blueGreen (preview service, prePromotionAnalysis, instant switch) — the blue-green flow from 05_deployment_strategies.md with the gate built in.

Fine-grained traffic percentages need an ingress/mesh integration (NGINX Ingress, ALB, Istio, Linkerd/SMI); without one, Rollouts approximates weight by scaling replica counts.

Flagger

Same goal, inverted ergonomics: you keep your plain Deployment; Flagger is an operator that watches it plus a Canary custom resource. On image change it clones the Deployment into -primary, sends weighted traffic to the canary through the mesh/ingress, steps the weight up while checking metrics (built-in success-rate/latency checks, plus custom PromQL), and promotes or rolls back. Webhooks slot in load tests or manual gates.

Both are CNCF-standard; the interview answer is knowing one concretely and the trade-off table.

What makes a good gate metric

Gate on symptoms users feel, measured on the canary pods only (label-scoped queries):

• HTTP 5xx / success rate, p95–p99 latency — the RED basics (../15_observability/03_prometheus.md).
• Domain metrics for the money path: payment-auth decline rate, checkout conversion.
• Tie thresholds to your SLOs — a canary burning error budget is the definition of “roll back” (../15_observability/12_slo_sli_sla.md).

Pitfalls

• Not enough traffic for significance: 5% of a low-QPS service = a handful of requests per interval; one flaky request “fails” the gate, or real breakage passes. Use longer windows, higher starting weights, or synthetic load via webhooks.
• Sticky/stateful traffic: weighted routing assumes any pod can serve any request; session affinity or in-pod state skews both traffic and metrics (09_stateful_vs_stateless.md — stateless services are the prerequisite).
• Database migrations don’t canary: schema is shared by old and new versions simultaneously — expand/contract discipline still applies (../08_databases/sql/13_zero_downtime_migrations.md).
• Gate on averages: a 1% cohort disaster vanishes in a global average — scope queries to canary pods, alert on ratios not counts.
• Metrics lag (scrape + rate windows) — pauses shorter than ~2× the window gate on noise.

Interview angle

• “How would you automate a canary release on Kubernetes?” — name the Deployment’s three gaps, then Rollout steps: weight → pause → analysis → promote/abort, with a Prometheus success-rate gate.
• “Argo Rollouts vs Flagger?” — CRD-replacement + explicit steps vs operator + declarative thresholds; both need ingress/mesh for true traffic weights.
• “What metrics gate a rollout?” — canary-scoped error rate + tail latency + one business metric; thresholds derived from SLOs.
• “When does canary analysis fail you?” — low traffic, sticky sessions, shared DB schema, average-masking — and what you do about each.