Deployment Strategies

How code goes from CI artifact to production. The patterns trade off risk, complexity, and rollback speed. Interview questions ask “which would you pick?” — answer with trade-offs, not “blue-green is always better.”

The strategies

For most apps: rolling. For higher-stakes: canary. Feature flags layered on top for safety.

Recreate — the simplest

Stop all old instances → start all new instances

Downtime during the gap. Used for:

• Stateful migrations where old and new can’t coexist.
• Dev environments.
• Some database deploys.

Don’t use for user-facing services unless downtime is acceptable.

Rolling deployment

Replace instances gradually:

Time 0:  [old] [old] [old] [old]
Time 1:  [old] [old] [old] [NEW]
Time 2:  [old] [old] [NEW] [NEW]
Time 3:  [old] [NEW] [NEW] [NEW]
Time 4:  [NEW] [NEW] [NEW] [NEW]

Most orchestrators default to rolling (Kubernetes Deployment, ECS, Nomad). Configurable batch size and health checks between batches.

# Kubernetes Deployment
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1     # at most 1 pod down at a time
      maxSurge: 1            # at most 1 extra pod beyond desired count

Trade-offs:

• No downtime if both versions can coexist.
• Standard, well-tooled.
• Brief period where requests hit both versions — backwards compatibility needed.
• Slow rollback (must roll the new version back through the same gradual process).

Compatibility requirement: new and old code must coexist during the transition. Database schema changes need to be backwards-compatible (add columns, don’t rename).

Blue-green

Two complete environments. Switch traffic between them.

[ Blue (live) ]  ← traffic
[ Green (idle) ]

→ Deploy new version to Green
[ Blue (live) ]  ← traffic
[ Green (new, idle) ]

→ Switch traffic
[ Blue (old, idle) ]
[ Green (live) ]  ← traffic

→ Validate Green, decommission Blue

The “switch” is typically a load balancer change (point at the other target group) or a DNS update.

Trade-offs:

• Instant cutover; instant rollback (flip back).
• Validate new version with smoke tests before switching.
• 2× infrastructure cost during transition.
• Database changes still need backwards compatibility (new code runs against the same DB the old will need if you rollback).
• In-flight connections: WebSocket / long-poll connections to old code stay until they close.

Good for: critical services where rollback speed matters more than infrastructure cost. Less common at very large scale (running 2× capacity is expensive).

Canary deployment

Roll out to a small percentage of users / hosts first. Watch metrics. If healthy, expand.

Time 0:  100% of traffic → old
Time 1:   1% → canary, 99% → old
Time 2:   5% → canary, 95% → old
Time 3:  25% → canary, 75% → old
Time 4:  100% → canary (now the new version)

Implementations:

• Service mesh (Istio, Linkerd) — traffic-split rules.
• Kubernetes + Argo Rollouts / Flagger — automated canary controller.
• Load balancer with weighted targets — AWS ALB, GCP LB.
• Feature flags at the application layer — gradual rollout per user / region.

Health check during canary:

• Error rate.
• Latency.
• Custom business metrics (conversion rate, etc.).
• Compare canary vs baseline statistically.

# Argo Rollouts canary
spec:
  strategy:
    canary:
      steps:
        - setWeight: 5
        - pause: { duration: 10m }
        - analysis:
            templates: [{name: success-rate}]
        - setWeight: 25
        - pause: { duration: 10m }
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100

If analysis fails at any step, auto-rollback.

Trade-offs:

• Limits blast radius of bad deploys.
• Real production validation before full rollout.
• Needs good metrics + automated analysis.
• Both versions running for the duration.
• Complex when state is involved (cache, session) — users may bounce between versions.

For high-traffic critical services: canary is the standard.

A/B testing — canary with a different goal

Same mechanics as canary (traffic split) but the goal is measuring user behavior, not validating deploys:

Often combined with feature flags so each user sees a consistent variant.

Shadow deployment (dark launch)

Send a copy of production traffic to the new version. Don’t return its responses to users.

User → [LB] → [old version] → response to user
              [new version (shadow)] → discarded

Use cases:

• Test performance under real load.
• Verify new version handles real traffic patterns.
• Migrate a major dependency (new DB, new search engine).

Implementation: load balancer / service mesh duplicates requests. Or instrumented code does the same work twice and compares.

Trade-offs:

• Zero user impact even if new version is broken.
• Real-world testing.
• 2× load on infrastructure during shadow.
• Side effects (writes, external calls) — must be carefully handled. Usually: shadow reads only, or sandbox the writes.
• Comparison complexity.

For high-stakes migrations (payment systems, search), shadow is invaluable.

Feature flags — decouple deploy from release

Code ships behind a flag, disabled. Release = flip the flag.

if features.is_enabled("new_checkout", user):
    return new_checkout_flow(user)
return old_checkout_flow(user)

Deploy:  ship code, flag off → no user impact
Release: turn flag on for 5% of users → measure
         turn flag on for 25% → measure
         turn flag on for 100% → done
Rollback: turn flag off → instant

Trade-offs:

• Deploy and release are independent.
• Instant rollback (no redeploy).
• Granular: per-user, per-tenant, per-region rollout.
• Combine with canary for layered safety.
• Flag debt: dead code behind flags accumulates.
• Testing: must test both branches.
• Runtime overhead (flag evaluation).

Tools: LaunchDarkly, Unleash, Statsig, Flipt, Split.io, or your own DB-backed implementation.

The modern best practice: feature flags + small frequent merges to main. Trunk-based development becomes safe because incomplete features ship behind flags.

Layered strategies

Real production: multiple of these combine.

PR merged → CI passes → deploy to staging (rolling)
                        ↓ smoke tests
                       deploy to production canary 5%
                        ↓ analyze 10 min
                       promote to 50%
                        ↓ analyze 10 min
                       promote to 100% (rolling within prod)
                        ↓
                       feature flag still off
                        ↓
                       turn flag on 1% → 5% → ... → 100%

Three layers: canary deploy, rolling deploy, feature flag rollout. Each catches different categories of issues.

Database migrations — the hard part

Application code can blue-green. The database can’t (easily).

Patterns for zero-downtime schema changes:

Expand-contract (parallel change)

Phase 1 (expand):  add new column, keep old
Phase 2:            new code writes to BOTH old and new
Phase 3:            backfill old data into new column
Phase 4 (contract): new code reads from new column only
Phase 5:            stop writing to old column
Phase 6:            drop old column

Each phase is independently deployable. Old code keeps working throughout.

Backwards-compatible only

Allowed:

• Add nullable column.
• Add index.
• Add new table.

Not allowed in one step:

• Rename column.
• Drop column still referenced by old code.
• Change column type incompatibly.
• Add NOT NULL without default.

The deploy pipeline enforces this — migrations can only contain backwards-compatible changes. The “drop old column” lands later, after the old code is fully retired.

See ../08_databases/sql/13_zero_downtime_migrations.md.

Rollback strategy

Every deploy strategy needs a rollback plan:

For mission-critical: combine blue-green / canary (fast infrastructure rollback) with feature flags (fast logic rollback). Belt and suspenders.

Important: practice rollbacks. The first time you need to rollback in prod should not be the first time you’ve done it. Quarterly fire drills.

Smoke tests post-deploy

Don’t just deploy and hope. Run synthetic tests against the deployed version:

deploy_production:
  script: kubectl apply -f manifests/
  after_script: |
    # Wait for rollout to complete
    kubectl rollout status deployment/api -w
    # Run smoke tests against production
    curl --fail https://api.example.com/health
    pytest tests/smoke/ --base-url https://api.example.com

If smoke tests fail → automatic rollback. Some orchestrators (Argo Rollouts) integrate this natively as “analysis steps.”

Common pitfalls

• No health checks during rolling deploy — old pods drain mid-request; new pods aren’t ready yet. Set liveness/readiness probes.
• Blue-green with shared DB and incompatible schema — Green expects new schema; Blue expects old. Switch the LB → Blue breaks. Use expand-contract.
• Canary % too aggressive — 50% canary skips meaningful sample of 5%. Start small.
• Canary metrics that don’t differ at small percentages — 1% of traffic may not generate enough signal in 10 min. Adjust thresholds.
• Feature flag forever — flag turned on, never cleaned up. Tech debt. Schedule cleanup sprints.
• No deploy-time rollback drill — first rollback is the worst time to discover the runbook is wrong.
• Sticky sessions + new deploy — users pinned to old pods don’t see the new version. See ../12_protocols/nginx/.

Common interview confusions

• “Blue-green is the same as canary.” — blue-green is all-or-nothing flip. Canary is gradual percentage rollout. Different risk profiles.
• “Rolling deploys have no risk.” — bad version rolls out gradually; eventually all pods are bad. Health checks help but won’t catch logic bugs that pass health checks.
• “Feature flags replace deploys.” — they decouple deploy from release. You still deploy; you just don’t expose the change to users until flipped.

Interview angle

• “What’s the difference between blue-green and canary deployments?” — blue-green: two complete environments, switch all traffic at once (fast cutover and rollback, 2× cost). Canary: gradual percentage rollout to subset of users / hosts, observe metrics, expand. Different trade-offs in risk vs cost.
• “How does rolling deployment work?” — replace instances gradually (batches), with health checks between batches. Old and new versions coexist during the rollout. No downtime if they’re backwards-compatible. Standard for Kubernetes Deployment, ECS, etc.
• “When would you use blue-green over rolling?” — when rollback speed matters more than infrastructure cost. Mission-critical services with stable user base where 2× capacity is acceptable.
• “How do canary deployments minimize risk?” — small percentage of real traffic hits the new version. If metrics (errors, latency, business KPIs) regress, rollback before full rollout. Service meshes / Argo Rollouts automate the analysis steps.
• “What’s a feature flag and why pair it with canary?” — feature flag decouples deploy from release. Canary catches deploy-time issues (new code crashes, doesn’t start); feature flags catch logic issues (new behavior is wrong for users). Layered safety.
• “How do you do database migrations without downtime?” — expand-contract: backwards-compatible schema changes first (add column, dual-write, backfill), then deploy code that reads from the new column, finally drop the old column. Multiple deploys, never breaking compatibility.
• “What’s a shadow deployment?” — duplicate real traffic to a new version; discard its responses. Tests performance / correctness against real load without user impact. Used for high-stakes migrations (payments, search). Careful with side effects (writes).