Probes and HPA

How k8s decides if your pod is healthy (probes) and how it decides to scale (HPA).

Probes — the three kinds

The classic mistake: only setting liveness. A slow-starting service gets liveness-killed during boot before it ever serves traffic — CrashLoopBackOff with a misleading symptom. Set readiness too, or use a startup probe to gate the others.

Configuring probes

containers:
- name: app
  startupProbe:                     # gates the others until passes once
    httpGet: { path: /healthz/start, port: 8000 }
    failureThreshold: 30
    periodSeconds: 2                # 30*2s = 60s startup window
  readinessProbe:
    httpGet: { path: /healthz/ready, port: 8000 }
    periodSeconds: 5
    failureThreshold: 3             # 15s unhealthy → out of LB
  livenessProbe:
    httpGet: { path: /healthz/live, port: 8000 }
    periodSeconds: 30
    failureThreshold: 3             # 90s unresponsive → restart
    timeoutSeconds: 5

What to expose

Three endpoints, each fast and side-effect-free:

@app.get("/healthz/live")
async def live():
    return {"ok": True}   # process exists and event loop runs

@app.get("/healthz/ready")
async def ready():
    if not await db.is_connected():
        return JSONResponse({"ok": False}, status_code=503)
    if not redis.ping():
        return JSONResponse({"ok": False}, status_code=503)
    return {"ok": True}

@app.get("/healthz/start")
async def started():
    return {"ok": migrations_applied and cache_warm}

Liveness is just “I’m alive.” Don’t ping the DB here — a brief DB blip would kill every pod simultaneously, taking down the service.

Readiness checks dependencies. A pod that can’t reach its DB shouldn’t serve traffic, but a sweeping DB outage shouldn’t restart every pod.

Probe types

# HTTP — what 99% of services use
httpGet: { path: /healthz/ready, port: 8000, scheme: HTTP }

# TCP — port-open check; for non-HTTP services
tcpSocket: { port: 5432 }

# exec — run a command; non-zero exit = fail
exec:
  command: ["/usr/local/bin/health"]

# gRPC (k8s 1.24+) — health.v1 protocol
grpc: { port: 9000 }

Common probe bugs

• Liveness pings the DB. Brief DB outage → every pod restarts → cascading failure.
• Readiness too aggressive. A 1s blip flips a pod out of rotation; user sees 503. Use failureThreshold: 3 with a 5s period.
• No startup probe on slow-booting apps. Liveness kills the pod mid-boot. Loop.
• timeoutSeconds too short. GC pause exceeds timeout → false failure. Real-world async Python services need ≥3-5s.
• Probe blocks on the event loop. Async app, but /healthz does time.sleep(2) — every probe is broken under load.

HPA — Horizontal Pod Autoscaler

Scales pod replicas based on metrics. Three flavors:

CPU-based (the default)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata: { name: orders }
spec:
  scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: orders }
  minReplicas: 3
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target: { type: Utilization, averageUtilization: 70 }

“Add pods until average CPU across all pods ≈ 70% of requests.” Requires resources.requests.cpu set on the pods. Without that, HPA can’t compute utilization.

Custom / external metrics

For queue-depth-based scaling (Celery / SQS workers):

metrics:
- type: External
  external:
    metric:
      name: sqs_approximate_messages_visible
      selector: { matchLabels: { queue: "prod-orders" } }
    target: { type: AverageValue, averageValue: "50" }

Means: “scale so each pod has on average 50 queued messages.” More messages → scale up. KEDA is the de-facto operator for this — supports SQS, Kafka, Redis, Prometheus, etc. without you wiring metrics-server adapters.

Memory-based

metrics:
- type: Resource
  resource:
    name: memory
    target: { type: Utilization, averageUtilization: 80 }

Memory rarely makes sense for HPA — apps usually have stable memory floors; scaling up doesn’t shed memory pressure unless you have leaks.

What CPU to scale on (Python-specific)

GIL means a single CPython process uses ~1 core max. If you ran one process per pod with requests: { cpu: "1000m" }, the HPA decision is straightforward. If you run gunicorn with 4 workers per pod, requests: { cpu: "4000m" } and the HPA still works on aggregate.

Async services (uvicorn, no workers) — 1 process, 1 core. Set requests to ~700m and limits to 1000m; scale on CPU ~70%.

Scale-down behavior

HPA defaults to:

• Stabilization window: 5 min for scale-down (avoid flapping).
• Policy: at most 100% of current replicas removed per 60s (i.e., halve, but don’t drop to zero).

behavior:
  scaleDown:
    stabilizationWindowSeconds: 300
    policies:
    - type: Percent
      value: 50
      periodSeconds: 60
  scaleUp:
    stabilizationWindowSeconds: 30
    policies:
    - type: Percent
      value: 100
      periodSeconds: 30
    - type: Pods
      value: 4
      periodSeconds: 30
    selectPolicy: Max     # take the most aggressive

Scale up fast, scale down slow — standard production wisdom.

VPA — Vertical Pod Autoscaler

Resizes resource requests/limits on pods (vs HPA which adds replicas). Three modes:

• Off — just recommend.
• Initial — set requests on pod creation, never resize.
• Auto — evicts pods to apply new sizes.

Don’t run VPA and HPA on the same resource simultaneously — they fight. Use VPA “Off” mode to inform manual resource-request tuning; use HPA for live scaling.

K8s 1.27+ has in-place pod resize which removes the “evict to resize” pain, but it’s still beta.

Cluster autoscaler / Karpenter

HPA adds pods; if no node has room, pods are Pending. Then:

• Cluster Autoscaler (classic) — checks node groups, scales the node group up.
• Karpenter (AWS, newer) — provisions nodes directly based on pending pods’ requests, picks the right instance type from a set of allowed shapes.

Karpenter is the modern default on EKS; consolidates underutilized nodes automatically.

Pod Disruption Budget

Prevents too many pods from being voluntarily evicted at once (during node drain, upgrades, etc.).

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata: { name: orders }
spec:
  minAvailable: 2
  selector: { matchLabels: { app: orders } }

“At least 2 orders pods must always be available during voluntary disruptions.” Without this, a node drain can take all replicas down briefly.

Interview angle

• “Liveness vs readiness — concrete failure for each?” — liveness fails: container is deadlocked / event loop stuck → restart. Readiness fails: DB pool exhausted / migration not yet run → remove from LB but don’t restart.
• “What should a liveness endpoint do?” — almost nothing. Return 200 if the process is alive and the event loop runs. Don’t hit the DB, don’t compute anything.
• “What should readiness check?” — the dependencies you actually need to serve a request: DB connection, cache, queue. A failing readiness shouldn’t kill the pod; the dependency might come back.
• “You set requests.cpu: 200m and HPA on 70% CPU. Why isn’t it scaling?” — CPU usage divided by requests reaches 70%. If the app uses 600m on a 200m request, that’s 300% — HPA scales aggressively. If it’s only at 100m, that’s 50% — no scale. The metric is utilization vs request, not absolute.
• “How do you autoscale a Celery worker on queue depth?” — HPA with external metric (or KEDA + SQS/Redis scaler). Target average queue length per pod; HPA adds replicas as backlog grows.
• “Why scale up fast but scale down slow?” — scale-up failure = user pain (timeouts). Scale-down failure = cost. Asymmetric penalty → asymmetric defaults.