ECS — Task Placement and Lifecycle

The ECS overview covers task definitions, services, and the task-vs-execution-role split. This file is the operational depth: placement strategies, graceful shutdown, CPU/memory semantics, and health-check failure behavior.

Task placement strategies (EC2 launch type)

When ECS launches a task on the EC2 launch type, it has to pick which instance. Placement strategies control that (Fargate manages placement for you, so this is EC2-only).

You combine them — the common production pattern is spread across AZs, then binpack within an AZ:

"placementStrategy": [
  {"type": "spread", "field": "attribute:ecs.availability-zone"},
  {"type": "binpack", "field": "memory"}
]

That gives you AZ-level resilience and cost-efficient packing within each AZ.

Placement constraints are a separate thing — hard rules, not preferences:

"placementConstraints": [
  {"type": "memberOf", "expression": "attribute:ecs.instance-type =~ c5.*"}
]

“Only place this task on c5 instances.” Use for GPU tasks, license-bound tasks, etc.

CPU and memory semantics

A frequent source of confusion. ECS task/container CPU and memory have reservation vs limit semantics:

CPU units: 1 vCPU = 1024 CPU units. A container with cpu: 256 gets a quarter-vCPU worth of scheduling weight — but on an uncontended EC2 host it can use more. On Fargate, the task-level cpu is a real cap.

Practical guidance:

• Set a container hard memory limit slightly above steady-state — too tight and a transient spike OOM-kills a healthy container; too loose and one container can starve the host.
• Use memoryReservation (soft) for the scheduler’s bin-packing math, memory (hard) as the safety ceiling.

Health checks and failure behavior

Two layers of health check, and they do different things:

Container health check (in the task definition)

"healthCheck": {
  "command": ["CMD-SHELL", "curl -f http://localhost:8000/healthz || exit 1"],
  "interval": 30,
  "timeout": 5,
  "retries": 3,
  "startPeriod": 60
}

If the container fails retries consecutive checks → ECS marks the container unhealthy → the service stops the task and launches a replacement. startPeriod is a grace window after start where failures don’t count (gives the app time to boot).

Load balancer health check (in the target group)

The ALB/NLB independently health-checks the task’s registered target. If it fails, the LB stops routing to that task — and if it’s part of a service, ECS will also replace the task after the target-group health check fails.

The two can disagree: the container check says “process is up” while the LB check says “not serving traffic correctly.” For a service behind an LB, the LB health check is usually the one that drives task replacement; the container health check is a backup that also catches non-LB tasks.

Graceful shutdown — the SIGTERM sequence

When ECS stops a task (deploy, scale-in, instance drain), it does not just kill it:

1. ECS sends SIGTERM to the container's main process.
2. ECS waits up to `stopTimeout` seconds (default 30, max 120).
3. If the process hasn't exited, ECS sends SIGKILL.

Your app must handle SIGTERM: stop accepting new requests, finish in-flight ones, close DB connections, then exit. If it ignores SIGTERM, in-flight requests get killed at the SIGKILL.

import signal, sys

def shutdown(signum, frame):
    server.stop_accepting_connections()
    server.wait_for_inflight(timeout=25)   # under stopTimeout
    db_pool.close()
    sys.exit(0)

signal.signal(signal.SIGTERM, shutdown)

Also relevant: the deregistration delay on the target group (connection draining). When a task is being stopped, the LB stops sending new connections but lets existing ones finish for the deregistration-delay window (default 300s — often too long; tune to ~30s). Sequence the two: LB stops new traffic → app drains in-flight → SIGTERM grace → exit.

Set stopTimeout high enough to cover your real drain time, and make sure terminationGracePeriod/deregistration-delay align so you’re not SIGKILLed mid-drain.

Deployment circuit breaker

"deploymentConfiguration": {
  "deploymentCircuitBreaker": {"enable": true, "rollback": true},
  "maximumPercent": 200,
  "minimumHealthyPercent": 100
}

The circuit breaker watches a rolling deployment: if new tasks keep failing to reach healthy, it stops the deployment and (with rollback: true) automatically reverts to the last good task definition. Without it, a broken deploy can churn forever launching failing tasks. Always enable it.

minimumHealthyPercent: 100 + maximumPercent: 200 means a rolling deploy never drops below the current task count and can temporarily double — zero-downtime, at the cost of briefly running 2× tasks.

Service-level failure routing

ECS doesn’t have a built-in “DLQ for tasks” — but the patterns that matter:

• Failed task → circuit breaker → rollback (above).
• For worker services consuming a queue — the queue’s DLQ (SQS maxReceiveCount) handles poison messages; the ECS task just keeps pulling. The reliability lives in the queue, not ECS.
• CloudWatch alarms on RunningTaskCount vs DesiredCount — alert when the service can’t keep tasks healthy.

Capacity providers and Spot

A capacity provider tells ECS where to run tasks. The interesting one is mixing Fargate and Fargate Spot:

"capacityProviderStrategy": [
  {"capacityProvider": "FARGATE", "weight": 1, "base": 2},
  {"capacityProvider": "FARGATE_SPOT", "weight": 4}
]

base: 2 on FARGATE = always keep 2 tasks on on-demand (survive a Spot reclaim). weight 1:4 = beyond the base, 80% of new tasks go to Spot. For interruptible workers, this is a big cost saving; Spot tasks get a SIGTERM with ~2 minutes notice on reclaim — your graceful-shutdown handler covers it.

Common gotchas

• No placement strategy on EC2 — tasks pile onto one instance; one instance failure takes them all down. spread across AZs.
• App ignores SIGTERM — in-flight requests killed on deploy/scale-in. Handle the signal.
• stopTimeout too short for the real drain time — SIGKILL mid-drain. Default 30s; raise it if your requests are long.
• Deregistration delay default 300s — slow deploys; tune to ~30s for fast HTTP services.
• Hard memory limit too tight — transient spike OOM-kills a healthy container.
• Circuit breaker not enabled — a broken deploy churns forever.
• Container health check vs LB health check disagreeing — know which one is driving task replacement (usually the LB check for LB-backed services).

Interview angle

• “How does ECS decide which instance to place a task on?” — placement strategies on the EC2 launch type: binpack (cost — fewest instances), spread (HA — across instances/AZs), random. Common production combo: spread across AZs, binpack within. Fargate manages placement itself.
• “Walk me through graceful shutdown of an ECS task.” — ECS sends SIGTERM, waits stopTimeout (default 30s, max 120s), then SIGKILL. The app must catch SIGTERM: stop accepting new work, drain in-flight, close connections, exit. The LB’s deregistration delay stops new connections while draining; sequence the two.
• “Container CPU/memory — reservation vs limit?” — container cpu is a soft share (scheduling weight, can burst on EC2); container hard memory is an OOM-kill ceiling; memoryReservation is the soft value the scheduler bin-packs with. Task-level cpu/memory on Fargate is a real billed cap.
• “What’s the deployment circuit breaker?” — it watches a rolling deploy; if new tasks keep failing to go healthy, it halts the deploy and (with rollback: true) auto-reverts to the last good task definition. Prevents a broken deploy from churning forever.
• “How do you run ECS workers cheaply but safely?” — capacity provider strategy mixing FARGATE (with a base for resilience) and FARGATE_SPOT (weighted for the bulk). Spot reclaims send SIGTERM with ~2 min notice; your graceful-shutdown handler covers it.
• “Container health check vs load balancer health check?” — container check (“is the process alive”) drives replacement for non-LB tasks; LB target-group check (“is it serving traffic correctly”) drives it for LB-backed services. They can disagree — know which one is authoritative for your service.