AWS Step Functions

Managed workflow orchestration. You define a state machine (JSON / Amazon States Language); Step Functions runs it, handles retries, branching, parallelism, and keeps the execution history. The AWS-native alternative to Celery canvas / Temporal / hand-rolled orchestration.

Express vs Standard workflows

The first decision, and a guaranteed interview question.

Rule of thumb: Standard for orchestration you need to see and audit; Express for high-throughput event processing where cost matters and you’ve made steps idempotent.

State types

A state machine is a graph of states. The main ones:

{
  "Comment": "Order processing",
  "StartAt": "ValidateOrder",
  "States": {
    "ValidateOrder": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:...:function:validate-order",
      "Retry": [{"ErrorEquals": ["States.TaskFailed"], "MaxAttempts": 3, "BackoffRate": 2.0}],
      "Catch": [{"ErrorEquals": ["ValidationError"], "Next": "RejectOrder"}],
      "Next": "ChargeCard"
    },
    "ChargeCard": {
      "Type": "Task",
      "Resource": "arn:aws:lambda:...:function:charge-card",
      "Next": "FulfillOrder"
    },
    "FulfillOrder": { "Type": "Task", "Resource": "...", "End": true },
    "RejectOrder": { "Type": "Fail", "Error": "OrderRejected" }
  }
}

Error handling — Retry and Catch

The reason to use Step Functions over hand-rolled orchestration: retries and error routing are declarative.

"Retry": [
  {
    "ErrorEquals": ["States.Timeout", "Lambda.ServiceException"],
    "IntervalSeconds": 2,
    "MaxAttempts": 3,
    "BackoffRate": 2.0
  }
],
"Catch": [
  {
    "ErrorEquals": ["States.ALL"],
    "ResultPath": "$.error",
    "Next": "HandleFailure"
  }
]

• Retry — retry the same state on matching errors, with exponential backoff.
• Catch — on unrecoverable error, route to a fallback state (compensating action, alert, cleanup).

This is the saga pattern made declarative: each Task can Catch into a compensation state.

Input/output processing

The most confusing part. Four filters, applied in order:

raw input
  → InputPath        select a portion of the input to pass to the state
  → Parameters       construct the actual input to the task (can reference $$ context)
  → [task runs]
  → ResultSelector   reshape the task's raw result
  → ResultPath       where to put the result in the state's input ($ = replace, $.x = merge into x)
  → OutputPath       select a portion of the combined result to pass to the next state
  → output

The common gotcha: ResultPath: "$" (default) replaces the entire state input with the task result — you lose the original input. To keep the input and add the result, use ResultPath: "$.result".

"ChargeCard": {
  "Type": "Task",
  "Resource": "...",
  "ResultPath": "$.chargeResult",   // keep original input, add result under .chargeResult
  "Next": "FulfillOrder"
}

Map state — and Distributed Mode

Map runs the same sub-workflow over each element of an array.

• Inline Map — up to 40 concurrent iterations; iteration state held in the execution. For small batches.
• Distributed Map — for large-scale fan-out: process millions of items (e.g., every object in an S3 bucket), up to 10,000 concurrent executions. Each iteration is a child execution; results aggregated. This replaces “Lambda fan-out via SQS” for big batch jobs.

"ProcessFiles": {
  "Type": "Map",
  "ItemReader": {
    "Resource": "arn:aws:states:::s3:listObjectsV2",
    "Parameters": {"Bucket": "my-data-bucket"}
  },
  "ItemProcessor": {
    "ProcessorConfig": {"Mode": "DISTRIBUTED", "ExecutionType": "EXPRESS"},
    "StartAt": "ProcessOne",
    "States": {"ProcessOne": {"Type": "Task", "Resource": "...", "End": true}}
  },
  "MaxConcurrency": 1000,
  "End": true
}

Direct SDK integrations — no Lambda needed

Step Functions can call ~200 AWS services directly. You don’t need a Lambda just to PutItem to DynamoDB or send an SNS message.

"SaveOrder": {
  "Type": "Task",
  "Resource": "arn:aws:states:::dynamodb:putItem",
  "Parameters": {
    "TableName": "Orders",
    "Item": {"orderId": {"S.$": "$.orderId"}, "status": {"S": "confirmed"}}
  },
  "Next": "NotifyUser"
}

Removes a whole class of trivial “glue” Lambdas. Cheaper, fewer cold starts, less code.

Callback pattern with task tokens

For steps that wait on something external (human approval, a third-party webhook, a long async job):

"WaitForApproval": {
  "Type": "Task",
  "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
  "Parameters": {
    "FunctionName": "request-approval",
    "Payload": {"taskToken.$": "$$.Task.Token", "orderId.$": "$.orderId"}
  },
  "Next": "Fulfill"
}

Step Functions pauses the execution and passes a task token. Something external eventually calls SendTaskSuccess(taskToken, output) or SendTaskFailure(taskToken) to resume it. The execution can wait up to a year (Standard) without consuming compute.

Step Functions vs Celery vs Temporal vs in-app orchestration

Decision shortcut:

• AWS-native shop, want a visible auditable workflow with branching/retries → Step Functions.
• Background jobs / async tasks, no orchestration → Celery.
• Complex long-running workflows, want code not JSON, need portability → Temporal.
• Two sequential steps, no failure complexity → just call them in app code.

Common gotchas

• ResultPath default replaces the whole input — use $.something to merge instead.
• Express at-least-once — steps run more than once; idempotency is your job.
• State machine size limits — ASL definition has size limits; very large machines need to be decomposed (nested state machines).
• Per-state-transition pricing (Standard) — a workflow with thousands of states gets expensive; use Express for high-volume.
• Lambda payload limits — 256 KB between states; pass S3 references for big payloads.
• Debugging JSON — ASL is verbose and JSON has no comments; the Workflow Studio visual editor helps. Consider CDK or the aws-stepfunctions constructs to generate it.

Interview angle

• “Express vs Standard?” — Standard: up to 1 year, exactly-once, full history, per-transition pricing — for auditable business processes. Express: ≤5 min, at-least-once, CloudWatch-only history, cheap at volume — for high-throughput event processing (make steps idempotent).
• “How does error handling work?” — declarative Retry (exponential backoff on matching errors) and Catch (route to a fallback/compensation state). This makes the saga pattern declarative instead of hand-coded.
• “Explain ResultPath.” — controls where a task’s result lands in the state’s data. Default $ replaces the entire input with the result — common bug. Use $.result to keep the input and merge the result alongside.
• “What’s the callback pattern?” — waitForTaskToken: Step Functions pauses, hands out a task token, and resumes only when something external calls SendTaskSuccess/SendTaskFailure. For human approval or long async waits — can pause up to a year without compute cost.
• “Step Functions vs Celery vs Temporal?” — Step Functions for AWS-native, visible, JSON-defined orchestration; Celery for background jobs with no orchestration needs; Temporal for complex durable workflows in code with portability. Don’t use any of them for two trivial sequential steps.
• “When would you use Distributed Map?” — large-scale parallel batch over millions of items (e.g., every object in an S3 bucket), up to 10k concurrent child executions. Replaces hand-rolled Lambda + SQS fan-out.