backend / message queues / temporal / 02_workflows_vs_activities.md

Workflows vs Activities

4 interview angles 5 min read source

Workflows vs Activities

The single most important Temporal distinction. Get this wrong and your workflows break in non-obvious ways during replay.

The split

Workflow Activity
Role orchestration — decide what runs and when the actual work — call APIs, DBs, files
Must be deterministic yes no
Can do I/O no (except via activities) yes — anything
Survives crashes yes (replayed from history) no (retried by workflow)
Can sleep for days yes (workflow.sleep(timedelta(days=30))) no (use heartbeats for long activities, but not days)
Can use random / time only via workflow.random() / workflow.now() freely
Side effects none allowed this is where side effects live

The mental model: workflow code is a recipe; activities are the cooking. Reading the recipe twice gives the same instructions. Cooking the same dish twice may give different results.

Why determinism

Temporal recovers workflow state by replaying the workflow code against the event history. Every decision the workflow made — which activity to call, what to pass it — must be reproducible from the history alone.

# WRONG — non-deterministic
@workflow.defn
class BadWorkflow:
    @workflow.run
    async def run(self):
        if random.random() < 0.5:        # different value on replay!
            await workflow.execute_activity(path_a, ...)
        else:
            await workflow.execute_activity(path_b, ...)

On the original run, suppose random() returns 0.3 and path_a runs. The history records: “executed path_a, got result X”. On replay, random() returns 0.8 — the workflow now wants to call path_b, but the history says path_a was called. Non-determinism error: workflow execution panics.

# RIGHT — deterministic
@workflow.defn
class GoodWorkflow:
    @workflow.run
    async def run(self):
        # workflow.random() uses a seeded PRNG; same value on every replay
        if workflow.random().random() < 0.5:
            await workflow.execute_activity(path_a, ...)
        else:
            await workflow.execute_activity(path_b, ...)

Things forbidden in workflow code

Forbidden Use instead
time.time(), datetime.now() workflow.now()
random.random(), uuid.uuid4() workflow.random(), workflow.uuid4()
requests.get(...), any HTTP an activity
File I/O, DB queries an activity
Threading, multiprocessing structured concurrency via asyncio.gather
os.environ reads at runtime pass values as workflow input
asyncio.sleep workflow.sleep(timedelta(...))
Mutable global state workflow instance state only

The Python SDK ships with a sandbox that intercepts many of these and raises at import or call time. You can mark trusted modules as passthrough.

Activities — everything you’d actually want to do

Activities are normal Python functions. They can fail, retry, time out, be cancelled.

from temporalio import activity
from temporalio.exceptions import ApplicationError

@activity.defn
async def charge_card(order_id: str, amount: int) -> str:
    info = activity.info()
    activity.logger.info(f"attempt {info.attempt} for {order_id}")
    try:
        return await stripe.charge(order_id, amount)
    except StripeRateLimitError:
        # Retryable — Temporal will retry per the policy
        raise
    except StripeInvalidCardError as e:
        # Non-retryable — fail fast
        raise ApplicationError("Invalid card", non_retryable=True) from e

Retry policy

Configured at the call site, not the activity:

from temporalio.common import RetryPolicy

await workflow.execute_activity(
    charge_card,
    args=[order_id, 1000],
    start_to_close_timeout=timedelta(seconds=30),
    retry_policy=RetryPolicy(
        initial_interval=timedelta(seconds=1),
        maximum_interval=timedelta(minutes=1),
        backoff_coefficient=2.0,
        maximum_attempts=5,
        non_retryable_error_types=["InvalidCardError"],
    ),
)

Timeouts

Four timeouts; pick deliberately:

Timeout Meaning
schedule_to_start how long the task can sit in queue before a worker picks it up
start_to_close how long the activity has to run once started (the one you usually set)
schedule_to_close end-to-end including retries
heartbeat for long activities, how often the activity must call heartbeat() to prove it’s alive

For long-running activities, heartbeat regularly so a worker crash mid-activity can be detected and retried:

@activity.defn
async def process_large_file(path: str):
    for i, chunk in enumerate(read_chunks(path)):
        process(chunk)
        activity.heartbeat(i)  # also persists progress for resume

Signals, queries, updates

Workflows aren’t just call-and-wait. They can be interacted with while running.

@workflow.defn
class SubscriptionWorkflow:
    def __init__(self):
        self._cancelled = False
        self._plan = "basic"

    @workflow.run
    async def run(self, user_id: str):
        while not self._cancelled:
            await workflow.execute_activity(charge_monthly, user_id, self._plan, ...)
            await workflow.sleep(timedelta(days=30))

    @workflow.signal
    def cancel(self):                      # fire-and-forget, no return value
        self._cancelled = True

    @workflow.signal
    def change_plan(self, new_plan: str):
        self._plan = new_plan

    @workflow.query
    def current_plan(self) -> str:          # read-only, must not mutate
        return self._plan

Then from any client:

handle = client.get_workflow_handle("subscription-123")
await handle.signal(SubscriptionWorkflow.change_plan, "premium")
plan = await handle.query(SubscriptionWorkflow.current_plan)

Signals mutate state. Queries read state synchronously without writing history. Updates (newer feature) are like signals but return a result.

continue_as_new

Workflows have a history size cap (~50k events by default). Long-running workflows (e.g., subscription that runs forever) hit this. Solution: at a clean boundary, start fresh:

@workflow.run
async def run(self, state: SubscriptionState):
    for _ in range(100):  # process 100 cycles, then restart
        await charge_and_wait(state)
    workflow.continue_as_new(state)  # new execution, fresh history, same workflow ID

Common bugs

  • “Workflow task failure: non-determinism” — you used a non-deterministic call in workflow code. Look for datetime, random, requests, threading.
  • “Activity not registered” — worker started without the activity passed to its activities=[...] list.
  • “Workflow worker not picking up tasks” — task queue name mismatch between starter and worker.
  • Activity retries forever — you didn’t set maximum_attempts and the error isn’t marked non-retryable.
  • Signal arrives but doesn’t trigger logic — workflow is awaiting something blocking; use workflow.wait_condition(lambda: self._something_changed) to react to signals.

Interview angle

  • “Why split workflow code from activity code?” — workflows must be deterministic for replay; activities do all the side effects. The split makes durability mechanical instead of requiring the developer to checkpoint manually.
  • “What happens if a worker dies mid-activity?” — the activity is retried per its retry policy on another worker (after the start_to_close timeout fires or the worker crashes detectably). If the activity was non-idempotent and partially completed, that’s on the application to design idempotently.
  • “How do you handle a workflow that needs to wait 30 days?”await workflow.sleep(timedelta(days=30)). The workflow is unloaded from memory; Temporal wakes it up at the scheduled time. No worker resources are held during the wait.
  • “What’s continue_as_new for?” — to reset workflow history while preserving logical continuity, avoiding the ~50k event history cap on long-running workflows.