backend / iac / terraform / 03_workspaces_and_environments.md

Terraform — Workspaces and Environment Strategy

5 interview angles 5 min read source

Terraform — Workspaces and Environment Strategy

How to run the same infrastructure config for dev / staging / prod without copy-pasting — and the interview-relevant debate of workspaces vs separate directories.

The problem

You have one infrastructure definition. You need it deployed three times — dev, staging, prod — with different sizes, counts, and (critically) separate state so a plan in dev cannot possibly touch prod. How do you organize that?

Approach 1 — Terraform workspaces

terraform workspace creates multiple named states from the same config and same backend:

terraform workspace new staging
terraform workspace new prod
terraform workspace select prod
terraform apply   # uses the "prod" state, same .tf files

Inside the config you branch on terraform.workspace:

locals {
  instance_count = terraform.workspace == "prod" ? 5 : 1
  instance_type  = terraform.workspace == "prod" ? "m5.large" : "t3.micro"
}

State is stored at a workspace-suffixed key in the same backend (e.g. env:/prod/...).

Pros: zero directory duplication; one config to maintain.

Cons (and why many teams avoid workspaces for environments):

  • Same backend — all environments’ state in one bucket; weaker isolation. The blast radius of a backend-level mistake spans every environment.
  • terraform.workspace conditionals scatter environment logic through the config — easy to miss a branch, easy to apply prod settings to dev.
  • Easy to apply to the wrong environment — workspace select is stateful; forget to switch and you apply to prod thinking you’re in dev. There’s no structural guard.
  • Workspaces were really designed for short-lived parallel copies (per-feature-branch ephemeral environments), not long-lived prod/staging.

Approach 2 — separate directories per environment (the common production choice)

environments/
  dev/
    main.tf          # calls shared modules with dev values
    backend.tf       # backend "s3" { key = "dev/terraform.tfstate" }
    terraform.tfvars # dev-specific values
  staging/
    main.tf
    backend.tf       # key = "staging/terraform.tfstate"
    terraform.tfvars
  prod/
    main.tf
    backend.tf       # key = "prod/terraform.tfstate"
    terraform.tfvars
modules/             # shared building blocks
  vpc/
  app/

Each environment is a root module with its own backend / own state file (separate key, often a separate bucket for prod). The shared modules/ keep it DRY; the per-environment .tfvars and main.tf supply the differences.

Pros:

  • Strong isolation — prod has its own state file (and ideally its own bucket / account). A mistake in dev/ structurally cannot reach prod state.
  • cd environments/prod && terraform apply — your location is the environment; no stateful “did I switch workspace?” footgun.
  • Per-environment backends — prod state can have stricter IAM, separate KMS key, even a separate AWS account.
  • Environment differences live in .tfvars (data), not terraform.workspace conditionals (scattered logic).

Cons: some structural duplication (each environment has a main.tf and backend.tf) — but the modules are shared, so it’s wiring, not logic.

The interview answer

“Workspaces are good for ephemeral parallel environments — a temporary copy per feature branch. For long-lived dev/staging/prod, I prefer separate directories with separate backends: stronger state isolation, no ‘wrong workspace’ footgun, and prod can have its own bucket/account with tighter access. Shared modules keep it DRY.”

That nuance — workspaces for ephemeral, directories for permanent — is what they’re listening for.

Passing environment values

With the directory approach, environment differences are data in .tfvars:

# environments/prod/terraform.tfvars
environment    = "prod"
instance_count = 5
instance_type  = "m5.large"
db_multi_az    = true

# environments/dev/terraform.tfvars
environment    = "dev"
instance_count = 1
instance_type  = "t3.micro"
db_multi_az    = false
cd environments/prod
terraform apply -var-file=terraform.tfvars   # (auto-loaded if named terraform.tfvars)

Secrets do not go in .tfvars committed to git — they come from environment variables (TF_VAR_db_password), or are pulled from Secrets Manager / SSM via data sources at plan time.

Per-environment AWS accounts

The strongest isolation: each environment is a separate AWS account (via AWS Organizations). Prod resources literally cannot be touched by dev credentials. The Terraform provider block assumes a per-environment role:

provider "aws" {
  assume_role { role_arn = "arn:aws:iam::PROD_ACCOUNT_ID:role/terraform" }
}

Common in mature setups; worth mentioning as “the gold standard for blast-radius control.”

State-key namespacing

Within a backend, the key path namespaces state — and you split not just by environment but by component:

prod/network/terraform.tfstate     # VPC, subnets, routing
prod/data/terraform.tfstate        # RDS, ElastiCache
prod/app/terraform.tfstate         # ECS services, ALB

Why: a smaller state file = faster plan, smaller blast radius, and the lock only serializes that component. Cross-component references use terraform_remote_state data sources or (better) published outputs / SSM parameters. Don’t put the entire org in one state file.

Common gotchas

  • Workspaces for prod/staging — the workspace select footgun and shared-backend weak isolation. Fine for ephemeral, risky for permanent.
  • One state file for everything — slow plans, huge blast radius, lock contention. Split by environment and component.
  • Secrets in .tfvars in git — use TF_VAR_ env vars or pull from a secret store.
  • Drift between environments — dev and prod slowly diverge because changes were applied to one and not the other. Shared modules + a promotion pipeline (apply to dev → staging → prod from the same module version) keep them aligned.
  • Forgetting -var-file — if your tfvars isn’t named terraform.tfvars/*.auto.tfvars, it isn’t auto-loaded; an apply without it uses defaults.

Interview angle

  • “Workspaces or separate directories for environments?” — separate directories with separate backends for long-lived dev/staging/prod: stronger state isolation, no stateful ‘wrong workspace’ footgun, prod gets its own bucket/account. Workspaces are better suited to ephemeral parallel environments (per-feature-branch). Shared modules keep the directory approach DRY.
  • “Why is one giant state file bad?” — slow plan (re-reads everything), large blast radius (one mistake risks everything), and the lock serializes all infra work. Split state by environment and by component (network / data / app).
  • “How do environment differences get expressed?” — as data in per-environment .tfvars (sizes, counts, flags), feeding shared modules — not as terraform.workspace conditionals scattered through the config. Secrets come from TF_VAR_ env vars or a secret store, never committed tfvars.
  • “What’s the strongest environment isolation?” — separate AWS accounts per environment via Organizations; the Terraform provider assumes a per-environment role. Prod resources are structurally unreachable with dev credentials.
  • “How do you keep dev and prod from drifting apart?” — shared modules pinned to versions, plus a promotion pipeline that applies the same module version through dev → staging → prod. Ad-hoc applies to one environment only are how drift starts.
Previous
Terraform — Modules and Composition