Terraform — Secrets, Testing, and CI/CD

The production-discipline layer: keeping secrets out of state and git, validating Terraform before it runs, and the safe CI/CD pipeline shape.

Secrets — the hard problem

Terraform’s awkward truth: secrets that pass through Terraform end up in the state file in plaintext. A DB password you set on an aws_db_instance, a generated key — they’re in terraform.tfstate. So:

Don’t put secrets in .tf or .tfvars committed to git

• Pass them via environment variables: TF_VAR_db_password=... → consumed by variable "db_password".
• Or pull them at plan time from a secret store with a data source:

data "aws_secretsmanager_secret_version" "db" {
  secret_id = "prod/db/master"
}
# reference: data.aws_secretsmanager_secret_version.db.secret_string

Better: don’t let Terraform handle the secret value at all

The cleanest pattern — Terraform creates the secret container, something else fills it:

• Terraform creates the aws_secretsmanager_secret (the resource), but not the value.
• The actual value is set out-of-band (a rotation Lambda, a manual one-time aws secretsmanager put-secret-value, a separate bootstrap process).
• For RDS specifically: manage_master_user_password = true lets RDS+Secrets Manager own the password entirely — Terraform never sees it.

Protect the state file

Whatever you do, secrets that do flow through Terraform are in state — so:

• Encrypt state at rest (S3 SSE/KMS).
• Tight IAM on the state bucket — reading state = reading those secrets.
• sensitive = true on variables/outputs redacts CLI output (not state storage).

Validation — catch problems before apply

A ladder of checks, fast to slow, run in this order in CI:

fmt and validate are instant and need no credentials — run them on every commit. The security scanners (tfsec/checkov) are the high-value ones for a senior context: they catch “this Terraform would create a publicly-readable S3 bucket” before it’s applied.

Testing Terraform

For most teams: fmt + validate + tflint + tfsec + a plan on module examples/ covers a lot cheaply. terraform test for critical modules. Terratest only where the cost of a bug justifies real-infrastructure tests.

The CI/CD pipeline shape

The safe pattern, and the thing to describe in an interview:

on pull request:
  terraform fmt -check
  terraform validate
  tflint
  tfsec / checkov
  terraform plan -out=tfplan       ← post the plan as a PR comment
  (human reviews the plan in the PR)

on merge to main:
  terraform apply tfplan           ← apply the SAVED plan, not a fresh one

Key principles:

• plan on PR, apply on merge — the diff is reviewed by a human before it touches infrastructure. The PR comment showing the plan is the review artifact.
• Apply the saved plan (plan -out=tfplan → apply tfplan) — guarantees apply does exactly what was reviewed; no re-diff, no drift-between-plan-and-apply surprise.
• CI authenticates via OIDC, not stored keys — GitHub Actions assumes an IAM role via OIDC federation (no long-lived AWS_ACCESS_KEY_ID secret). See the IAM advanced-patterns file.
• Promotion through environments — apply to dev → staging → prod, same module versions, gated. Prod apply often needs a manual approval step.
• State locking does its job — concurrent pipeline runs serialize on the lock instead of corrupting state.
• Prod guardrails — prevent_destroy on critical resources, policy-as-code blocking dangerous changes, manual approval for prod applies.

terraform fmt and validate locally too

Run terraform fmt and validate as a pre-commit hook so problems are caught before CI even runs — same philosophy as ruff/black for Python (see ../../26_code_quality/04_pre_commit_and_review.md).

Common gotchas

• Secrets in .tfvars in git — the classic mistake. Use TF_VAR_ env vars, secret-store data sources, or let the resource own the secret (manage_master_user_password).
• Forgetting state holds secrets — even with secrets sourced cleanly, anything that flows through a Terraform resource is in state. Encrypt it, lock down the bucket.
• apply from a fresh plan in CI — re-diffs; what was reviewed isn’t guaranteed to be what runs. Always plan -out → apply the file.
• No security scanning — tfsec/checkov catch public buckets and open security groups before they exist. Skipping them means finding out in prod.
• apply straight from a feature branch — no review of the diff. plan on PR, apply on merge.
• Long-lived AWS keys in CI — use OIDC federation.
• No prod approval gate — apply to prod should be a deliberate, approved action, not an automatic consequence of a merge.

Interview angle

• “How do you keep secrets out of Terraform?” — don’t commit them: pass via TF_VAR_ env vars or pull from Secrets Manager/SSM with a data source. Better — let Terraform create the secret container and have something else (RDS itself via manage_master_user_password, a rotation Lambda) own the value. And remember: anything flowing through a Terraform resource lands in state, so encrypt state and lock down the bucket.
• “How do you validate Terraform before applying?” — a ladder: fmt -check (formatting), validate (syntax, no credentials), tflint (provider lint), tfsec/checkov (security misconfig — public buckets, open SGs, unencrypted volumes), then plan. The security scanners are the high-value step for catching dangerous infra before it exists.
• “What does a safe Terraform CI/CD pipeline look like?” — plan on PR (posted as a comment for human review), apply on merge — and apply the saved plan file so it does exactly what was reviewed. CI authenticates via OIDC (no stored keys). Promote dev → staging → prod with a manual approval gate on prod.
• “Why apply a saved plan instead of running apply fresh?” — a fresh apply re-runs refresh+diff, so it can do more or less than the plan a human reviewed if reality changed in between. plan -out=tfplan then apply tfplan guarantees the reviewed diff is the executed diff.
• “How do you test Terraform?” — native terraform test (.tftest.hcl, plan/apply + assertions) for critical modules; tflint/tfsec for static checks; policy-as-code (OPA/Sentinel) to block dangerous patterns against the plan; Terratest for real-infrastructure assertions where a bug’s cost justifies the slow, paid tests.
• “How does CI authenticate to AWS for Terraform?” — OIDC federation: the CI job exchanges its OIDC token for short-lived AWS credentials via an IAM role scoped to the repo. No long-lived access keys stored as CI secrets.