Pipeline Design Patterns

The “how” of structuring CI/CD pipelines beyond the basic linear flow. Caching, matrix builds, monorepo strategies, reusable workflows, conditional execution. Interview questions probe whether you’ve designed pipelines that scale beyond “one repo, one branch, one deploy.”

Fan-out / fan-in (DAG)

The default linear pipeline:

build → test → deploy   (sequential)

Becomes:

         ┌──► unit_tests ──┐
build ──┼──► lint ───────┤──► deploy
         └──► type_check ──┘

Independent jobs run in parallel; dependent jobs wait via needs: (GitHub Actions) or needs: (GitLab). Net pipeline time = longest path, not sum of all jobs.

For most pipelines, the obvious split: lint + unit tests + type check + security scan all in parallel after build; deploy waits for all.

Matrix builds — test the cross-product

strategy:
  matrix:
    python: ["3.12", "3.13", "3.14"]
    os: [ubuntu-latest, macos-latest, windows-latest]
    include:
      - python: "3.14"
        os: ubuntu-latest
        with-coverage: true
    exclude:
      - python: "3.12"
        os: windows-latest

9 combos × OS, with one extra cell adding coverage, and one cell excluded → 8 parallel jobs.

Use cases:

• Multiple Python versions for libraries.
• Multiple OSes for tools.
• Multiple databases (Postgres 15/16/17/18) for ORM compatibility.
• Multiple CPU architectures (x86_64 + arm64) for container images.

Cost: each matrix cell is a parallel job. Large matrices burn CI minutes fast.

Caching strategies

Cache the slow things:

- uses: actions/cache@v4
  with:
    path: ~/.cache/pip
    key: pip-${{ runner.os }}-${{ hashFiles('requirements.txt') }}
    restore-keys: |
      pip-${{ runner.os }}-

key is the precise cache. restore-keys is partial-match fallback — even if requirements.txt changes, you reuse most of the cache.

Cache anti-patterns:

• Including ${{ github.sha }} in the key → fresh cache every commit. Defeats caching.
• Caching node_modules without including the package-manager version → can break if Node updates.
• Cache too large (>5 GB) → slower to upload/download than to rebuild.

Docker BuildKit cache

For container builds, BuildKit’s layer cache is the biggest single optimization:

- uses: docker/setup-buildx-action@v3
- uses: docker/build-push-action@v5
  with:
    context: .
    push: true
    tags: ghcr.io/org/app:${{ github.sha }}
    cache-from: type=registry,ref=ghcr.io/org/app:buildcache
    cache-to: type=registry,ref=ghcr.io/org/app:buildcache,mode=max

cache-from and cache-to push the build cache to a registry. Subsequent builds pull cached layers — only changed layers rebuild.

For multi-stage builds, mode=max exports cache for every stage. Slower upload but maximum reuse.

Path-filtered triggers (monorepo)

on:
  push:
    paths:
      - "services/api/**"
      - "shared/**"
      - ".github/workflows/api-ci.yml"

The workflow only runs when relevant paths change. Saves CI minutes on commits to unrelated directories.

For complex monorepos with dynamic dependencies, use the dorny/paths-filter action:

- uses: dorny/paths-filter@v3
  id: changes
  with:
    filters: |
      api:
        - 'services/api/**'
        - 'shared/**'
      worker:
        - 'services/worker/**'
        - 'shared/**'

- if: steps.changes.outputs.api == 'true'
  run: pytest services/api/

- if: steps.changes.outputs.worker == 'true'
  run: pytest services/worker/

Conditionally run only the affected pieces.

Affected-graph for monorepos

For larger monorepos with explicit dependency graphs:

# nx (JS monorepo tool)
nx affected --target=test --base=origin/main

# bazel
bazel query 'rdeps(//..., set(changed_files))' --output=label

# pants (Python)
pants --changed-since=origin/main test

These tools compute the impact graph from your code: changed files → affected packages → affected tests. Run only the affected.

For pure Python monorepos without these tools: build your own with git diff + a manifest.

Reusable workflows / composite actions

For patterns repeated across many repos:

# .github/workflows/python-test-template.yml
on:
  workflow_call:
    inputs:
      python-version: { required: true, type: string }
      run-coverage: { default: false, type: boolean }

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: ${{ inputs.python-version }}
          cache: pip
      - run: pip install -r requirements.txt
      - run: pytest --cov=src
        if: inputs.run-coverage
      - run: pytest
        if: ! inputs.run-coverage

Caller:

jobs:
  test:
    uses: org/ci-templates/.github/workflows/python-test-template.yml@v1
    with:
      python-version: "3.14"
      run-coverage: true

One central template; many repos call it. Update once → all repos get the new behavior.

GitLab equivalent: include: from a templates repo.

For smaller reuse: composite actions (GitHub) or extends: (GitLab) — partial reuse within a workflow.

Conditional execution

- name: Deploy to prod
  if: github.ref == 'refs/heads/main' && github.event_name == 'push'
  run: deploy.sh

- name: Comment on PR
  if: github.event_name == 'pull_request' && failure()
  uses: actions/github-script@v7
  with:
    script: |
      github.rest.issues.createComment({
        issue_number: context.issue.number,
        owner: context.repo.owner,
        repo: context.repo.repo,
        body: 'Build failed; please check the logs.'
      });

Common conditions:

• Branch / tag.
• Event type (push vs PR).
• Previous step’s status (success(), failure(), always()).
• File change patterns.
• Manual approval gates.

Concurrency control

concurrency:
  group: deploy-${{ github.ref }}
  cancel-in-progress: false

Only one workflow with this group runs at a time. Subsequent triggers queue (or cancel if cancel-in-progress: true).

Common patterns:

For deploys: never cancel in progress (let the deploy finish or fail explicitly). For PR CI: cancel in progress to save runtime when commits come fast.

Multi-environment promotion

jobs:
  build: { ... }

  deploy_dev:
    needs: build
    environment: dev
    runs-on: ubuntu-latest
    steps: [deploy.sh dev]

  deploy_staging:
    needs: deploy_dev
    environment: staging          # requires approval if env protection set
    runs-on: ubuntu-latest
    steps: [deploy.sh staging]

  smoke_tests:
    needs: deploy_staging
    runs-on: ubuntu-latest
    steps: [pytest tests/smoke/]

  deploy_prod:
    needs: smoke_tests
    environment: production       # required reviewers gate
    runs-on: ubuntu-latest
    steps: [deploy.sh prod]

Promotion through environments. Each environment can have its own protection rules (required reviewers, branch restrictions, secret scope).

Test sharding

For large test suites, shard across parallel jobs:

strategy:
  matrix:
    shard: [1, 2, 3, 4, 5, 6, 7, 8]

steps:
  - run: pytest --shard-id=${{ matrix.shard }} --num-shards=8

Plugins like pytest-split distribute tests across shards based on previous timings. 8x parallel = roughly 1/8 wall-clock time (minus setup overhead per shard).

Trade-off: each shard pays setup costs (checkout, install deps). Net speedup is sublinear. Sweet spot is often 4-8 shards for a multi-thousand-test suite.

Build matrix vs explicit jobs

# Matrix — concise, parallel
strategy:
  matrix:
    package: [api, worker, scheduler]
steps:
  - run: pytest packages/${{ matrix.package }}

# vs explicit — clearer separation, different config per job
test_api:
  steps: [...]
test_worker:
  steps: [...]

Matrix wins when jobs are nearly identical. Explicit wins when each has meaningfully different setup. Don’t shoehorn into a matrix when explicit is clearer.

Failing fast vs running all

strategy:
  fail-fast: true   # default — cancel siblings on first failure
  # OR
  fail-fast: false  # let all matrix cells run to completion

For “I want to know which Python versions break”: false. For “save money on PR CI”: true.

Artifact passing between jobs

build:
  steps:
    - run: python -m build
    - uses: actions/upload-artifact@v4
      with:
        name: dist
        path: dist/

test:
  needs: build
  steps:
    - uses: actions/download-artifact@v4
      with:
        name: dist
        path: dist/
    - run: pip install dist/*.whl && pytest

Pattern: build once, test/deploy many times. Avoids rebuilding for each downstream job.

For container images, push to a registry instead — artifacts are slow above a few hundred MB.

Service containers vs Testcontainers

# Service container — orchestrated by CI platform
services:
  postgres:
    image: postgres:16
    env: { POSTGRES_PASSWORD: test }

# Testcontainers — orchestrated from within the test code
- run: pytest    # tests use testcontainers-python to spin up Postgres on demand

Testcontainers makes “the test that runs in CI” match “the test that runs locally.” Worth the extra dependency for non-trivial test setups.

Pipeline as code — modular structure

For large monorepos, one giant workflow file becomes unmaintainable. Split:

.github/workflows/
├── ci.yml                    # top-level orchestrator
├── lint.yml                  # reusable lint workflow
├── test-python.yml           # reusable test workflow
├── build-and-push.yml        # container build
└── deploy.yml                # deploy

ci.yml invokes the others via uses:. Each file is focused; reusable across triggers.

GitLab equivalent: include: to split .gitlab-ci.yml across files.

Common pitfalls

• Sequential pipelines for parallel work — needs: everywhere, no fan-out. Pipeline takes 30 min when 10 would do.
• Cache keys that always miss — including timestamps or SHAs in the key. Always rebuilds from scratch.
• One giant workflow file — 2000-line YAML that nobody can edit safely. Split into reusable workflows.
• No path filters on monorepo workflows — every commit triggers every workflow. Wasteful.
• fail-fast: true on cross-version matrix — Python 3.12 fails first, you never see if 3.14 also fails. Use false for diagnostic matrices.
• Tests running before lint — lint fails fast (seconds); tests fail slow (minutes). Run lint first.
• No artifact for build output — every downstream job rebuilds. Build once, share via artifact.
• Sharding without timing data — shards finish at wildly different times; pipeline is bottlenecked by slowest. Use timing-aware sharders (pytest-split).

Common interview confusions

• “More parallelism is always better.” — diminishing returns due to setup overhead per job. Sweet spot exists per workload.
• “Matrix builds replace explicit jobs.” — matrix is great when jobs are identical-shape with parameter variation. Explicit when each is meaningfully different.
• “Caching always speeds things up.” — bad cache keys can miss every time, costing more than no cache (waste storage + cache-miss-then-restore overhead).

Interview angle

• “How would you speed up a 30-minute pipeline?” — profile first (where’s the time?). Then: parallelize independent jobs (fan-out/fan-in), cache dependencies, shard slow test suites, use built-in caching in setup-* actions, pre-bake runner images for self-hosted, build container layers with BuildKit cache.
• “How do you structure a monorepo’s CI?” — path-filter triggers so unrelated changes don’t trigger every workflow. Affected-graph tools (nx, bazel, pants) for complex dependency graphs. Per-package CI workflows triggered by path. Reusable workflow templates for common patterns.
• “What’s a matrix build and when do you use one?” — strategy.matrix runs the same job with parameter variations in parallel. Use for cross-version testing (Python 3.12/3.13/3.14), multi-OS, multi-DB compatibility. Don’t use when each job is meaningfully different.
• “How do you share build output between jobs?” — actions/upload-artifact + actions/download-artifact in GitHub; artifacts: and needs: in GitLab. For containers, push to a registry instead.
• “What’s concurrency: for in workflows?” — limit how many workflows of a kind run simultaneously. Common uses: one deploy per environment at a time (no cancel-in-progress), one CI per PR with cancel-in-progress: true to save runtime on fast pushes.
• “How do you test against a real Postgres in CI?” — services: block in the workflow (sidecar container reachable on localhost) or Testcontainers (orchestrated by test code). Testcontainers makes local and CI behave identically.
• “How do you avoid duplicating CI config across 30 repos?” — reusable workflows in a templates repo, called via uses: org/ci-templates/.github/workflows/X.yml@v1. Or GitLab include: from a templates project. Centralized maintenance, version-pinned consumption.