Secrets and Supply Chain in CI

The two cross-cutting security concerns in any CI/CD pipeline. Secrets: how to give the pipeline credentials without leaking them. Supply chain: how to trust the dependencies, base images, and actions you pull in.

For platform-specific syntax: 02_github_actions.md, 03_gitlab_ci.md. This file is cross-platform principles.

Secrets — where they live

Trend: minimize stored secrets. Use OIDC for cloud; fetch app secrets at runtime from a secret manager.

Scope of secrets

Organization → Repository / Project → Environment / Branch → Job

Each level can have its own secrets. Principle: most-restrictive scope that allows the work.

Production credentials should be at the Environment level, with required reviewers. Anyone with repo write can read repository secrets; environment secrets are gated by who can approve.

Secrets must NOT

• Appear in logs.
• Be committed to the repo.
• Be passed as command-line arguments (visible in ps).
• Be sent to telemetry / error tracking unscrubbed.
• Be embedded in artifact metadata.
• Be in environment variables of child processes you don’t trust.

Common leaks:

# BAD
- run: echo "API key is $API_KEY"          # logged

# BAD
- run: curl -u user:$PASSWORD api.example   # visible in `ps`; better:
- run: curl --user user:$PASSWORD api.example  # still visible

# OK
- env:
    API_KEY: ${{ secrets.API_KEY }}
  run: ./script.sh                           # script reads $API_KEY env var

Pass via env vars; let the script consume them. Never inline secrets in commands.

OIDC — the modern cloud auth

GitHub Actions / GitLab CI                      AWS / GCP / Azure
─────────────────────────                       ─────────────────
1. Job starts; CI platform issues a JWT          ──→ (verify JWT signature,
   identifying: repo, branch, job, sha           ──→  match against IAM policy)
2. Use the JWT to call STS / AssumeRoleWithWebIdentity
3. AWS returns short-lived credentials (15 min – 12 hours)
4. Job uses those creds; they expire after the job ends

Setup (AWS example):

# Create an IAM identity provider for GitHub's OIDC issuer
aws iam create-open-id-connect-provider \
  --url https://token.actions.githubusercontent.com \
  --client-id-list sts.amazonaws.com \
  --thumbprint-list <thumbprint>

# Create an IAM role with trust policy referencing GitHub OIDC
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": { "Federated": "arn:aws:iam::<acct>:oidc-provider/token.actions.githubusercontent.com" },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringLike": {
        "token.actions.githubusercontent.com:sub": "repo:myorg/myrepo:ref:refs/heads/main"
      }
    }
  }]
}

The condition is critical — it limits which workflows can assume this role. Without it, ANY GitHub Actions workflow could assume the role.

In the workflow:

permissions:
  id-token: write
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123:role/deploy-role
          aws-region: us-east-1

No long-lived AWS keys anywhere. The credentials are issued per-job and expire automatically.

Similar patterns for GCP, Azure, HashiCorp Vault (which can validate the OIDC JWT directly).

Fetching app secrets at runtime

For runtime secrets (DB password, API keys), fetch from a secret manager rather than storing in CI:

- name: Configure AWS
  uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: arn:aws:iam::123:role/deploy-role

- name: Fetch secrets
  run: |
    DB_PASSWORD=$(aws secretsmanager get-secret-value --secret-id prod/db --query SecretString --output text)
    echo "::add-mask::$DB_PASSWORD"     # explicitly mask
    echo "DB_PASSWORD=$DB_PASSWORD" >> $GITHUB_ENV

Source of truth is the secret manager. CI is just authenticated to fetch what it needs.

This pattern centralizes secret rotation — change in the secret manager, all consumers pick up the new value on their next run.

Detecting secrets in code

Pre-commit hook + CI scan:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/gitleaks/gitleaks
    rev: v8.18.0
    hooks:
      - id: gitleaks

CI scan:

- uses: gitleaks/gitleaks-action@v2

Tools: gitleaks, truffleHog, GitGuardian, GitHub’s secret scanning (free for public repos). They scan commits for high-entropy strings matching known secret patterns (AWS keys, Stripe tokens, etc.).

When a secret leaks into git history:

1. Rotate the secret immediately. Anyone who pulled the commit has it. The leak is permanent.
2. Optionally rewrite history (git-filter-repo, BFG) and force-push — but anyone with the old clone still has the leaked value.
3. The rotation is what matters; the history scrub is cosmetic.

See ../22_git/15_dangerous_commands_safety.md.

Supply chain security

The other security concern: trusting what you pull in.

Your code ──> CI ──> [ Base image ] [ Dependencies ] [ Actions / Marketplace ]
                          ↓               ↓                    ↓
                       Docker Hub     pip / npm           GitHub Marketplace
                                       PyPI

Each is a potential entry point. Real incidents:

• event-stream (npm, 2018): maintainer transferred to malicious actor; payload targeting bitcoin wallets.
• colors.js / faker.js (npm, 2022): author sabotaged own packages.
• xz-utils backdoor (2024): long-running social engineering attack injected malicious code into a widely-used library.
• PyPI typosquatting: malicious packages named like numpyy, requests2.

Pinning dependencies

# requirements.txt — pinned
requests==2.31.0
django==5.0.1

vs

# requirements.txt — floating
requests>=2.30
django>=5.0

Floating versions = unpredictable builds, vulnerable to surprise dependency changes. Pin exact versions. Use a lockfile (requirements.lock, poetry.lock, pdm.lock) for transitive dependencies.

For Python:

pip-compile requirements.in --output-file requirements.txt --generate-hashes

--generate-hashes adds hash assertions; pip refuses to install if the hash doesn’t match. Defends against package re-publishing attacks (rare but real).

Dependency scanning

Continuous scans for known vulnerabilities:

CI integration:

- name: Audit Python deps
  run: pip-audit -r requirements.txt

Set CI to fail on high-severity issues; periodically review medium-severity.

SBOM — Software Bill of Materials

A formal list of every component in your software. Standards: SPDX, CycloneDX.

# Generate SBOM for Python project
pip install cyclonedx-bom
cyclonedx-py -o bom.json

# Generate SBOM for container image
syft ghcr.io/myorg/myapp:1.2.3 -o cyclonedx-json > bom.json

Increasingly required by enterprise customers and regulators (US Executive Order 14028). Ship the SBOM with releases.

Signed artifacts and attestations

Beyond pinning: cryptographically verify the artifact is what the build produced.

# Sign a container image with cosign (keyless, uses OIDC)
cosign sign ghcr.io/myorg/myapp:1.2.3

# Verify at deploy time
cosign verify --certificate-identity=... ghcr.io/myorg/myapp:1.2.3

Modern admission controllers (Kubernetes’ Sigstore Policy Controller) refuse to run unsigned images. Defense against compromised registries.

Pinning third-party CI actions

For high-security pipelines:

# BAD — tags can move
- uses: actions/checkout@v4

# BETTER — pinned to immutable SHA
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29  # v4.1.1

Tags are mutable; an action author can re-tag v4 to point at malicious code. SHA pins are immutable.

Trade-off: harder to keep up to date. Use Dependabot for actions:

# .github/dependabot.yml
version: 2
updates:
  - package-ecosystem: github-actions
    directory: /
    schedule:
      interval: weekly

Dependabot opens PRs to update pinned SHAs.

Container image scanning

- name: Build image
  run: docker build -t myapp:${{ github.sha }} .

- name: Scan with Trivy
  uses: aquasecurity/trivy-action@master
  with:
    image-ref: myapp:${{ github.sha }}
    severity: CRITICAL,HIGH
    exit-code: 1                    # fail CI on findings

Scans the image’s OS packages and language deps for known CVEs. Run on every build; fail on high severity.

Also: minimize base image. Alpine, distroless, scratch images dramatically reduce attack surface vs full Ubuntu.

Provenance — SLSA levels

SLSA (Supply-chain Levels for Software Artifacts) is a framework for build trustworthiness:

Most teams aim for SLSA 2-3. SLSA 4 is for high-stakes (kernel, cryptographic libraries).

GitHub Actions has built-in SLSA Level 3 provenance generation via actions/attest-build-provenance.

Reproducible builds

The same source → the same binary, byte-for-byte. Allows independent verification.

# Pin everything: base image, build tool version, dependency versions, build flags
docker buildx build \
  --build-arg SOURCE_DATE_EPOCH=$(git log -1 --pretty=%ct) \
  --no-cache \
  -t myapp:reproducible .

SOURCE_DATE_EPOCH controls timestamps. Without it, every build differs by build-time. Reproducible builds matter for SLSA 4 and for some security audits.

Common pitfalls

• Secrets in command-line args — visible in ps and CI logs.
• pull_request_target running untrusted code with secrets — classic GitHub Actions security hole.
• Long-lived AWS access keys in CI — should be OIDC.
• No SHA pinning of third-party actions — supply chain risk.
• docker pull image:latest in pipelines — tag can move; pin to digest (image@sha256:...).
• Sharing one Vault token across all CI jobs — least-privilege violation. Per-environment.
• No secret rotation cadence — leaked secret stays valid forever. Rotate quarterly minimum.
• Public repo + sensitive secrets — anyone can submit a PR that exfiltrates secrets. Use environment protections.

Common interview confusions

• “GitHub Secrets are encrypted, so they can’t leak.” — encrypted at rest, masked in logs, BUT runnable workflows can read them in plaintext during execution and can be tricked into exfiltration.
• “Pinning actions to a tag is fine.” — tags can move. SHA pins are immutable.
• “OIDC is harder than storing keys.” — initial setup is, but operationally it’s simpler (no rotation, no secret sprawl).

Interview angle

• “How do you handle secrets in CI?” — platform secret store for CI-only secrets (API tokens), with scope (org / repo / environment). For cloud auth, use OIDC instead of stored keys. For app secrets, fetch from a secret manager (AWS Secrets Manager, Vault) at runtime so source of truth is centralized.
• “What’s OIDC in CI and why use it?” — CI platform issues a short-lived JWT identifying the workflow; cloud provider validates and grants temporary credentials. No long-lived access keys stored in CI. Credentials expire after the job; rotation handled automatically.
• “How would you prevent secrets from leaking in CI logs?” — never echo secrets, pass via env vars not command args, use the platform’s masking (::add-mask:: in GitHub), audit your error reporting / Sentry config for unscrubbed env vars.
• “What’s supply chain security in CI?” — trust the inputs: pinned dependency versions (lockfile + hashes), pinned base images (digest, not tag), pinned third-party CI actions (SHA, not tag), scan deps and images for CVEs, sign artifacts (cosign), generate SBOMs.
• “What’s SLSA?” — Supply-chain Levels for Software Artifacts; framework for build trustworthiness from level 0 (none) to 4 (hermetic + reproducible + two-person review). GitHub Actions can attest to level 3 natively.
• “You discover a secret was committed to a public repo. What do you do?” — rotate the secret immediately. Even if you rewrite git history, the secret was public; anyone who pulled has it. History rewrite is cosmetic; rotation is what saves you.
• “Why pin third-party GitHub Actions to a SHA, not a tag?” — tags are mutable; an action’s author can re-tag v4 to point at malicious code. SHA pins are immutable. Combine with Dependabot to keep them updated safely.