backend / observability / 12_slo_sli_sla.md

SLO, SLI, SLA, and Error Budgets

6 interview angles 7 min read source

SLO, SLI, SLA, and Error Budgets

The Google SRE framework for turning “is the service healthy?” into a number that drives engineering and product decisions.

The four terms

Term What it is Audience
SLI (Service Level Indicator) a measured quantity: “fraction of requests succeeding” engineering
SLO (Service Level Objective) the internal target for an SLI: “99.9% of requests succeed over 28 days” engineering + product
SLA (Service Level Agreement) a contractual commitment with consequences legal / customer
Error budget (1 - SLO) × total events; how much “unavailability” you can spend engineering + product

SLO is the engineering target; SLA is what you promise externally and may pay penalties for missing. SLO is always tighter than SLA (so internal incidents don’t immediately become contractual violations).

What makes a good SLI

Properties:

  • Measurable directly from observable signals (not “user satisfaction” — measurable as 5xx rate, p95 latency, etc.).
  • Reflects user experience. “CPU usage on the DB” is a system metric, not an SLI — users don’t care about CPU.
  • Aggregatable over a time window. “good_events / total_events” works.

Common SLIs:

  • Availability: (successful_requests / total_requests) over 28 days.
  • Latency: (requests faster than 200ms / total_requests).
  • Correctness: (correctly-processed orders / total orders).
  • Freshness: (data writes visible to readers within 1 minute / total writes).

What makes a good SLO

  • Achievable — based on historical performance, not aspirational.
  • Meaningful to users — picked because users care, not because they’re easy to measure.
  • Has a defined window — “99.9% over the last 28 days” is concrete.
  • Linked to user experience — exceeding it should mean users are happy; missing should mean they noticed.

Picking too tight an SLO (99.999%) when reality is 99.5% means constant fire-fighting; you’re not delivering features, just patching.

Picking too loose (99%) when reality is 99.99% means easy SLO, customers get annoyed by the disparity.

Start where you are; aim for “what users would notice the change from.”

The 9s

SLO Downtime per year Downtime per 28-day window
99% 3.65 days 6.7 hours
99.5% 1.83 days 3.4 hours
99.9% 8.76 hours 40 minutes
99.95% 4.38 hours 20 minutes
99.99% 52 minutes 4 minutes
99.999% 5 minutes 24 seconds

99.99% means roughly one 5-minute incident a year. 99.999% means you can’t even acknowledge a page before you’ve burned the budget. Be realistic.

Error budget

(1 - SLO) × total events = the budget for failure.

If your SLO is 99.9% and you have 30M requests in 28 days, your error budget is 30k failed requests. Spend it on:

  • Real outages and incidents.
  • Risky deploys / experiments.
  • Planned maintenance.

The framework: if you have budget remaining, ship faster; if you’ve burned the budget, slow down and stabilize.

Burn rate alerts

The traditional alert: “5xx rate > 1% for 5 minutes”. Triggers on every blip; alert fatigue.

The SLO-based alert: burn rate — how fast you’re spending the error budget.

burn_rate = (actual_error_rate / (1 - SLO))

If SLO is 99.9% (error rate of 0.1%), and you’re seeing 1% errors, burn rate is 10× — at this rate you’ll consume the 28-day budget in 2.8 days.

Tiered alerts:

  • Fast burn (page immediately): burn rate ≥ 14.4 in last hour AND last 5 min — burns the month’s budget in 2 days at this rate.
  • Slow burn (ticket): burn rate ≥ 1 in last 24h AND last 1h — sustained moderate degradation.

This pages on real problems, not transient blips, and catches gradual erosion that wouldn’t trip a static threshold.

Reference: Google SRE Workbook, “Alerting on SLOs” chapter.

Multi-window multi-burn-rate

The standard:

Window Burn rate Action
1h + 5m ≥ 14.4 page
6h + 30m ≥ 6 page
24h + 2h ≥ 3 ticket
72h + 6h ≥ 1 ticket

Pages only when both windows agree (filter out short blips). Tickets for slower issues.

Latency SLOs

Availability is binary (good/bad). Latency is continuous. SLO it as a percentile:

“99% of requests complete in < 200ms over 28 days”

SLI:

good_events = requests with latency < 200ms
total_events = all requests

Then standard burn rate / error budget math.

For multi-step user flows, latency SLO on the user-visible step (the slowest one matters most).

Picking the budget consumer

The error budget is shared across:

  • Code bugs (deploy regressions).
  • Infra problems (DB outage, network).
  • Planned operations (database migration).
  • Feature experiments (load test).

Different teams want to spend it differently. The framework’s contribution: everyone agrees on the total, and you can negotiate “we want to do a risky deploy; that’ll cost N% of the budget.”

SLA vs SLO

SLO SLA
Internal/external internal target external commitment
Consequences engineering process changes (freezes, etc.) refunds, contract penalties
Tightness tighter than SLA looser than SLO
Audience engineers, PMs sales, legal, customers

If you promise 99.95% SLA, run with 99.99% SLO. The buffer absorbs incidents without violating the contract.

Negotiating an SLA: pick a number you can deliver 99% of months. Customers care about consistency more than peak performance.

What’s NOT an SLI

  • CPU utilization (a system metric, not user experience).
  • Memory usage.
  • Queue depth (proxy for problems, not a problem itself).
  • Number of pods.

These are operational signals. They drive alerts and dashboards but aren’t the thing users care about.

Common mistakes

  • Too many SLOs. Aim for 2-4 per service. More than that and no one knows which to prioritize.
  • Aspirational SLOs. Picked because they sound good, not because you can hit them. Erodes credibility.
  • Static-threshold alerts alongside SLO alerts. Double-paging on the same problem. Migrate fully.
  • Not tracking burn rate. Just reporting SLO at end of month tells you what already happened; burn rate tells you where you’re going.
  • SLO without ownership. No team responsible = no one acts on a breach.

Production rollout

  1. Measure first. What’s your actual availability over the last 90 days?
  2. Pick a slightly tighter SLO as the target — achievable but not trivial.
  3. Define one or two SLIs per service. Don’t proliferate.
  4. Set up burn-rate alerts. Page on fast burn; ticket on slow.
  5. Review monthly. Is the SLO too loose? Too tight? Are users happy?
  6. Negotiate with product. Budget remaining = ship faster. Budget burned = stabilize.

Tools

  • Sloth — open-source SLO generator for Prometheus.
  • PromQL + alerting — most common DIY path.
  • Datadog SLOs / New Relic / Honeycomb — managed SLO tooling.
  • Google Cloud Operations SLO — for GCP.

Interview angle

  • “What’s the difference between SLI, SLO, SLA?” — SLI is a measurement (e.g., success rate). SLO is the internal target on the SLI (99.9% over 28 days). SLA is the external contractual commitment, with consequences for breach. SLO is tighter than SLA.
  • “What’s an error budget?”(1 - SLO) × total events. If SLO is 99.9% and you have 30M req/28d, budget is 30k failed requests. Used to prioritize: budget remaining = ship faster, budget burned = stabilize.
  • “What’s a burn rate alert?” — alert based on how fast you’re consuming the error budget, not on a static threshold. If burn rate ≥ 14.4 (sustained 1h), you’d consume the month’s budget in 2 days — page. Filters transient blips; catches sustained issues.
  • “How tight should you set your SLOs?” — based on current performance + user expectations. Tight enough that breach correlates with user complaints; loose enough that you can occasionally do risky things without depleting the budget. Start where you are; tighten over time.
  • “What’s NOT an SLI?” — system metrics (CPU, memory, queue depth). They’re operational signals — useful for alerts and dashboards — but users don’t experience them directly. SLI must reflect user-visible outcomes.
  • “Why is 99.999% expensive?” — 24 seconds of downtime per 28 days. One deploy that goes 30 seconds bad = SLO violation. To hit it you need redundancy at every layer, near-zero-downtime deploys, and very fast incident response. The cost grows nonlinearly with each 9.