Model-provider abstraction and LLM resilience

Production LLM code that calls one vendor’s SDK directly is a liability. This is the layer that makes the model a swappable dependency and the call path survivable — and it’s ordinary backend engineering applied to a flaky, expensive, non-deterministic dependency.

Why abstract the provider

Models change under you. A provider deprecates a version, or a better one ships next month.
Routing by difficulty is the main cost lever — cheap model for easy requests, frontier for hard.
Fallback across vendors when one has an outage or rate-limits you.
Data residency may force a different deployment (Bedrock, Azure OpenAI, Vertex) per region or tenant.
Testing. A fake provider makes tests fast and deterministic.

The failure mode of not doing it: provider-specific message shapes, token fields and error classes leak into business logic, and swapping models becomes a refactor instead of a config change.

The interface

Keep it narrow. The temptation is to model every provider feature; resist it.

from typing import Protocol

class LLMResponse(BaseModel):
    text: str
    input_tokens: int
    output_tokens: int
    model: str
    finish_reason: Literal["stop", "length", "tool_call", "filtered"]
    cost_usd: float

class LLMProvider(Protocol):
    async def complete(
        self, messages: list[Message], *, tools: list[Tool] | None = None,
        max_tokens: int = 1024, temperature: float = 0.0,
    ) -> LLMResponse: ...

Normalising tokens, cost and finish reason is the part that pays off — those feed your metrics, budgets and retry logic, and every provider names them differently.

Don’t abstract away tool calling. Structured output and tool schemas differ enough between providers that a lowest-common-denominator wrapper loses the feature you most need. Normalise the request and response shapes; keep the capability.

LiteLLM does this off the shelf across 100+ providers with a unified OpenAI-style interface, plus routing, fallbacks and budget tracking. Reaching for it rather than hand-rolling is a defensible answer — the argument for your own thin layer is when you want a domain-specific interface rather than a generic one.

Routing

The largest cost lever in most LLM products.

async def route(request: Request) -> LLMProvider:
    if request.requires_reasoning:
        return providers["frontier"]
    if request.classification_only:
        return providers["small"]
    return providers["default"]

Classify first — by task type, input length, or a cheap classifier — then send the easy majority to a small model and escalate only what needs it. Measure the escalation rate; if it’s 90%, your classifier isn’t earning its cost.

Resilience

An LLM API fails more than a typical internal service. Treat it as an unreliable third party, because it is.

@retry(
    retry=retry_if_exception_type((RateLimitError, APIConnectionError, InternalServerError)),
    wait=wait_exponential_jitter(initial=1, max=30),
    stop=stop_after_attempt(4),
    reraise=True,
)
async def call_with_retry(provider: LLMProvider, messages) -> LLMResponse:
    async with asyncio.timeout(30):
        return await provider.complete(messages)

Error	Do
429 rate limit	back off longer; honour `Retry-After` if present
5xx	retry with jitter
4xx (bad request)	don’t retry — fix the request
context length exceeded	truncate or summarise, then retry once
content filtered	don’t retry the same input
timeout	retry, but count it against a total wall-clock budget

Jitter is not optional. Synchronised retries after a provider blip produce a thundering herd that extends the outage. See ../../system_design/02_resilience/01_timeouts_retries_backoff.md.

Fallback and degradation

async def complete_with_fallback(messages) -> LLMResponse:
    for provider in (primary, secondary):
        try:
            return await call_with_retry(provider, messages)
        except (ProviderUnavailable, RateLimitError):
            log.warning("falling back", extra={"from": provider.name})
    return cached_or_degraded_response(messages)      # never just 500

Degradation options, in order of preference: a cached similar answer, a smaller/faster model, a template response, or an honest “this feature is temporarily unavailable”. Deciding this in advance is the difference between a degraded feature and an incident.

Circuit breaker on a consistently failing provider so you stop paying the timeout on every request. See ../../system_design/02_resilience/02_circuit_breakers_and_bulkheads.md.

Caching

Three distinct layers, often confused:

Layer	Key	Saves
Exact-response cache	hash of (model, prompt, params)	the whole call
Provider prompt cache	shared prompt prefix	input token cost, latency
Semantic cache	embedding similarity of the query	the call, approximately

Exact caching is free money for repeated queries — internal tools and documentation assistants repeat heavily. Requires temperature=0 to be meaningful.

Prompt caching is the provider-side one: put stable content (system prompt, tool definitions, few-shot examples) first so the prefix is reusable, and variable content last. Discounted input tokens and lower latency, and it costs nothing but prompt ordering discipline. See ../06_transformers_llm/05_kv_cache.md.

Semantic caching returns a cached answer for a similar query. Use with care: “what’s the refund policy for EU orders” and “…for US orders” are similar and have different answers. Set the threshold conservatively and exclude anything where precision matters.

Rate limiting and budgets

You are rate-limited on requests and tokens per minute, so a queue that only counts requests will still trip the token limit.

class TokenBucket:
    """Limit on estimated tokens, not just call count."""
    async def acquire(self, estimated_tokens: int) -> None: ...

Enforce a per-tenant budget so one customer can’t consume the shared quota, and a global daily cap with an alert well before it. A runaway agent loop is the usual cause of a surprise bill — see ../10_agents_orchestration/07_agent_failure_modes.md.

Pin the version

MODEL = "provider-model-name-2026-05-01"    # pinned, never a floating alias

Pointing production at a floating alias means the provider can change your behaviour with no deployment on your side. Pin it, stamp the version on every response in logs, and re-run the eval suite before adopting a new one. See ../13_evaluation/04_online_eval_and_experiments.md.

Interview angle

“Why wrap the provider SDK?” — so the model is a swappable dependency. It enables routing by difficulty (the main cost lever), cross-vendor fallback, region-specific deployments, and fast deterministic tests. Without it, provider-specific shapes leak into business logic.
“What do you normalise in the response?” — text, input and output tokens, model version, finish reason and cost. Those feed metrics, budgets and retry decisions, and every provider names them differently. Don’t flatten tool calling into a lowest common denominator.
“How do you handle failures?” — classify the error: back off longer on 429 honouring Retry-After, retry 5xx with exponential backoff and jitter, never retry 4xx, truncate and retry once on context-length errors. Then fall back to a secondary provider, then degrade to a cache or an honest unavailable message.
“How would you halve the cost?” — route by difficulty so a small model handles the easy majority; enable prompt caching by putting stable content first; cache exact repeats; cap max_tokens; and move offline work to a batch API.
“Is semantic caching a good idea?” — with care. Similar queries can have different correct answers, so set the threshold conservatively and exclude precision-sensitive paths. Exact-match caching is the safe win.
“Why not point at the latest model alias?” — the provider can change your behaviour without a deployment, and you have no way to correlate a quality shift with it. Pin the version, log it per response, and re-run evals before upgrading.
“Would you build this or use LiteLLM?” — LiteLLM covers unified interface, routing, fallbacks and budget tracking across many providers, and is the pragmatic default. A thin in-house layer makes sense when you want a domain-specific interface rather than a generic completion API.