ai_ml / README.md

Inference and serving

1 min read index source

Inference and serving

The operational layer — what determines latency, throughput and cost.

# File Covers
01 01_inference_basics.md prefill vs decode, metrics, sampling, streaming, cost model, reliability
02 02_batching_and_serving_engines.md continuous batching, PagedAttention, prefix caching, speculative decoding, vLLM
10 10_bedrock_and_agentcore.md AWS managed inference

The cost levers, in order of return

  1. Route by difficulty — cheap model for easy requests, frontier only when needed.
  2. Prefix caching — stable content first in the prompt; ~30% throughput gain when requests share a system prompt.
  3. FP8 KV cache — halves cache memory, doubles concurrency, negligible quality cost.
  4. Cap max_tokens — output tokens are the expensive half.
  5. Batch offline work — asynchronous batch APIs are heavily discounted.

Latency and throughput trade off directly. There is no setting that maximises both; you optimise one under a constraint on the other.