Reasoning models and test-time compute

The biggest capability shift since instruction tuning, and completely absent from pre-2025 notes. The core idea: spend more compute at inference, get better answers — a scaling axis independent of model size.

The shift

Classical scaling: bigger model, more pretraining data, better results. That’s expensive and has been showing diminishing returns.

Test-time compute scaling: keep the model, let it think longer before answering. A smaller model given an adequate thinking budget can outperform a substantially larger one at matched total compute — reported at up to roughly 14x the size difference. That changes the economics of model selection.

How reasoning models work

They generate a long internal chain of thought before the visible answer:

<thinking>
Let me work through this. The constraint is X, so...
Actually that's wrong, because Y. Let me reconsider...
Checking: if I substitute back, I get Z. That holds.
</thinking>

The answer is Z because...

Mechanically that’s still next-token prediction. What’s different is training: reinforcement learning against verifiable outcomes teaches the model to produce reasoning traces that actually lead to correct answers, rather than traces that merely look plausible.

Prompted chain-of-thought (“think step by step”) on a non-reasoning model gets a fraction of the benefit. The trained behaviour includes backtracking, self-checking, and trying alternative approaches — behaviours a prompt can request but not reliably induce.

RLVR

Reinforcement Learning from Verifiable Rewards is what makes this trainable at scale.

For domains where correctness is machine-checkable — maths with a known answer, code that passes tests, formally verifiable logic — you don’t need human preference labels or a learned reward model. The verifier is the reward:

reward = 1 if tests_pass(model_output) else 0

No reward-model drift, no human labelling bottleneck, no reward hacking against a learned proxy. Generate many attempts, reward the ones that verify, reinforce the reasoning patterns that produced them.

The limitation follows directly: it works where verification is cheap and reliable. Extending it to open-ended domains — writing quality, strategic judgement — is an active problem, because there’s no verifier.

The RL methods

GRPO is the one to be able to explain. Instead of training a value network to estimate expected reward, sample a group of responses to the same prompt, and use the group’s mean reward as the baseline. The advantage of each response is its reward relative to its siblings.

That removes the critic model entirely — roughly halving memory and compute for the RL stage — which is why the field moved toward it.

DAPO addresses what breaks when chains get very long: it adds asymmetric clipping, dynamic sampling to avoid wasting compute on groups where every response is right or every one is wrong, token-level loss rather than sequence-level, and shaped rewards for overlong outputs.

The overall trajectory: from expensive critic-based PPO toward cheaper, critic-free, group-relative methods. Saying that shows you’re tracking the field rather than reciting the 2022 RLHF diagram.

Thinking budgets

Reasoning models expose the compute/quality trade-off as a parameter:

response = client.messages.create(
    model=MODEL,
    thinking={"type": "enabled", "budget_tokens": 8000},
    messages=[...],
)

Practical consequences for anyone building on them:

• Latency rises substantially. Thinking tokens are generated sequentially like any others. A response might take tens of seconds.
• Cost rises. You pay for reasoning tokens, and they often outnumber the visible answer.
• Streaming UX changes. Time-to-first-visible-token is long. Show a thinking indicator or stream the reasoning if the provider allows.
• More thinking isn’t monotonically better. Past a point, extra budget adds cost without accuracy, and can introduce overthinking on simple questions.

When to use one

Routing is the production pattern. Classify the request, send routine work to a fast cheap model and hard work to a reasoning model with a budget. Sending everything to a reasoning model is the expensive mistake; sending nothing is the quality mistake.

Adjacent techniques

• Self-consistency — sample n answers at non-zero temperature, take the majority. Simple, effective, n times the cost, and it works without a reasoning model.
• Best-of-n with a verifier — generate several, pick using a checker or reward model. Directly applicable when you have tests.
• Process supervision — reward correct intermediate steps rather than only the final answer, which gives denser signal and catches right-answer-wrong-reasoning.

Interview angle

• “What’s a reasoning model?” — one trained, usually with RL against verifiable outcomes, to generate an extended chain of thought before answering. The distinguishing feature is trained behaviour like backtracking and self-checking, not just a prompt asking for steps.
• “What is test-time compute scaling?” — trading inference compute for accuracy, as a scaling axis separate from parameters and pretraining data. A smaller model with an adequate thinking budget can beat a much larger one at matched total compute.
• “What is RLVR and why does it matter?” — reinforcement learning where the reward comes from an automatic verifier (unit tests, a known answer) instead of a learned reward model. It removes the human labelling bottleneck and the reward-hacking failure mode, but only applies where correctness is cheaply checkable.
• “Explain GRPO versus PPO.” — PPO trains a separate value critic to compute advantages. GRPO samples a group of responses to the same prompt and uses the group mean as the baseline, eliminating the critic and roughly halving RL-stage memory and compute.
• “How do you use reasoning models in production?” — route by difficulty. Cheap fast model for routine requests, reasoning model with a bounded thinking budget for hard ones. Account for the latency in the UX and cap the budget, since accuracy plateaus while cost doesn’t.
• “Cheaper way to get some of the benefit without a reasoning model?” — self-consistency: sample several answers and take the majority, or best-of-n with a verifier where one exists. Linear cost increase, no special model required.