A frozen model gets smarter if you let it think longer

I once spent a whole post arguing that a transformer has no reasoning module hiding inside it. Just attention shuffling tokens around, a feed-forward step refining each one, and scale doing the rest. I still think that's right. So here is something that ought to bother you, because it bothered me.

Take one of those models. Freeze every weight so it cannot learn a single new thing. Ask it a hard question and note the answer. Now ask it again, identically, except this time you let it spend ten times as much compute thinking before it commits. The second answer is measurably more likely to be correct. Nothing about what the model knows has changed. We only gave it more time.

So the reasoning we couldn't find inside the weights, it turns out we can bolt onto the outside. Let's build that loop up one piece at a time: writing the working out, sampling many attempts and voting, training a judge to grade them, and finally an RL step that teaches the model to spend its thinking time well. The last piece is the genuinely strange one, so I'll save it.

Chain of thought is a scratchpad

Start with a fact that sounds like a limitation. A transformer does a fixed amount of work per token: one forward pass, the same whether it's predicting the token after "2 + 2 =" or the next line of a hard proof. (We pulled that pass apart in the transformer and KV-cache posts, so I won't re-derive it here.) It is, in a real sense, a fixed-depth machine. How does a fixed-depth machine ever do something that genuinely needs twenty careful steps?

It writes them down. Every intermediate token the model emits becomes part of the context the next step reads, so the half-finished working on the page is extra computation the model gets to lean on. Ask it to "think step by step" and it serialises a problem that wouldn't fit in one forward pass into a chain of small ones that do.

$$p(a \mid q) = \sum_{z} p(a \mid z, q)\, p(z \mid q)$$

Copy LaTeXwith $$ delimitersCopy mathexpression only

Sample many paths, and let them vote

Greedy decoding takes the single most likely next token at every step, which quietly commits you to one line of reasoning long before you know whether it pans out. So don't be greedy. Nudge the temperature up, run the model a handful of times, and you get a handful of different chains, each arriving at its own answer. Then count.

This is self-consistency, and it works for a rather nice reason: a model's wrong reasoning tends to be wrong in many uncorrelated ways, so the mistakes scatter across different answers, while the genuinely correct reasoning keeps converging on the same one. The right answer wins by plurality even when no single chain is reliable on its own.

Drag $N$ up and watch the majority answer firm up as the votes come in. The aggregate is doing something none of the individual chains can promise.

$$\hat a = \arg\max_{a}\; \sum_{i=1}^{N} \mathbf{1}\!\left[ a^{(i)} = a \right], \quad z^{(i)} \sim p(z \mid q)$$

Copy LaTeXwith $$ delimitersCopy mathexpression only

A verifier turns a guesser into a searcher

Majority vote is a little dim, though. It counts answers but cannot tell a careful chain from a lucky one. The next idea is to train a second model, a verifier, to score how likely a chain is to be right, and then keep the best-scoring of your $N$ candidates instead of the most popular. With a good verifier this stops being a vote and becomes a search: a small model that generates many candidates, plus a verifier that picks among them, can beat a much larger model that answers once.

There are two flavours, and the difference matters. An outcome reward model grades only the final answer. A process reward model grades every step, which catches the chain that blunders into the right answer through a wrong move. That's the result behind Lightman's "Let's Verify Step by Step": denser feedback, at the cost of having to label correctness step by step.

It helps to think of the whole thing as navigation, which is exactly what the pathfinding lab does.

$$\hat a = a^{(i^\star)}, \quad i^\star = \arg\max_{i \in \{1,\dots,N\}} V\!\left( q, z^{(i)} \right)$$

Copy LaTeXwith $$ delimitersCopy mathexpression only

The inference-time scaling law

The first scaling law, the one I leaned on in the transformer post, says a bigger model trained on more data is predictably better. There's a second axis, and the surprise is how cleanly it sits at right angles to the first. Hold the model fixed, spend more compute at the moment you ask, and accuracy rises roughly with the logarithm of that compute. Snell and colleagues pushed this far enough that, on some problems, spending test-time compute well matched a model more than ten times larger.

Slide the budget along and watch the small-model-plus-search line climb past the flat baseline of the bigger model answering once. The crossover point is the whole argument in one dot.

$$\text{acc}(C) \approx a + b \, \log C$$

Copy LaTeXwith $$ delimitersCopy mathexpression only

Teaching the model to think

So far the loop is bolted on from outside. We wrote the prompt that asks for steps, we wrote the sampling loop, we trained the verifier. The leap of the last year folds it back into training: reward the model only when its final answer checks out, and let it work out for itself how to use the thinking time.

DeepSeek-R1 did this with GRPO, which is about as plain as reinforcement learning gets. Sample a group of answers to the same question, score each one by whether it's correct, and push up the ones that beat the group's average. No human-written reasoning to imitate, no separate value network to train. And from that thin a signal the model grows long chains of thought on its own, pausing to check its work and backtrack, behaviour nobody ever wrote down for it.

$$\hat A_i = \frac{r_i - \operatorname{mean}(r_1,\dots,r_G)}{\operatorname{std}(r_1,\dots,r_G)}$$

Copy LaTeXwith $$ delimitersCopy mathexpression only

When thinking longer makes it worse

I don't want to oversell the dial. It backfires in at least three ways, and they're worth knowing before you reach for it.

Overthinking. Past a point, more tokens hurt. The model talks itself out of an answer it already had, or pours budget into a question it had nailed on the first pass. Reward hacking. Optimise hard against a verifier and the policy learns to sound correct rather than be correct, manufacturing confident-looking steps a weak grader rewards. Unfaithful reasoning. The written chain can be a story told after the fact rather than the actual cause of the answer.

Here's the honest aside I keep coming back to. None of this is really new, if you've done any operations research. A* with an admissible heuristic, branch-and-bound, Monte Carlo tree search in a game engine: they all let you trade more compute for a better answer and stop when the budget runs out. We've known since the 1960s that search buys quality. The genuinely new part is what's being searched over. The thing the model explores is its own natural language, and you can train it, with nothing but a final-answer reward, to write better search trajectories for itself. The data structure is prose. That's the weird bit, and I don't think we've finished being surprised by it.

Tokens or parameters?

So when do you spend on thinking versus on a bigger model? They're substitutes over a useful range, but not everywhere. Search helps most when the base model is sometimes right, so the verifier has a signal to amplify, and when checking an answer is cheaper than producing one. For a fact the model simply does not hold, no amount of thinking conjures it. Search rearranges what's in the prior; it can't add to it.

Recap

A language model's knowledge is frozen in its weights, but its accuracy at the moment you ask is not. Write the working out and a fixed-depth machine gains serial depth. Sample many chains and vote and the right answer wins by plurality. Score them with a verifier and voting becomes search. Spend more compute and accuracy climbs with the log of the budget. And reward the right final answer, with nothing else, and the model learns to do all of that thinking for itself. The reasoning module was never inside the weights. We built it out of search and verification, in the open, in tokens.

Reading further

• Wei et al. (2022), Chain-of-Thought Prompting Elicits Reasoning in LLMs: the paper that showed prompting for intermediate steps unlocks multi-step reasoning. arXiv:2201.11903
• Wang et al. (2022), Self-Consistency Improves Chain of Thought Reasoning: sample many chains, take the majority, the idea behind equation (2). arXiv:2203.11171
• Lightman et al. (2023), Let's Verify Step by Step: process reward models that grade each step beat outcome-only verifiers. arXiv:2305.20050
• Snell et al. (2024), Scaling LLM Test-Time Compute Optimally: the inference-time scaling law, and small-model-plus-search beating a bigger model. arXiv:2408.03314
• DeepSeek-AI (2025), DeepSeek-R1: RL-only reasoning with GRPO, and the emergent backtracking from a final-answer reward alone. arXiv:2501.12948
• OpenAI (2024), Learning to Reason with LLMs: the o1 report, accuracy scaling smoothly with both train-time RL and test-time thinking. openai.com

The first scaling law told us how to spend a training budget. This second axis is us still learning how to spend an inference budget, and the curve of where the next dollar buys the most accuracy is one we're only starting to draw. Worth watching closely.