RL Is Not Enough When There Are Many Right Answers
RL fries the models' brains on complex domains
Reward optimization chooses among the answers pretraining already taught, and the more ways there are to be right, the more that choosing costs.
Colab Jasper Gilley · Independent researcher
RL from a blank slate conquers games. On language we pretrain first and use reward only to fine-tune at the end. Here is what reward optimization actually does to a model whose generating process we know exactly — three mechanisms, five levels of synonymy, two verifiers, and a boundary that never moves. Scroll
SECTION 01
The question
Games yield to RL from scratch. Language does not.
RL from a blank slate conquers games. On language we pretrain first and use reward only to fine-tune at the end. One major difference between the two: how many right answers each state admits.
The sharper question
Call the set of rules pretraining actually taught the model its basis: everything it can produce at all. Does reward optimization _add_ to the basis, or only rearrange and degrade what is already there?
A dial for “how many ways to say the same thing”
The Random Hierarchy Model: a grammar over 8 symbols. Each symbol rewrites into 2 children by one of rules, and every rule is a different way of saying the same thing. Six levels down, a sequence of 64 tokens.
: one way to say everything, so the whole language is **8 sequences**. A game. : every pairing of children is some rule, so every sequence is grammatical. Noise.
Sweep with tree shape, sequence length, model and rollout budget all fixed.
Cagnetta et al., _Phys. Rev. X_, 2024.
SECTION 02
A test bench with exact ground truth
The task: finish the sequence — and two ways to grade it
exact — matches the suffix of the original sequence the prefix was cut from, token by token. A **canonical-answer** verifier: a synonym counts as _wrong_.
parse — any grammatical continuation. Score: the fraction of tree levels at which prefix
suffix parses. A **validity** verifier: a synonym counts as _right_. “Root-valid” means grammatical all the way up.
Start every mechanism from the same place
A separate model for each
, pretrained until its loss plateaus. Every mechanism below starts from that checkpoint, so the question is always what reward does to a _given_ basis.
Three mechanisms, one budget, ending at the cleanest one
Two kinds of result
Bounds: the best any optimizer could do, computed exactly on the known tree. Attainment: what each mechanism actually captures, and at what cost.
Why end at expert iteration
Sample, keep what the verifier passes, fine-tune on it. Every policy-gradient method is a noisy version of that loop. If the clean version cannot teach the model a new rule, no noisier variant will.
SECTION 03
What is achievable at all
Demand one canonical answer, and there is nothing to optimize toward
Even at the ceiling is 0.842, not 1: two root rules can share a left half, so the prefix does not always pin the root.
Computed exactly on the known tree, so _no optimizer can beat it_. Past the prefix barely helps: 32 tokens in, the suffix is nearly independent of it, and the canonical-answer reward has nothing left to teach.
What remains is sharpening: always emitting the most likely answer instead of sampling one. That lifts accuracy without any new knowledge ( at ).
The validity verifier runs the other way
More synonyms means more grammatical strings, so its reward gets _denser_ on the same knob. How hard a domain is for RL depends on the verifier you pair it with.
Pretraining builds the basis, and its boundary sinks as grows
Same 2.7M-parameter model at every , trained to plateau. Colour how much of what is learnable at that level the model learned, read off how far its next-token loss sits above the best achievable (the Bayes floor). That gap, in nats, is the “excess” quoted from here on.
SECTION 04
What each mechanism attains
Vanilla REINFORCE destroys the basis at every
At every the model ends 10–14 nats above the floor, _worse than uniform guessing_, and its policy has collapsed to one answer per prompt (greedy sampled).
- was already solved (0.849 vs. ceiling 0.842); REINFORCE dragged it to 0.388.
- Its layer features stop lining up with the hierarchy levels (, how cleanly they separate by level: ).
Diagnosis
The reward was climbing when it crashed. The signal was there; the optimizer destroyed the model that was reading it.
Anchor it to the pretrained policy, and there is almost nothing to gain
Add a KL penalty toward the frozen pretrained policy and the destruction is cured: – nats above floor, no collapse. What is left, as sampled-policy accuracy:
| $m$ | exact: pretrained $\rightarrow$ RL | calibrated sampler* | parse: pretrained $\rightarrow$ RL | nats above floor | | --- | --- | --- | --- | --- | | 1 | 0.836 → 0.841 | 0.842 (ceiling) | 0.977 → 0.758 | 2.93 | | 2 | 0.191 → 0.190 | 0.191 | 0.980 → 0.993 | 0.29 | | 3 | 0.157 → 0.157 | 0.160 | 0.919 → 0.925 | 0.07 | | 4 | 0.137 → 0.136 | 0.136 | 0.817 → 0.834 | 0.42 | | 6 | 0.130 → 0.133 | 0.130 | 0.731 → 0.743 | 0.79 |
*the accuracy a model gets by sampling from the true answer distribution rather than picking its single best guess.
Canonical: zero gain at every . The policy sits exactly where a calibrated sampler would, as the bound says it must. At it now _holds_ the ceiling.
Validity: a to polish at .
What the anchor buys
The anchor stops the damage. It does not start any learning.
Expert iteration is far stronger — the flywheel really spins
Root-level validity: share of sampled generations that parse all the way to the root. Validity verifier, final round.
- Recovers fully (0.999); the anchored run’s damage there was a quirk of that variant.
- Nearly doubles root validity at () and ().
- No collapse; nats above floor.
- The flywheel spins: winners fed back at go root-valid.
So
Whatever selecting and fine-tuning on winners can do, this is the arm that does the most of it.
Pass rate doubles, but the model learns no new rule.
The dissociation
Root validity doubled, and root validity _requires_ the deep rules. Yet no deep rule was learned: probability mass moved between sequences the basis could already produce (
unchanged;
shows the same).
The more right answers, the steeper the price
Canonical verifier: exact-match gain from six rounds of EI
Validity verifier: share of pretrained knowledge erased by six rounds of EI
erased 1 (learned after EI) (learned before), with “learned” the pyramid’s colour scale read over the whole sequence.
Same mechanism, same budget
As synonymy rises, reward buys _less_ on the one-answer verifier and erases _more_ of the pretrained distribution on the many-answer one. The model gets more fried as the task gets more sophisticated, even under the best-behaved mechanism in the class.
SECTION 05
What it means
Reward optimization is selection within the basis
Across 3 mechanisms 5 2 verifiers, deep-rule knowledge stayed where pretraining left it. Everything else varied:
- REINFORCE burned it (9–14 nats);
- the anchor froze it (0 gain);
- EI gained by selection and sharpening within it.
In language terms
Pass rates go up, the model still does not know the rules, and the distribution pays for it. Only a known generating process lets you tell those three apart.
If the boundary is to move, something other than reward has to move it
**Not all language tasks are created equal.** Code and math sit at the left of this dial, creative writing and original thinking at the right, and language-model RL works on the former and not on the latter.
The one channel that _could_ teach a missing rule is the one that looks like pretraining: fine-tuning on selected samples is more data. At this budget it taught none, while the reward doubled.
Takeaway
Continual learning that _expands_ what a model can do needs a mechanism that moves the basis boundary. Vanilla policy reward is, structurally, not it. The natural objection is that RL on real language behaves differently. Said explicitly, that objection is a bet: for RL to carry continual learning, it would have to gain capabilities on language that it does not gain on this compositional grammar.