The algorithm that trains reasoning LLMs can't teach a robot to walk
Not as well as PPO, anyway. There are two reasons. The first is a bias, and it has been fixed. The second is still open. Both come straight out of the one idea the last story turned on: subtract a baseline.
1A baseline has one rule
Last time: subtract any baseline, the gradient stays honest, the noise drops. There was fine print on the word any.
The pivot of the whole last series was a one-line change. Subtract a baseline from the return before you learn from it, because a return means nothing until you know what to expect. Score 80 on a test. Good or bad? That depends on the class average.
The promise was that subtracting \(\bs{b}\) is free: the expected gradient does not change, only the noise around it. Here is the fine print. Split the update into the part we want and the part we hope is zero:
The first piece is the true gradient. The second is the bias, and "free" means it vanishes. When does it?
It hinges on one fact about the score function: its average over all trajectories is zero.
Now the two cases.
The expectation is over trajectories, and the trajectories are produced by the agent's actions. So any \(\bs{b}\) that does not depend on those actions can be taken outside the expectation: \(\E[\bs{b}\,g(\tau)]=\bs{b}\cdot\E[g(\tau)]=\bs{b}\cdot 0=0\). Free.
If \(\bs{b}\) changes from trajectory to trajectory, it cannot come outside. \(\E[\bs{b}(\tau)\,g(\tau)]\) is a weighted sum of the \(g(\tau)\), and a weighted sum of terms that only cancel when weighted equally need not be zero. Whatever is left over is bias.
Let's make this concrete. Consider an environment with one state and two actions: Left with probability 0.3, Right with 0.7. For a two-way softmax, \(\nabla\log\pi\) comes out to \(+0.7\) for Left and \(-0.3\) for Right, and you can check they already average to zero: \(0.3(0.7)+0.7(-0.3)=0\). Now attach a baseline.
A baseline that depends on which action was taken leaves \(+0.63\) behind. That number is pure bias: it will push Left up forever, regardless of reward.
A baseline must not know what the agent just did.
The critic \(\bs{V(s_t)}\) of actor-critic and PPO passes. It depends on the state, not on the action chosen there, so the same cancellation works one step at a time. (Why it beats a constant, by centring the weights around zero and cutting variance, is a separate story, not covered here.)
The problem we leave with
Any baseline that doesn't peek at the action is free. Now meet an algorithm whose baseline peeks. Twice.
2GRPO breaks it
The group mean is a perfectly good baseline. It's what GRPO multiplies it by that breaks the rule.
Recall the epilogue of the last story. GRPO deletes the critic. From one start state \(s_0\) it samples a group of \(G\) trajectories and uses the group's mean return as the baseline. In LLM terms: the start state is the prompt, a trajectory is a response, an action is a token, the return is the grader's score, and the length \(T_i\) is the token count.
What GRPO is after is the gradient at that one start state. That is an expectation over everything the policy might write, and the group of \(G\) samples is its Monte Carlo estimate:
So far, textbook. GRPO fills in \(\bs{b}\) with the group mean \(\bs{\bar R}\), then makes two extra divisions. In words, the weight on every action of trajectory \(i\) is:
(return − group mean) ÷ spread of the group ÷ length of this trajectory
(GRPO also wraps this in PPO's clip and a KL penalty. Neither matters here.) Each of the two divisions has a reason to exist. Each breaks the rule in its own way.
Dividing by length: a baseline that knows how long you talked
The \(\gp{T_i}\) division averages the loss over tokens, a habit carried over from supervised LLM training. Whatever its motive, look at what it does to the baseline. Everything, \(\bs{\bar R}\) included, is divided by the length of the very trajectory being scored. That is exactly what the rule forbids: the baseline now knows something about what the agent did, namely when it decided to stop.
Does the bias survive the averaging? Be generous: treat \(\bs{\bar R}\) and \(\gp{\sigma}\) as fixed numbers. The baseline term still does not cancel:
In words: the leftover is the gradient of "how short the policy tends to be". It is zero only when every episode has the same length. A response that can stop when it likes does not qualify. Neither does a robot that can fall over.
This is not a hypothetical. Liu et al. (2025) found that under GRPO, responses grew steadily longer over training, and the wrong ones grew most of all. A length-dependent baseline, left running, learned to be long.
Dividing by spread: the loudest voices are the easy prompts
The \(\gp{\sigma}\) division is quieter. Inside one group every trajectory is divided by the same \(\gp{\sigma}\), so within a group it mostly rescales the update rather than tilting it. (Mostly: \(\gp{\sigma}\) is computed from the group, so it contains \(\rw{R_i}\), the return of the trajectory it is dividing. The same thing will turn out to be true of \(\bs{\bar R}\), and we deal with both in the next section.)
The damage is across start states. The real objective averages over them, so every start state's gradient should count equally. GRPO divides each one by its own group's spread:
Where is \(\gp{\sigma_{s_0}}\) smallest? Where the \(G\) trajectories scored about the same: prompts the policy nearly always solves, or nearly always fails. Those get the loudest updates. Liu et al. call this the question-level difficulty bias. With pass/fail rewards and \(G=8\), a prompt solved once in eight tries is pushed 1.5 times harder than one solved four times in eight. The learner leans towards what it has nearly mastered or nearly given up on, and away from the middle, where the most is left to learn.
The problem we leave with
Two divisions, two ways the baseline peeks at the trajectory it is judging. The question practically asks itself: what if you just don't divide?
3Dr. GRPO: just delete them
Keep \(R_i-\bar R\). Drop \(\sigma\). Drop \(T_i\). That is the entire fix.
That is what Liu et al. did, and they called it Dr. GRPO: GRPO Done Right.
weight on every action = return − group mean. Nothing else.
Does it pass the rule? Almost, and the "almost" is harmless. The group mean \(\bs{\bar R}\) contains \(\rw{R_i}\) itself, one of its \(G\) ingredients. So a trajectory sits inside its own baseline, and exactly \(1/G\) of the gradient leaks through. But that leak is a constant: the expected update is \(\frac{G-1}{G}\nabla J\), the right direction with a slightly shorter step, and the learning rate absorbs it. GRPO's leftover depended on the trajectory and bent the direction. Dr. GRPO's only shortens the arrow.
Where the 1/G comes from, and how to get rid of it · optional, with a figure to click
Split the baseline into its \(G\) ingredients. The other \(G-1\) returns are independent of \(\tau_i\) and contribute nothing. One term survives:
To pass exactly, use \(\bs{\bar R_{-i}}\), the mean of the other \(G-1\) returns. It knows nothing about \(\tau_i\), so the bias is zero. And Dr. GRPO's weight is that leave-one-out weight, rescaled:
The paper notes the same equivalence to leave-one-out REINFORCE. Side by side:
| Weight on \(g(\tau_i)\) | Baseline term \(\E[b\,g]\) | Expected update | |
|---|---|---|---|
| GRPO | \(\dfrac{R_i-\bar R}{\sigma\,T_i}\) | Not zero \(\;\tfrac{b}{\sigma}\,\nabla\E[1/T]\) | Not a multiple of \(\nabla J\) |
| Dr. GRPO | \(R_i-\bar R\) | A constant share \(\;\tfrac{1}{G}\nabla J\) | \(\dfrac{G-1}{G}\,\nabla J\) |
| Leave-one-out | \(R_i-\bar R_{-i}\) | Zero | \(\nabla J\) |
Reason one, solved
GRPO divided by the length of the trajectory it was scoring, and the bias did not cancel. Dr. GRPO deletes the divisions and points the right way.
The problem we leave with
"Points the right way" is a statement about the average update. It says nothing about what each individual action is told. Look at that next.
4Not yet prime time for control
Put Dr. GRPO on a walking robot and the fixed algorithm has a problem the fix never touched.
Reset the simulator to \(s_0\), roll out \(G\) trajectories, and update with Dr. GRPO. Read the formula once more, but this time look at the inner sum:
One number, \(\rw{R_i}-\bs{\bar R}\), is attached to every action in trajectory \(i\). Step 3 and step 300 get exactly the same feedback. This is Part 1 of the last story all over again: REINFORCE's first problem was that credit is assigned collectively, every action in a good episode gets praised. The critic was invented to fix that. Dr. GRPO deleted the critic, and the problem came back.
Take a robot learning to walk. Most of its actions are small balance corrections while it is upright. Those are good, and each deserves a small positive advantage. Then one bad step trips it. That action deserves a large negative advantage, and the few before it some of the blame. Dr. GRPO cannot say any of this. The episode ended in a fall, its return came in below the group mean, and every action gets the same negative number. The good corrections are punished along with the step that tripped it.
Over many episodes this still averages out, which is what section 3 proved. But every action hears about the whole episode and almost nothing about itself, so it takes far more rollouts to learn the same thing.
The problem we leave with
Every action learns from the whole episode and nothing about itself. A critic exists to fix exactly this. So how did GRPO ever work?
5Why LLMs get away with it
A language model hides the problem. A walking robot does not. Count how often the environment speaks.
Dr. GRPO leans on four things. An LLM supplies all four. Most classical RL environments supply none.
| It takes for granted | In LLM training | In classical RL |
|---|---|---|
| Reward arrives once, at the end | A grader scores the finished response. | A walking robot is paid at every step. One summed return throws the timing away. |
| Every step deserves the same feedback | Tolerable: a pretrained model starts with most of its choices already sensible. | The step that trips the robot gets the same weight as the 150 good ones before it. |
| The environment stays quiet mid-episode | The next state is the text so far plus one token. | Physics, noise and opponents answer back at every step. |
| The same start state can be replayed \(G\) times | Send the prompt again. | Needs a simulator that resets to an exact state. A robot or a live game cannot. |
The problem we leave with
Four assumptions, all true for a prompt and a grader, none true for a robot. If that story is right, the data should show it.
6The evidence
Someone has already run the experiment. The scoreboard says what section 4 said.
De Oliveira et al. (2025) did the first systematic study of GRPO on classical control: Learning Without Critics? Revisiting GRPO in Classical Reinforcement Learning Environments. Five environments, critic-free methods against PPO.
The critic wins on long tasks. PPO beat every critic-free method on Acrobot, MountainCarContinuous, HalfCheetah and Humanoid. In the authors' words, "learned critics remain essential for long-horizon tasks". That is reason two, showing up in the data.
The exception is the short one. The paper counts CartPole as short-horizon. With few decisions per episode, a single return says a lot about each action, which is exactly where reason two bites least. (The authors also found PPO over-training on it.)
Fixing the weights bought little. GRPO's group-relative weight did no better than simpler critic-free baselines, down to a plain batch mean. On these tasks, how each episode is scaled matters far less than the fact that every action shares one number.
So the bias fix is necessary, not sufficient. Dr. GRPO points the right way. On long control tasks it hears too little about each action to get there quickly.
What the paper skipped
Two things. It tested GRPO with the \(\gp{\sigma}\) division, not Dr. GRPO, though its batch-mean ablation comes close. And it never formed a real group. Its groups are whichever episodes happened to run in parallel, each from its own random start, so \(\rw{R_i}-\bs{\bar R}\) partly measures which start was lucky rather than which actions were good.
7The whole story in one breath
- The ruleSubtracting a baseline is free only if the baseline doesn't know what the agent just did. The critic \(V(s)\) passes.
- GRPOGroup mean as baseline: fine. Then it divides by the trajectory's own length and its group's spread. The length division breaks the rule, and responses drift longer, wrong ones most.
- Dr. GRPODelete both divisions. The remaining 1/G leak is a constant and only shortens the step. Reason one, solved.
- Reason twoEvery action in an episode gets the same number. The balance corrections are punished with the stumble. REINFORCE's oldest problem, back because the critic is gone.
- Why LLMs copeOne reward at the end, a quiet environment, a replayable start state, and a pretrained policy that is mostly right already. A robot has none of that.
- The evidenceOn five classical control tasks, PPO's critic beat every critic-free method on all but the shortest one.
The pattern from the last story holds. GRPO is PPO minus the critic, plus a cheap stand-in. The stand-in had a bug, and the bug was fixed. But the critic was doing a second job, per-step credit, that no group average can do. On a prompt you can get away without it. On a robot you cannot, yet.
Next
The benchmark has a loose end. Its "groups" were a batch of episodes from different random starts, so part of PPO's lead might be down to that, not to the missing critic. There is one clean way to find out: run Dr. GRPO with proper groups, \(G\) rollouts from the same start state, on the same five tasks. If PPO's lead shrinks, the sloppy groups were costing something. If it doesn't, the critic is the whole story.
Stay tuned.