One of the most effective hacks in modern machine learning — and the group trick that carried the whole story into the LLM era.
Where we left off: TRPO's trust region is the right idea, but enforcing it needs Fisher-matrix machinery nobody enjoys implementing. Can we get the effect without the math?
PPO (Proximal Policy Optimization, 2017) answers yes. Instead of constraining the update with second-order machinery, it changes the objective so that large policy changes simply stop being rewarded.
Define the probability ratio between the new and old policy for an action:
$$r=\frac{\pi_{\text{new}}(a\mid s)}{\pi_{\text{old}}(a\mid s)}$$
If \(r=1\), the policy hasn't changed for this action. PPO multiplies the advantage by this ratio — but clips the ratio to stay inside \([1-\varepsilon,\;1+\varepsilon]\) (with \(\varepsilon\approx 0.2\)), and takes the more pessimistic of the clipped and unclipped versions:
$$L \;=\; \min\!\big(\,r\,A,\;\; \mathrm{clip}(r,\,1-\varepsilon,\,1+\varepsilon)\,A\,\big)$$
That formula is easier to see than to read:
In words: once the new policy differs from the old one by more than ~20% on an action, making it differ even more earns you nothing. There is no incentive to leave the neighborhood of the old policy.
And because that's just a modified loss function, everything becomes ordinary again: plain first-order gradient descent, Adam, minibatches, several epochs on the same batch, ~30 lines of core logic. PPO trades TRPO's guarantees for TRPO's spirit at a fraction of the complexity — and empirically it matched or beat TRPO almost everywhere. It became the default workhorse of deep RL, and years later it was the algorithm behind RLHF, the technique used to fine-tune chat models from human feedback.
The pattern — find the bottleneck, fix exactly that — kept going right into the LLM era. When you run PPO on a large language model, the bottleneck turns out to be the critic. Remember why the critic exists (Part 3): it's a variance-reduction device — it supplies the baseline. But for an LLM, the critic is itself a billion-parameter model that must be trained, stored, and evaluated alongside the actor. That's an enormous price for a baseline.
GRPO (Group Relative Policy Optimization, used to train DeepSeek's reasoning models) asks: what if we get the baseline a cheaper way? For each prompt, sample a group of responses from the current policy, score them all, and use the group's average score as the baseline:
The critic is deleted entirely — the group plays its role — while PPO's clipping machinery stays. And notice what happened: the field circled all the way back to the Part 2 insight ("subtract how well I usually do"), just with a new, cheaper way to estimate "usually."
None of these algorithms fell from the sky. Each one is the previous one, plus a patch for its most painful failure. If you ever forget an equation, you can re-derive the shape of it just by remembering which problem it was born to solve — and that's a far more durable kind of understanding than a list.
Where to go from here: this series followed the on-policy, policy-gradient lineage because it's the cleanest single storyline. There's a parallel evolutionary tree on the value-based side (Q-learning → DQN → Double DQN → Rainbow) and an off-policy actor-critic branch (DDPG → TD3 → SAC), each with its own chain of problem-and-fix. Same game, different family.