A series · The Evolution of RL

Nobody Invented PPO From Scratch

Reinforcement learning algorithms are usually taught as a list. But each one exists because the previous one had a specific, painful problem — and someone found a specific fix. This series follows that chain.

THE OBJECTIVEmaximize expected discounted reward

how do we follow its gradient?

REINFORCEreinforce actions in good episodes

crushing variance — everything in a good episode gets credit

+ BASELINE → ADVANTAGEreward being better than expected

nobody hands you the value function

ACTOR-CRITIClearn the baseline with a second network

one oversized update poisons your own data

TRPOonly step inside a trust region

needs Hessian-scale math nobody wants to implement

PPOfake the trust region with a clip

for LLMs, the critic is a giant model of its own

GRPOlet a group of samples be the baseline

PART 1

One Goal, and the Most Direct Way to Chase It

The RL objective, REINFORCE, and why the honest gradient is almost unusably noisy. With an interactive discount dial and a variance demo.

PART 2

Subtract the Average: Baselines and the Advantage

A one-line change that keeps the gradient honest but kills the noise — and quietly invents the most important quantity in RL.

PART 3

Actor-Critic: Learn the Baseline

A second network, bootstrapping, and the GAE bias–variance dial. Ends with the trap that motivates everything after it.

PART 4

TRPO: Don't Trust Big Steps

Why on-policy learning collapses, what a trust region is, and the Hessian that made TRPO famous and unloved. With a collapse simulator.

PART 5

PPO and the Road to GRPO

The clip that faked the trust region, took over deep RL, trained chatbots — and the group trick that deleted the critic.

Best read in order — each part opens with the problem the previous part left behind. Math is rendered with KaTeX and kept to a handful of expressions, each explained in words.