The Evolution of RL · Part 2 of 5

Subtract the Average: Baselines and the Advantage

A one-line change that keeps the gradient honest, kills most of the noise — and quietly invents the most important quantity in reinforcement learning.

Series: Nobody Invented PPO From Scratch · Previously: Part 1 · ~5 min read

Where we left off: REINFORCE's gradient is unbiased but drowning in noise, and in a game where every score sits between +90 and +100, even our worst episodes get applauded.

A small miracle of probability

Here's the fix, and it's almost suspiciously simple. You can subtract any fixed reference value \(b\) from the return —

$$\nabla_\theta J\;\approx\;\sum_t \nabla_\theta \log \pi_\theta(a_t\mid s_t)\cdot (R-b)$$

— and the expected gradient doesn't change at all. On average you still point in exactly the right direction. But the variance of the estimate can drop dramatically if you pick \(b\) well.

Intuition: instead of asking "was this episode good?", you're now asking "was this episode better than usual?" Watch what happens to the learning signal in that 90–100 game as you raise the baseline:

baseline b b = 0

The useless "everything is positive" component of the gradient is gone. What remains is pure comparison — and comparisons are exactly what a policy needs in order to choose.

The best baseline has a name

So what's the best value for \(b\)? Not one global constant — the natural choice is "how well I usually do from this state." That quantity has a name: the value function,

$$V(s)\;=\;\text{expected return, starting from } s \text{ and following the current policy.}$$

And the gap between what happened and what usually happens also has a name — the advantage:

$$A(s,a)\;=\;\underbrace{\text{return after taking } a}_{\text{what happened}}\;-\;\underbrace{V(s)}_{\text{what usually happens}}$$

It answers: how much better was this particular action than my typical outcome from here? Positive advantage → do it more. Negative → do it less. Its magnitude even tells you how strongly to feel about it.

This reframing is the pivot point of the entire field:

Stop reinforcing actions for being in good episodes. Reinforce actions for being better than expected.

Every algorithm from here to the end of the series — actor-critic, TRPO, PPO, GRPO — keeps this exact idea and only changes how the advantage is estimated or how big a step to take on it.

One problem, though. \(V(s)\) isn't given to us. To subtract "what usually happens from this state," we'd have to know what usually happens from every state — and no one is going to hand us that function.