If nobody will hand you the value function, train a second network to predict it. Then discover the trap that motivates everything after.
Where we left off: the perfect baseline is the value function \(V(s)\) — "how well I usually do from this state" — and nobody's going to give it to us.
If we need \(V(s)\) and can't look it up… train a second network to predict it. Now there are two networks in the loop:
The actor is the policy \(\pi_\theta(a\mid s)\). It chooses actions. It's trained with the policy gradient from Part 2, weighted by the advantage.
The critic is the value network \(V_\phi(s)\). It predicts expected return, and it's trained by plain regression: predict the returns you actually observe. The critic never picks an action. Its entire job is to make the actor's learning signal cleaner — it supplies the baseline that turns raw returns into advantages.
The critic buys you a second superpower. With REINFORCE you had to wait for an episode to finish before you could compute \(R\). With a critic, you can estimate the future without living it: after a single step,
$$r \;+\; \gamma\, V(s')$$
— "the reward I just got, plus the critic's estimate of everything after" — is already an estimate of the full return. This is called bootstrapping, and it means you can update mid-episode, from short snippets of experience. This family is the A2C / A3C style of algorithm.
Bootstrapping introduces a choice. How many real rewards do you collect before handing off to the critic's estimate? Use many, and your advantage estimates are accurate but noisy (real life is noisy). Hand off after one step, and they're smooth but only as good as the critic — which, early in training, is wrong.
GAE (Generalized Advantage Estimation) refuses to choose: it blends all the hand-off lengths together, weighted by one knob, \(\lambda\). You don't need its formula — you need its feel:
So: variance tamed, updates possible mid-episode, one tidy knob. Are we done?
One bad update can destroy everything. Here's why. Policy gradient methods are on-policy: the data you learn from is generated by the current policy itself. Now imagine the learning rate is a touch too high, or one batch is unlucky, and an update makes the policy noticeably worse.
A worse policy now collects worse data. Worse data produces worse gradient estimates and a worse critic. Which produce a worse policy. Unlike supervised learning — where a bad step just means the next step starts from a slightly worse spot on a fixed dataset — here a bad step poisons your future data. Training doesn't dip. It collapses, and often never recovers.
The obvious fix — "use a tiny learning rate" — doesn't really work, because the right step size varies wildly across states and stages of training. What we actually want is a principled answer to a sharper question.
On-policy learning eats its own cooking: one oversized update ruins the policy, which ruins the data, which ruins everything after. How big a step is safe?