Measure step size where it matters — in behavior, not weights — and never leave the bubble where your estimates can be trusted.
Where we left off: in on-policy learning, one oversized update wrecks the policy — and because the policy generates its own training data, the damage compounds. See it happen:
TRPO's (Trust Region Policy Optimization, 2015) key insight is diagnostic before it is algorithmic: "learning rate" limits the wrong thing. A small step in parameter space can be a huge step in behavior space — nudge a few weights and a softmax can flip which action it prefers. What we should limit is not how much the weights move, but how much the policy's behavior moves.
So TRPO poses each update as a constrained problem:
Maximize expected advantage — subject to: the new policy's action distribution stays within a small KL-divergence of the old one.
KL divergence is just a measure of how different two probability distributions are. The constraint defines a trust region: a bubble of policies that behave similarly to the current one. Inside the bubble, the gradient estimates you computed from the current policy's data are still valid. Outside it, they're fiction — the data came from a policy that no longer resembles the one you're evaluating.
And it works. TRPO delivered near-monotonic improvement on hard control problems and made deep RL dramatically more stable. For a while, it was the state of the art.
The problem: actually solving that constrained problem is a nightmare. To respect the KL constraint properly, TRPO needs second-order information — the curvature of the KL divergence, a Hessian-like object called the Fisher information matrix. For a network with millions of parameters, you can't even store that matrix, let alone invert it. TRPO works around this with conjugate-gradient tricks to approximate the natural gradient, plus a backtracking line search to enforce the constraint.
It's genuinely elegant math. It is also: hard to implement, easy to get subtly wrong, computationally heavy, and awkward to combine with everyday deep-learning machinery — shared actor-critic layers, dropout, minibatch reuse. The community had a stable algorithm that few people could comfortably use.
The question practically asks itself.
The trust region is the right idea, but enforcing it needs Hessian-scale machinery. Can we get the trust region's effect without the trust region's math?