Policy Gradients and the Arithmetic of Variance
The policy gradient theorem is about four lines of algebra, and almost none of the work in making it function is in those four lines. The derivation gives you an estimator that is unbiased for any policy, any reward, any dynamics — including dynamics you cannot differentiate and rewards you cannot write down. Then you run it and nothing happens, because the estimator's variance is large enough to bury the signal it is carrying.
This post derives the gradient in Pieter Abbeel's notation, then treats variance as the actual subject. The claim I want to make precise is the one usually waved at: subtracting a baseline reduces variance. That is a theorem with an explicit form, an exactly optimal \(b^\star\), and a characterisation of which baselines help and which hurt — and the standard choice, the mean return, is not the optimal one.
Every number below comes from a simulation or an exact enumeration, and the closed forms are checked against Monte Carlo before they are stated.