How to reduce variance in a policy gradient?
A common way to reduce variance is subtract a baseline b (s) from the returns in the policy gradient. The baseline is essentially a proxy for the expected actual return, and it mustn’t introduce any bias to the policy gradient. In fact, the value function itself is a good candidate for baseline.
Are there any caveats to a policy gradient method?
The biggest caveat of policy gradient methods is high variance, and this is usually addressed by utilizing the temporal structure (REINFORCE), introducing a baseline, or adding bias (actor-critic). It’s also worthwhile to notice that policy gradient method only guarantee convergence to local maxima.
How are policy gradient algorithms used in economics?
The policy gradient methods target at modeling and optimizing the policy directly. The policy is usually modeled with a parameterized function respect to θ, πθ(a | s). The value of the reward (objective) function depends on this policy and then various algorithms can be applied to optimize θ for the best reward.
Which is a good candidate for a policy gradient?
The baseline is essentially a proxy for the expected actual return, and it mustn’t introduce any bias to the policy gradient. In fact, the value function itself is a good candidate for baseline. The new term we get after subtracting the baseline is defined advantage A_t.
Which is the best baseline for a policy gradient?
Using a baseline, in both theory and practice reduces the variance while keeping the gradient still unbiased. A good baseline would be to use the state-value current state. State Value: State Value is defined as the expected returns given a state following the policy π_θ .
How to keep policy gradients unbiased in practice?
To keep the gradient estimate unbiased, the baseline independent of the policy parameters. To see why, we must show that the gradient remains unchanged with the additional term (with slight abuse of notation). Using a baseline, in both theory and practice reduces the variance while keeping the gradient still unbiased.
How are policy gradients used in reinforcement learning?
Reinforcement learning of motor skills with policy gradients: very accessible overview of optimal baselines and natural gradient •Deep reinforcement learning policy gradient papers •Levine & Koltun (2013).