# Variance Reduction in Actor Critic Methods (ACM)

arXiv: https://arxiv.org/abs/1907.09765 · HTML: https://mlsanity.com/paper/1907.09765/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jul 2019.

## Built on by (most cited first, 1 of 1)

- [Decision S4: Efficient Sequence-Based RL via State Spaces Layers](https://mlsanity.com/paper/2306.05167/) (2023)

## Abstract (arXiv)

After presenting Actor Critic Methods (ACM), we show ACM are control variate estimators. Using the projection theorem, we prove that the Q and Advantage Actor Critic (A2C) methods are optimal in the sense of the L2 norm for the control variate estimators spanned by functions conditioned by the current state and action. This straightforward application of Pythagoras theorem provides a theoretical justification of the strong performance of QAC and AAC most often referred to as A2C methods in deep policy gradient methods. This enables us to derive a new formulation for Advantage Actor Critic methods that has lower variance and improves the traditional A2C method.