# Mean Actor Critic

arXiv: https://arxiv.org/abs/1709.00503 · HTML: https://mlsanity.com/paper/1709.00503/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Sep 2017.

## Built on by (most cited first, 8 of 8)

- [Hindsight Learning for MDPs with Exogenous Inputs](https://mlsanity.com/paper/2207.06272/) (2022)
- [A unified view of likelihood ratio and reparameterization gradients](https://mlsanity.com/paper/2105.14900/) (2021)
- [S-REINFORCE: A Neuro-Symbolic Policy Gradient Approach for Interpretable Reinforcement Learning](https://mlsanity.com/paper/2305.07367/) (2023)
- [On Many-Actions Policy Gradient](https://mlsanity.com/paper/2210.13011/) (2022)
- [Taylor TD-learning](https://mlsanity.com/paper/2302.14182/) (2023)
- [Learning in complex action spaces without policy gradients](https://mlsanity.com/paper/2410.06317/) (2024)
- [Distributions as Actions: A Unified Framework for Diverse Action Spaces](https://mlsanity.com/paper/2506.16608/) (2025)
- [Why Sample What You Can Enumerate? Exact Policy Optimization for Genomic Tool Selection](https://mlsanity.com/paper/2609.10221/) (2026)

## Abstract (arXiv)

We propose a new algorithm, Mean Actor-Critic (MAC), for discrete-action continuous-state reinforcement learning. MAC is a policy gradient algorithm that uses the agent's explicit representation of all action values to estimate the gradient of the policy, rather than using only the actions that were actually executed. We prove that this approach reduces variance in the policy gradient estimate relative to traditional actor-critic methods. We show empirical results on two control domains and on six Atari games, where MAC is competitive with state-of-the-art policy search algorithms.