# All-Action Policy Gradient Methods: A Numerical Integration Approach

arXiv: https://arxiv.org/abs/1910.09093 · HTML: https://mlsanity.com/paper/1910.09093/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Oct 2019.

## Built on by (most cited first, 4 of 4)

- [Flexible Option Learning](https://mlsanity.com/paper/2112.03097/) (2021)
- [On Many-Actions Policy Gradient](https://mlsanity.com/paper/2210.13011/) (2022)
- [Taylor TD-learning](https://mlsanity.com/paper/2302.14182/) (2023)
- [Learning in complex action spaces without policy gradients](https://mlsanity.com/paper/2410.06317/) (2024)

## Abstract (arXiv)

While often stated as an instance of the likelihood ratio trick [Rubinstein, 1989], the original policy gradient theorem [Sutton, 1999] involves an integral over the action space. When this integral can be computed, the resulting "all-action" estimator [Sutton, 2001] provides a conditioning effect [Bratley, 1987] reducing the variance significantly compared to the REINFORCE estimator [Williams, 1992]. In this paper, we adopt a numerical integration perspective to broaden the applicability of the all-action estimator to general spaces and to any function class for the policy or critic components, beyond the Gaussian case considered by [Ciosek, 2018]. In addition, we provide a new theoretical result on the effect of using a biased critic which offers more guidance than the previous "compatible features" condition of [Sutton, 1999]. We demonstrate the benefit of our approach in continuous control tasks with nonlinear function approximation. Our results show improved performance and sample efficiency.