On Many-Actions Policy Gradient
Not on the map. Why?
Abstract (arXiv)
We study the variance of stochastic policy gradients (SPGs) with many action samples per state. We derive a many-actions optimality condition, which determines when many-actions SPG yields lower variance as compared to a single-action agent with proportionally extended trajectory. We propose Model-Based Many-Actions (MBMA), an approach leveraging dynamics models for many-actions sampling in the context of SPG. MBMA addresses issues associated with existing implementations of many-actions SPG and yields lower bias and comparable variance to SPG estimated from states in model-simulated rollouts. We find that MBMA bias and variance structure matches that predicted by theory. As a result, MBMA achieves improved sample efficiency and higher returns on a range of continuous action environments as compared to model-free, many-actions, and model-based on-policy SPG baselines.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 32 of its 53 references that are in the map. First posted on arXiv 24 Oct 2022.
Builds on
32 of 53 references.
References · 32
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2015Trust Region Policy Optimizationcited by 2,580
- 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
- 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
- 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
- 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
- 2020Mastering Atari with Discrete World Modelscited by 1,007
- 2018DeepMind Control Suitecited by 956
- 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
- 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
20 more
- 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
- 2019Model-Based Reinforcement Learning for Ataricited by 502
- 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
- 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
- 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
- 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 274
- 2018Model-Ensemble Trust-Region Policy Optimizationcited by 219
- 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 207
- 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 168
- 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 162
- 2018Model-Based Value Estimation for Efficient Model-Free Reinforcement Learningcited by 156
- 2018Lipschitz Continuity in Model-based Reinforcement Learningcited by 91
- 2016Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Criticcited by 81
- 2020Model-Augmented Actor-Critic: Backpropagating through Pathscited by 63
- 2018The Mirage of Action-Dependent Baselines in Reinforcement Learningcited by 48
- 2019Is the Policy Gradient a Gradient?cited by 40
- 2020Estimating Gradients for Discrete Random Variables by Sampling without Replacementcited by 31
- 2018Expected Policy Gradients for Reinforcement Learningcited by 28
- 2017Mean Actor Criticcited by 8
- 2019All-Action Policy Gradient Methods: A Numerical Integration Approachcited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.