mlsanityData to 7 Oct 2026

Model-free Policy Learning with Reward Gradients

arXiv 2103.05147 ↗First posted 9 Mar 2021cs.LG cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Despite the increasing popularity of policy gradient methods, they are yet to be widely utilized in sample-scarce applications, such as robotics. The sample efficiency could be improved by making best usage of available information. As a key component in reinforcement learning, the reward function is usually devised carefully to guide the agent. Hence, the reward function is usually known, allowing access to not only scalar reward signals but also reward gradients. To benefit from reward gradients, previous works require the knowledge of environment dynamics, which are hard to obtain. In this work, we develop the textitReward Policy Gradient estimator, a novel approach that integrates reward gradients without learning a model. Bypassing the model dynamics allows our estimator to achieve a better bias-variance trade-off, which results in a higher sample efficiency, as shown in the empirical analysis. Our method also boosts the performance of Proximal Policy Optimization on different MuJoCo control tasks.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 36 of its 56 references that are in the map. First posted on arXiv 9 Mar 2021.

Builds on

36 of 56 references.

References · 36

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2013Auto-Encoding Variational Bayescited by 3,980
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  8. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  9. 2016OpenAI Gymcited by 1,684
  10. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  11. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  12. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
24 more
  1. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  2. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  3. 2019Challenges of Real-World Reinforcement Learningcited by 282
  4. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 281
  5. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 260
  6. 2021Reinforcement learning based recommender systems: A surveycited by 187
  7. 2016Sample Efficient Actor-Critic with Experience Replaycited by 185
  8. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
  9. 2020Phasic Policy Gradientcited by 121
  10. 2019When to use parametric models in reinforcement learning?cited by 118
  11. 2017Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agentscited by 107
  12. 2017Backpropagation through the Void: Optimizing control variates for black-box gradient estimationcited by 104
  13. 2018Deep Reinforcement Learning for Page-wise Recommendationscited by 99
  14. 2018Understanding and correcting pathologies in the training of learned optimizerscited by 97
  15. 2018Implicit Reparameterization Gradientscited by 84
  16. 2018Benchmarking Reinforcement Learning Algorithms on Real-World Robotscited by 64
  17. 2019PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaoscited by 56
  18. 2018Variance reduction properties of the reparameterization trickcited by 41
  19. 2018An Off-policy Policy Gradient Theorem Using Emphatic Weightingscited by 34
  20. 2020Mitigating Value Hallucination in Dyna Planning via Multistep Predecessor Modelscited by 27
  21. 2019On the Generalization Gap in Reparameterizable Reinforcement Learningcited by 21
  22. 2018Reparameterization Gradient for Non-differentiable Modelscited by 11
  23. 2020Batch Reinforcement Learning with a Nonparametric Off-Policy Policy Gradientcited by 3
  24. 2019Deterministic Value-Policy Gradientscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 1; 2024: 4; 2025: 1; 2026: 1.

9 papers.

  1. 2022Asynchronous Reinforcement Learning for Real-Time Control of Physical Robotscited by 19
  2. 2024Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Bufferscited by 16
  3. 2023Learning to Optimize for Reinforcement Learningcited by 12
  4. 2024Revisiting Sparse Rewards for Goal-Reaching Reinforcement Learningcited by 12
  5. 2024Revisiting Scalable Hessian Diagonal Approximations for Applications in Reinforcement Learningcited by 5
  6. 2022A Temporal-Difference Approach to Policy Gradient Estimationcited by 4
  7. 2025Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularizationcited by 3
  8. 2026Revisiting Mixture Policies in Entropy-Regularized Actor-Criticcited by 1
  9. 2024What to Do When Your Discrete Optimization Is the Size of a Neural Network?cited by 0