mlsanityData to 14 Oct 2026

The Definitive Guide to Policy Gradients in Deep Reinforcement Learning: Theory, Algorithms and Implementations

arXiv 2401.13662 ↗First posted 24 Jan 2024cs.LG cs.AIcited by 5

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

In recent years, various powerful policy gradient algorithms have been proposed in deep reinforcement learning. While all these algorithms build on the Policy Gradient Theorem, the specific design choices differ significantly across algorithms. We provide a holistic overview of on-policy policy gradient algorithms to facilitate the understanding of both their theoretical foundations and their practical implementations. In this overview, we include a detailed proof of the continuous version of the Policy Gradient Theorem, convergence results and a comprehensive discussion of practical algorithms. We compare the most prominent algorithms on continuous control environments and provide insights on the benefits of regularization. All code is available at https://github.com/Matt00n/PolicyGradientsJax.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 31 of its 82 references that are in the map. First posted on arXiv 24 Jan 2024.

Builds on

31 of 82 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2017Proximal Policy Optimization Algorithmscited by 15,215
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  7. 2015Continuous control with deep reinforcement learningcited by 3,407
  8. 2015Trust Region Policy Optimizationcited by 2,580
  9. 2021Convex optimizationcited by 2,515
  10. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  11. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  12. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
19 more
  1. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  2. 2018Maximum a Posteriori Policy Optimisationcited by 328
  3. 2014Identifying and attacking the saddle point problem in high-dimensional non-convex optimizationcited by 289
  4. 2014The Loss Surfaces of Multilayer Networkscited by 280
  5. 2021Trust Region Policy Optimisation in Multi-Agent Reinforcement Learningcited by 186
  6. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 177
  7. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 162
  8. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  9. 2019Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policiescited by 124
  10. 2020Phasic Policy Gradientcited by 123
  11. 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
  12. 2021The Modern Mathematics of Deep Learningcited by 63
  13. 2020On Linear Convergence of Policy Gradient Methods for Finite MDPscited by 52
  14. 2019Neural Proximal/Trust Region Policy Optimization Attains Globally Optimal Policycited by 43
  15. 2018Relative Entropy Regularized Policy Iterationcited by 32
  16. 2022Mirror Learning: A Unifying Framework of Policy Optimisationcited by 31
  17. 2020Convergence Proof for Actor-Critic Methods Applied to PPO and RUDDERcited by 19
  18. 2022Robust Policy Optimization in Deep Reinforcement Learningcited by 15
  19. 2021Activation Functions in Artificial Neural Networks: A Systematic Overviewcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Stabilizing Policy Gradient Methods via Reward Profilingcited by 1
  2. 2026PointVG-R: Internalizing Geometric Reasoning in MLLMs for Precise Pointing Localization via Visual Chain of Thoughtcited by 1
  3. 2025GreenPlanner: Practical Floorplan Layout Generation via an Energy-Aware and Function-Feasible Generative Frameworkcited by 0
  4. 2025MPC-Guided Safe Reinforcement Learning and Lipschitz-Based Filtering for Structured Nonlinear Systemscited by 0
  5. 2026Expected Sample Complexity in Multi-Armed Banditscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Deterministic Policy Gradient for Reinforcement Learning with Continuous Time and State
  2. 2024A General Control-Theoretic Approach for Reinforcement Learning: Theory and Algorithms
  3. 2025A Practical Introduction to Deep Reinforcement Learning
  4. 2024Identifying Policy Gradient Subspaces
  5. 2024A Comprehensive Survey of Reinforcement Learning: From Algorithms to Practical Challenges
  6. 2024Deep Reinforcement Learning: A Convex Optimization Approach
  7. 2024Efficient Exploration in Deep Reinforcement Learning: A Novel Bayesian Actor-Critic Algorithm
  8. 2022Beyond the Policy Gradient Theorem for Efficient Policy Updates in Actor-Critic Algorithms