mlsanityData to 7 Oct 2026

Is the Policy Gradient a Gradient?

arXiv 1906.07073 ↗First posted 17 Jun 2019cs.LG stat.MLcited by 40

Not on the map. Why?

Abstract (arXiv)

The policy gradient theorem describes the gradient of the expected discounted return with respect to an agent's policy parameters. However, most policy gradient methods drop the discount factor from the state distribution and therefore do not optimize the discounted objective. What do they optimize instead? This has been an open question for several years, and this lack of theoretical clarity has lead to an abundance of misstatements in the literature. We answer this question by proving that the update direction approximated by most methods is not the gradient of any function. Further, we argue that algorithms that follow this direction are not guaranteed to converge to a "reasonable" fixed point by constructing a counterexample wherein the fixed point is globally pessimal with respect to both the discounted and undiscounted objectives. We motivate this work by surveying the literature and showing that there remains a widespread misunderstanding regarding discounted policy gradient methods, with errors present even in highly-cited papers published at top conferences.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 8; 2022: 9; 2023: 9; 2024: 6; 2025: 6; 2026: 1.

40 papers.

Sort:

By area:

  1. 2024An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learningcited by 46
  2. 2021Adversarial Intrinsic Motivation for Reinforcement Learningcited by 40
  3. 2021Greedification Operators for Policy Optimization: Investigating Forward and Reverse KL Divergencescited by 40
  4. 2021Multi-Agent Reinforcement Learning with Temporal Logic Specificationscited by 27
  5. 2024No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPOcited by 27
  6. 2024Streaming Deep Reinforcement Learning Finally Workscited by 24
  7. 2022Influencing Long-Term Behavior in Multiagent Reinforcement Learningcited by 18
  8. 2023Behavior Alignment via Reward Function Optimizationcited by 16
  9. 2024Deep Policy Gradient Methods Without Batch Updates, Target Networks, or Replay Bufferscited by 16
  10. 2021Continual Learning In Environments With Polynomial Mixing Timescited by 13
  11. 2020Universal Value Density Estimation for Imitation Learning and Goal-Conditioned Reinforcement Learningcited by 12
  12. 2023A Coupled Flow Approach to Imitation Learningcited by 12
28 more
  1. 2023Accelerating Value Iteration with Anchoringcited by 10
  2. 2023Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysiscited by 10
  3. 2022Cliff Diving: Exploring Reward Surfaces in Reinforcement Learning Environmentscited by 9
  4. 2023Correcting discount-factor mismatch in on-policy policy gradient methodscited by 9
  5. 2025An Optimisation Framework for Unsupervised Environment Designcited by 9
  6. 2021Examining average and discounted reward optimality criteria in reinforcement learningcited by 6
  7. 2023On-Policy Policy Gradient Reinforcement Learning Without On-Policy Samplingcited by 6
  8. 2025Relative Entropy Pathwise Policy Optimizationcited by 6
  9. 2021Dr Jekyll and Mr Hyde: the Strange Case of Off-Policy Policy Updatescited by 5
  10. 2021Off-Policy Actor-Critic with Emphatic Weightingscited by 5
  11. 2021Taylor Expansion of Discount Factorscited by 4
  12. 2022A Temporal-Difference Approach to Policy Gradient Estimationcited by 4
  13. 2023A Tale of Sampling and Estimation in Discounted Reinforcement Learningcited by 4
  14. 2024OMPO: A Unified Framework for RL under Policy and Dynamics Shiftscited by 4
  15. 2022Is Vanilla Policy Gradient Overlooked? Analyzing Deep Reinforcement Learning for Hanabicited by 3
  16. 2022Beyond the Policy Gradient Theorem for Efficient Policy Updates in Actor-Critic Algorithmscited by 2
  17. 2022Distributional Reinforcement Learning for Scheduling of Chemical Production Processescited by 2
  18. 2025Optimizing Navigation And Chemical Application in Precision Agriculture With Deep Reinforcement Learning And Conditional Action Treecited by 2
  19. 2022On Many-Actions Policy Gradientcited by 1
  20. 2023Revisiting Estimation Bias in Policy Gradients for Deep Reinforcement Learningcited by 1
  21. 2025Analysis of On-policy Policy Gradient Methods under the Distribution Mismatchcited by 1
  22. 2025Reinforcement Learning with Random Time Horizonscited by 1
  23. 20223DPG: Distributed Deep Deterministic Policy Gradient Algorithms for Networked Multi-Agent Systemscited by 0
  24. 2022On the Convergence of Discounted Policy Gradient Methodscited by 0
  25. 2023An introduction to reinforcement learning for neurosciencecited by 0
  26. 2024Feint Behaviors and Strategies: Formalization, Implementation and Evaluationcited by 0
  27. 2025Policy Optimization Algorithms in a Unified Frameworkcited by 0
  28. 2026PACT: Self-Evolving Physical Safety Alignment for Diffusion Policies in Embodied Manipulationcited by 0