mlsanityData to 14 Oct 2026

Variational Policy Gradient Method for Reinforcement Learning with General Utilities

arXiv 2007.02151 ↗First posted 4 Jul 2020cs.LG stat.MLcited by 111

Not on the map. Why?

Cited by 35 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In recent years, reinforcement learning (RL) systems with general goals beyond a cumulative sum of rewards have gained traction, such as in constrained problems, exploration, and acting upon prior experiences. In this paper, we consider policy optimization in Markov Decision Problems, where the objective is a general concave utility function of the state-action occupancy measure, which subsumes several of the aforementioned examples as special cases. Such generality invalidates the Bellman equation. As this means that dynamic programming no longer works, we focus on direct policy search. Analogously to the Policy Gradient Theorem citesutton2000policy available for RL with cumulative rewards, we derive a new Variational Policy Gradient Theorem for RL with general utilities, which establishes that the parametrized policy gradient may be obtained as the solution of a stochastic saddle point problem involving the Fenchel dual of the utility function. We develop a variational Monte Carlo gradient estimation algorithm to compute the policy gradient based on sample paths. We prove that the variational policy gradient scheme converges globally to the optimal policy for the general objective, though the optimization problem is nonconvex. We also establish its rate of convergence of the order O(1/t) by exploiting the hidden convexity of the problem, and proves that it converges exponentially when the problem admits hidden strong convexity. Our analysis applies to the standard RL problem with cumulative rewards as a special case, in which case our result improves the available convergence rate.

Key facts. Cited by 111 papers in the map (data through 14 Oct 2026). First posted on arXiv 4 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 22; 2022: 19; 2023: 21; 2024: 24; 2025: 9; 2026: 15.

111 papers.

Sort:

By area:

  1. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  2. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  3. 2022On the Convergence Rates of Policy Gradient Methodscited by 119
  4. 2022Jump-Start Reinforcement Learningcited by 112
  5. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 110
  6. 2021Policy Mirror Descent for Regularized Reinforcement Learning: A Generalized Framework with Linear Convergencecited by 84
  7. 2021Reward is enough for convex MDPscited by 84
  8. 2021A general sample complexity analysis of vanilla policy gradientcited by 76
  9. 2021On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Methodcited by 70
  10. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 69
  11. 2023Stochastic Policy Gradient Methods: Improved Sample Complexity for Fisher-non-degenerate Policiescited by 58
  12. 2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpointcited by 56
38 more
  1. 2021Softmax Policy Gradient Methods Can Take Exponential Time to Convergecited by 48
  2. 2021On the Linear convergence of Natural Policy Gradient Algorithmcited by 48
  3. 2022Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policiescited by 46
  4. 2022Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimalitycited by 37
  5. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  6. 2021Regularized Policies are Reward Robustcited by 32
  7. 2022Challenging Common Assumptions in Convex Reinforcement Learningcited by 30
  8. 2023A Novel Framework for Policy Mirror Descent with General Parameterization and Linear Convergencecited by 30
  9. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangiancited by 29
  10. 2022Convergence and sample complexity of natural policy gradient primal-dual methods for constrained MDPscited by 27
  11. 2021On the Global Optimum Convergence of Momentum-based Policy Gradientcited by 26
  12. 2021Beyond Exact Gradients: Convergence of Stochastic Soft-Max Policy Gradient Methods with Entropy Regularizationcited by 25
  13. 2021Finite-Sample Analysis of Off-Policy Natural Actor-Critic with Linear Function Approximationcited by 22
  14. 2023Deep Reinforcement Learning for Cost-Effective Medical Diagnosiscited by 21
  15. 2022Stochastic Second-Order Methods Improve Best-Known Sample Complexity of SGD for Gradient-Dominated Functioncited by 20
  16. 2021Concave Utility Reinforcement Learning with Zero-Constraint Violationscited by 19
  17. 2024Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergencecited by 19
  18. 2023Reinforcement Learning with General Utilities: Simpler Variance Reduction and Large State-Action Spacecited by 18
  19. 2024Elementary Analysis of Policy Gradient Methodscited by 18
  20. 2025Flow Density Control: Generative Optimization Beyond Entropy-Regularized Fine-Tuningcited by 17
  21. 2022Anchor-Changing Regularized Natural Policy Gradient for Multi-Objective Reinforcement Learningcited by 15
  22. 2023Scalable Primal-Dual Actor-Critic Method for Safe Multi-Agent RL with General Utilitiescited by 15
  23. 2023Diversifying AI: Towards Creative Chess with AlphaZerocited by 15
  24. 2024Global Reinforcement Learning: Beyond Linear and Convex Rewards via Submodular Semi-gradient Methodscited by 15
  25. 2023Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learningcited by 13
  26. 2021Joint Optimization of Multi-Objective Reinforcement Learning with Policy Gradient Based Algorithmcited by 12
  27. 2021On the Sample Complexity and Metastability of Heavy-tailed Policy Search in Continuous Controlcited by 12
  28. 2023A Coupled Flow Approach to Imitation Learningcited by 12
  29. 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inferencecited by 12
  30. 2024Taming Nonconvex Stochastic Mirror Descent with General Bregman Divergencecited by 11
  31. 2022On the convex formulations of robust Markov decision processescited by 10
  32. 2024Quantum Advantage Actor-Critic for Reinforcement Learningcited by 9
  33. 2024On the Limitations of Markovian Rewards to Express Multi-Objective, Risk-Sensitive, and Modal Taskscited by 9
  34. 2024On the Global Convergence of Policy Gradient in Average Reward Markov Decision Processescited by 9
  35. 2023Inverse Reinforcement Learning with the Average Reward Criterioncited by 8
  36. 2023Active Coverage for PAC Reinforcement Learningcited by 8
  37. 2021Theoretical Guarantees of Fictitious Discount Algorithms for Episodic Reinforcement Learning and Global Convergence of Policy Gradient Methodscited by 7
  38. 2024Towards Principled, Practical Policy Gradient for Bandits and Tabular MDPscited by 7

50 most cited of 111 All 111 (JSON)