mlsanityData to 14 Oct 2026

On the Global Optimality of Policy Gradient Methods in General Utility Reinforcement Learning

arXiv 2410.04108 ↗First posted 5 Oct 2024cs.LG cs.AIcited by 6

On the map: Offline RL and imitation learning from data → Offline RL algorithms

also in 1 more
Abstract (arXiv)

Reinforcement learning with general utilities (RLGU) offers a unifying framework to capture several problems beyond standard expected returns, including imitation learning, pure exploration, and safe RL. Despite recent fundamental advances in the theoretical analysis of policy gradient (PG) methods for standard RL and recent efforts in RLGU, the understanding of these PG algorithms and their scope of application in RLGU still remain limited. In this work, we establish global optimality guarantees of PG methods for RLGU in which the objective is a general concave utility function of the state-action occupancy measure. In the tabular setting, we provide global optimality results using a new proof technique building on recent theoretical developments on the convergence of PG methods for standard RL using gradient domination. Our proof technique opens avenues for analyzing policy parameterizations beyond the direct policy parameterization for RLGU. In addition, we provide global optimality results for large state-action space settings beyond prior work which has mostly focused on the tabular setting. In this large scale setting, we adapt PG methods by approximating occupancy measures within a function approximation class using maximum likelihood estimation. Our sample complexity only scales with the dimension induced by our approximation class instead of the size of the state-action space.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 44 references that are in the map. First posted on arXiv 5 Oct 2024.

Builds on

29 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Challenging Common Assumptions in Convex Reinforcement Learning3 Feb

Other references · 28

  1. 2016Generative Adversarial Imitation Learningcited by 1,597
  2. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  3. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  4. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 264
  5. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  6. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 233
  7. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
  8. 2022On the Convergence Rates of Policy Gradient Methodscited by 119
  9. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 111
  10. 2021Reward is enough for convex MDPscited by 84
  11. 2021On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Methodcited by 70
  12. 2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpointcited by 56
16 more
  1. 2020On Linear Convergence of Policy Gradient Methods for Finite MDPscited by 52
  2. 2017Consistent On-Line Off-Policy Evaluationcited by 51
  3. 2022Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policiescited by 46
  4. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  5. 2015Cumulative Prospect Theory Meets Reinforcement Learning: Prediction and Controlcited by 34
  6. 2023Submodular Reinforcement Learningcited by 23
  7. 2022Active Exploration via Experiment Design in Markov Chainscited by 19
  8. 2023Reinforcement Learning in Low-Rank MDPs with Density Featurescited by 18
  9. 2023Reinforcement Learning with General Utilities: Simpler Variance Reduction and Large State-Action Spacecited by 18
  10. 2024Elementary Analysis of Policy Gradient Methodscited by 18
  11. 2023Scalable Primal-Dual Actor-Critic Method for Safe Multi-Agent RL with General Utilitiescited by 15
  12. 2024Global Reinforcement Learning: Beyond Linear and Convex Rewards via Submodular Semi-gradient Methodscited by 15
  13. 2021Joint Optimization of Multi-Objective Reinforcement Learning with Policy Gradient Based Algorithmcited by 12
  14. 2023Stochastic Optimization under Hidden Convexitycited by 12
  15. 2022Policy Gradient for Reinforcement Learning with General Utilitiescited by 6
  16. 2023On the Convergence of Projected Policy Gradient for Any Constant Step Sizescited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Soft Forward-Backward Representations for Zero-shot Reinforcement Learning with General Utilitiescited by 3
  2. 2026Breaking the Bias Barrier in Concave Multi-Objective Reinforcement Learningcited by 2
  3. 2026Robust Optimization for Mitigating Reward Hacking with Correlated Proxiescited by 2
  4. 2026Convex Markov Games and Beyond: New Proof of Existence, Characterization and Learning Algorithms for Nash Equilibriacited by 1
  5. 2026Robust General Utility for Reinforcement Learningcited by 1
  6. 2026Online Learning on Hidden-Convex Losses via Algorithmic Equivalence: Optimal Regret, Geometric Barrier, and Bandit Feedbackcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Generalized Proximal Policy Optimization: Unifying LLM RL Algorithms from the Perspective of Divergence
  2. 2024Learning Optimal Deterministic Policies with Stochastic Policy Gradients
  3. 2026Mathematical methods of reinforcement learning
  4. 2022On Practical Reinforcement Learning: Provable Robustness, Scalability, and Statistical Efficiency
  5. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization
  6. 2026Global Optimality for Constrained Exploration via Penalty Regularization
  7. 2026Towards a Theoretical Understanding to the Generalization of RLHF
  8. 2024A General Control-Theoretic Approach for Reinforcement Learning: Theory and Algorithms