mlsanityData to 7 Oct 2026

Policy Gradient for Robust Markov Decision Processes

arXiv 2410.22114 ↗First posted 29 Oct 2024cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

We develop a generic policy gradient method with the global optimality guarantee for robust Markov Decision Processes (MDPs). While policy gradient methods are widely used for solving dynamic decision problems due to their scalable and efficient nature, adapting these methods to account for model ambiguity has been challenging, often making it impractical to learn robust policies. This paper introduces a novel policy gradient method, Double-Loop Robust Policy Mirror Descent (DRPMD), for solving robust MDPs. DRPMD employs a general mirror descent update rule for the policy optimization with adaptive tolerance per iteration, guaranteeing convergence to a globally optimal policy. We provide a comprehensive analysis of DRPMD, including new convergence results under both direct and softmax parameterizations, and provide novel insights into the inner problem solution through Transition Mirror Ascent (TMA). Additionally, we propose innovative parametric transition kernels for both discrete and continuous state-action spaces, broadening the applicability of our approach. Empirical results validate the robustness and global convergence of DRPMD across various challenging robust MDP settings.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 11 of its 113 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

11 of 113 references.

References · 11

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016OpenAI Gymcited by 1,684
  3. 2021Global Convergence of Multi-Agent Policy Gradient in Markov Potential Gamescited by 89
  4. 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 82
  5. 2023Natural Actor-Critic for Robust Reinforcement Learning with Function Approximationcited by 35
  6. 2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximationcited by 31
  7. 2022First-order Policy Optimization for Robust Markov Decision Processcited by 31
  8. 2020Robust Constrained-MDPs: Soft-Constrained Robust Policy Optimization under Model Uncertaintycited by 23
  9. 2023Policy Gradient Algorithms for Robust MDPs with Non-Rectangular Uncertainty Setscited by 15
  10. 2022The Geometry of Robust Value Functionscited by 6
  11. 2024A Single-Loop Robust Policy Gradient Method for Robust Markov Decision Processescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Efficient Policy Optimization in Robust Constrained MDPs with Iteration Complexity Guaranteescited by 6
  2. 2026Robust Data-Collection Policy Learning for Low-Variance Online Policy Evaluationcited by 1
  3. 2026Policy Gradient for Continuous-Time Robust Markov Decision Processescited by 0