mlsanityData to 7 Oct 2026

Policy Gradient in Robust MDPs with Global Convergence Guarantee

arXiv 2212.10439 ↗First posted 20 Dec 2022cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

Robust Markov decision processes (RMDPs) provide a promising framework for computing reliable policies in the face of model errors. Many successful reinforcement learning algorithms build on variations of policy-gradient methods, but adapting these methods to RMDPs has been challenging. As a result, the applicability of RMDPs to large, practical domains remains limited. This paper proposes a new Double-Loop Robust Policy Gradient (DRPG), the first generic policy gradient method for RMDPs. In contrast with prior robust policy gradient algorithms, DRPG monotonically reduces approximation errors to guarantee convergence to a globally optimal policy in tabular RMDPs. We introduce a novel parametric transition kernel and solve the inner loop robust policy via a gradient-based method. Finally, our numerical results demonstrate the utility of our new algorithm and confirm its global convergence properties.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 33 of its 79 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

33 of 79 references.

References · 33

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  4. 2019On Gradient Descent Ascent for Nonconvex-Concave Minimax Problemscited by 266
  5. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
  6. 2018Stochastic model-based minimization of weakly convex functionscited by 157
  7. 2019Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methodscited by 148
  8. 2019A Unified Analysis of Extra-gradient and Optimistic Gradient Methods for Saddle Point Problems: Proximal Point Approachcited by 142
  9. 2021Independent Policy Gradient Methods for Competitive Reinforcement Learningcited by 97
  10. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 95
  11. 2022Robust Reinforcement Learning using Offline Datacited by 95
  12. 2021Global Convergence of Multi-Agent Policy Gradient in Markov Potential Gamescited by 89
21 more
  1. 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 82
  2. 2019Efficient Algorithms for Smooth Minimax Optimizationcited by 75
  3. 2020Stochastic Recursive Gradient Descent Ascent for Stochastic Nonconvex-Strongly-Concave Minimax Problemscited by 75
  4. 2013Accelerated Gradient Methods for Nonconvex Nonlinear and Stochastic Programmingcited by 74
  5. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 67
  6. 2020A Single-Loop Smoothed Gradient Descent-Ascent Algorithm for Nonconvex-Concave Min-Max Problemscited by 57
  7. 2017Reinforcement Learning under Model Mismatchcited by 52
  8. 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 51
  9. 2020On Linear Convergence of Policy Gradient Methods for Finite MDPscited by 51
  10. 2020Partial Policy Iteration for L1-Robust Markov Decision Processescited by 49
  11. 2021Twice regularized MDPs and the equivalence between robustness and regularizationcited by 46
  12. 2014Approximate Policy Iteration Schemes: A Comparisoncited by 39
  13. 2019Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPscited by 37
  14. 2020Non-convex Min-Max Optimization: Applications, Challenges, and Recent Theoretical Advancescited by 37
  15. 2020Convergence of a Stochastic Gradient Method with Momentum for Non-Smooth Non-Convex Optimizationcited by 32
  16. 2022First-order Policy Optimization for Robust Markov Decision Processcited by 31
  17. 2020Robust Constrained-MDPs: Soft-Constrained Robust Policy Optimization under Model Uncertaintycited by 23
  18. 2020Scalable First-Order Methods for Robust MDPscited by 22
  19. 2020First-Order Methods for Wasserstein Distributionally Robust MDPcited by 16
  20. 2018Distributionally Robust Optimization for Sequential Decision Makingcited by 11
  21. 2022The Geometry of Robust Value Functionscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 9; 2025: 14; 2026: 6.

35 papers.

Sort:

By area:

  1. 2023Policy Gradient for Rectangular Robust Markov Decision Processescited by 29
  2. 2024Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithmscited by 24
  3. 2024Robust Markov Decision Processes: A Place Where AI and Formal Methods Meetcited by 18
  4. 2023Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexitycited by 16
  5. 2023Policy Gradient Algorithms for Robust MDPs with Non-Rectangular Uncertainty Setscited by 15
  6. 2023Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coveragecited by 12
  7. 2024Near-Optimal Policy Identification in Robust Constrained Markov Decision Processes via Epigraph Formcited by 10
  8. 2023Solving Non-Rectangular Reward-Robust MDPs via Frequency Regularizationcited by 8
  9. 2024Imprecise Probabilities Meet Partial Observability: Game Semantics for Robust POMDPscited by 7
  10. 2023Bring Your Own (Non-Robust) Algorithm to Solve Robust MDPs by Estimating The Worst Kernelcited by 6
  11. 2024Constrained Reinforcement Learning Under Model Mismatchcited by 6
  12. 2025Efficient Policy Optimization in Robust Constrained MDPs with Iteration Complexity Guaranteescited by 6
23 more
  1. 2024Roping in Uncertainty: Robustness and Regularization in Markov Gamescited by 5
  2. 2025Robust Finite-Memory Policy Gradients for Hidden-Model POMDPscited by 5
  3. 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Dualitycited by 4
  4. 2024Stochastic Weakly Convex Optimization Beyond Lipschitz Continuitycited by 3
  5. 2025Dual Formulation for Non-Rectangular Lp Robust Markov Decision Processescited by 3
  6. 2025Qualitative Analysis of ω-Regular Objectives on Robust MDPscited by 2
  7. 2025Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learningcited by 2
  8. 2025Achieve Performatively Optimal Policy for Performative Reinforcement Learningcited by 2
  9. 2024Robust Reinforcement Learning with Dynamic Distortion Risk Measurescited by 1
  10. 2024Solving robust MDPs as a sequence of static RL problemscited by 1
  11. 2025Microservice Deployment in Space Computing Power Networks via Robust Reinforcement Learningcited by 1
  12. 2025Distributionally Robust Deep Q-Learningcited by 1
  13. 2025Distributionally Robust Self Paced Curriculum Reinforcement Learningcited by 1
  14. 2025Primal-Only Actor Critic Algorithm for Robust Constrained Average Cost MDPscited by 1
  15. 2025Provably Efficient Sample Complexity for Robust CMDPcited by 1
  16. 2026Provably Efficient Algorithms for S- and Non-Rectangular Robust MDPs with General Parameterizationcited by 1
  17. 2026Robust General Utility for Reinforcement Learningcited by 1
  18. 2026Robust Data-Collection Policy Learning for Low-Variance Online Policy Evaluationcited by 1
  19. 2025ADARL: Adaptive Low-Rank Structures for Robust Policy Learning under Uncertaintycited by 0
  20. 2025Robust Reinforcement Learning in Finance: Modeling Market Impact with Elliptic Uncertainty Setscited by 0
  21. 2026Optimistic Policy Learning under Pessimistic Adversaries with Regret and Violation Guaranteescited by 0
  22. 2026Central Limit Theorem for Two-Time-Scale Approximate Distributionally Robust RLcited by 0
  23. 2026Entropy Regularization Improves Policy Robustness in Continuous-Time Reinforcement Learningcited by 0