mlsanityData to 7 Oct 2026

Policy Gradient Method For Robust Reinforcement Learning

arXiv 2205.07344 ↗First posted 15 May 2022cs.LGcited by 67

Not on the map. Why?

Abstract (arXiv)

This paper develops the first policy gradient method with global optimality guarantee and complexity analysis for robust reinforcement learning under model mismatch. Robust reinforcement learning is to learn a policy robust to model mismatch between simulator and real environment. We first develop the robust policy (sub-)gradient, which is applicable for any differentiable parametric policy class. We show that the proposed robust policy gradient method converges to the global optimum asymptotically under direct policy parameterization. We further develop a smoothed robust policy gradient method and show that to achieve an ε-global optimum, the complexity is mathcal O(ε-3). We then extend our methodology to the general model-free setting and design the robust actor-critic method with differentiable parametric policy class and value function. We further characterize its asymptotic convergence and sample complexity under the tabular setting. Finally, we provide simulation results to demonstrate the robustness of our methods.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Builds on 49 of its 96 references that are in the map. First posted on arXiv 15 May 2022.

Builds on

49 of 96 references.

References · 49

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2016OpenAI Gymcited by 1,684
  4. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  5. 2017Constrained Policy Optimizationcited by 928
  6. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  7. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  8. 2017Robust Adversarial Reinforcement Learningcited by 439
  9. 2017Adversarial Attacks on Neural Network Policiescited by 314
  10. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
  11. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 225
  12. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 170
37 more
  1. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 169
  2. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 167
  3. 2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 163
  4. 2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 152
  5. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 146
  6. 2013Mini-batch Stochastic Approximation Methods for Nonconvex Stochastic Composite Optimizationcited by 142
  7. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 136
  8. 2022On the Convergence Rates of Policy Gradient Methodscited by 115
  9. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 108
  10. 2017Delving into adversarial attacks on deep policiescited by 104
  11. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 95
  12. 2017Implicit Regularization in Deep Learningcited by 90
  13. 2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 82
  14. 2018Robust Estimation via Robust Gradient Estimationcited by 76
  15. 2013Accelerated Gradient Methods for Nonconvex Nonlinear and Stochastic Programmingcited by 74
  16. 2020Robust Reinforcement Learning using Adversarial Populationscited by 64
  17. 2017Reinforcement Learning under Model Mismatchcited by 52
  18. 2020On Linear Convergence of Policy Gradient Methods for Finite MDPscited by 51
  19. 2019Wasserstein Robust Reinforcement Learningcited by 50
  20. 2020Partial Policy Iteration for L1-Robust Markov Decision Processescited by 49
  21. 2021Softmax Policy Gradient Methods Can Take Exponential Time to Convergecited by 46
  22. 2021Twice regularized MDPs and the equivalence between robustness and regularizationcited by 46
  23. 2015Distributionally Robust Counterpart in Markov Decision Processescited by 32
  24. 2021Robust Policy Gradient against Strong Data Corruptioncited by 30
  25. 2021Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptoticscited by 27
  26. 2018Robust Markov Decision Process: Beyond Rectangularitycited by 24
  27. 2020Robust Constrained-MDPs: Soft-Constrained Robust Policy Optimization under Model Uncertaintycited by 23
  28. 2018Analysis of nonsmooth stochastic approximation: the differential inclusion approachcited by 21
  29. 2014On TD(0) with function approximation: Concentration bounds and a centered variant with exponential convergencecited by 20
  30. 2020Robust Reinforcement Learning with Wasserstein Constraintcited by 18
  31. 2020Finite-sample Analysis of Greedy-GQ with Linear Function Approximation under Markovian Noisecited by 17
  32. 2021Global Optimality and Finite Sample Analysis of Softmax Off-Policy Actor Critic under State Distribution Mismatchcited by 17
  33. 2020Stable Policy Optimization via Off-Policy Divergence Regularizationcited by 16
  34. 2020Complexity of Finding Stationary Points of Nonsmooth Nonconvex Functionscited by 13
  35. 2019Convergence of a Stochastic Subgradient Method with Averaging for Nonsmooth Nonconvex Constrained Optimizationcited by 9
  36. 2018Robust Nonparametric Regression under Huber's ε-contamination Modelcited by 6
  37. 2021Dr Jekyll and Mr Hyde: the Strange Case of Off-Policy Policy Updatescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 19; 2024: 11; 2025: 20; 2026: 9.

67 papers.

Sort:

By area:

  1. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coveragecited by 51
  2. 2022Policy Gradient in Robust MDPs with Global Convergence Guaranteecited by 35
  3. 2023Natural Actor-Critic for Robust Reinforcement Learning with Function Approximationcited by 35
  4. 2022First-order Policy Optimization for Robust Markov Decision Processcited by 31
  5. 2023Policy Gradient for Rectangular Robust Markov Decision Processescited by 29
  6. 2023Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithmscited by 25
  7. 2024Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithmscited by 24
  8. 2023Robust Average-Reward Markov Decision Processescited by 22
  9. 2023Model-Free Robust Average-Reward Reinforcement Learningcited by 20
  10. 2022Efficient Policy Iteration for Robust Markov Decision Processes via Regularizationcited by 19
  11. 2022Online Policy Optimization for Robust MDPcited by 19
  12. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimizationcited by 17
38 more
  1. 2023Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexitycited by 16
  2. 2023Policy Gradient Algorithms for Robust MDPs with Non-Rectangular Uncertainty Setscited by 15
  3. 2024Distributionally Robust Constrained Reinforcement Learning under Strong Dualitycited by 15
  4. 2022Robust Constrained Reinforcement Learningcited by 14
  5. 2023Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coveragecited by 12
  6. 2022A Robust and Constrained Multi-Agent Reinforcement Learning Electric Vehicle Rebalancing Method in AMoD Systemscited by 11
  7. 2023Solving Long-run Average Reward Robust MDPs via Stochastic Gamescited by 10
  8. 2024Near-Optimal Policy Identification in Robust Constrained Markov Decision Processes via Epigraph Formcited by 10
  9. 2023Adjustable Robust Reinforcement Learning for Online 3D Bin Packingcited by 9
  10. 2023Solving Non-Rectangular Reward-Robust MDPs via Frequency Regularizationcited by 8
  11. 2023Bring Your Own (Non-Robust) Algorithm to Solve Robust MDPs by Estimating The Worst Kernelcited by 6
  12. 2023First-order Policy Optimization for Robust Policy Evaluationcited by 6
  13. 2024Constrained Reinforcement Learning Under Model Mismatchcited by 6
  14. 2025Finite-Sample Analysis of Policy Evaluation for Robust Average Reward Reinforcement Learningcited by 6
  15. 2025DeepSeek-Inspired Exploration of RL-based LLMs and Synergy with Wireless Networks: A Surveycited by 6
  16. 2026Flow Actor-Critic for Offline Reinforcement Learningcited by 6
  17. 2023An Efficient Solution to s-Rectangular Robust Markov Decision Processescited by 5
  18. 2024Roping in Uncertainty: Robustness and Regularization in Markov Gamescited by 5
  19. 2025ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learningcited by 5
  20. 2024FLAME: Self-Supervised Low-Resource Taxonomy Expansion using Large Language Modelscited by 4
  21. 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Dualitycited by 4
  22. 2025Performative Policy Gradient: Optimality in Performative Reinforcement Learningcited by 4
  23. 2024Robust Model-Based Reinforcement Learning with an Adversarial Auxiliary Modelcited by 3
  24. 2025Reinforcement Learning with Continuous Actions Under Unmeasured Confoundingcited by 3
  25. 2025Hybrid Cross-domain Robust Reinforcement Learningcited by 3
  26. 2025DR-SAC: Distributionally Robust Soft Actor-Critic for Reinforcement Learning under Uncertaintycited by 3
  27. 2024Model-Free Robust φ-Divergence Reinforcement Learning Using Both Offline and Online Datacited by 2
  28. 2024Q-Learning under Finite Model Uncertaintycited by 2
  29. 2025Qualitative Analysis of ω-Regular Objectives on Robust MDPscited by 2
  30. 2025Pessimism Principle Can Be Effective: Towards a Framework for Zero-Shot Transfer Reinforcement Learningcited by 2
  31. 2025Efficient Q-Learning and Actor-Critic Methods for Robust Average-Reward Reinforcement Learningcited by 2
  32. 2026Robust Adversarial Policy Optimization Under Dynamics Uncertaintycited by 2
  33. 2022Robust Phi-Divergence MDPscited by 1
  34. 2023Twice Regularized Markov Decision Processes: The Equivalence between Robustness and Regularizationcited by 1
  35. 2024A Generalized Version of Chung's Lemma and its Applicationscited by 1
  36. 2025Microservice Deployment in Space Computing Power Networks via Robust Reinforcement Learningcited by 1
  37. 2025Distributionally Robust Deep Q-Learningcited by 1
  38. 2025Action Robust Reinforcement Learning via Optimal Adversary Aware Policy Optimizationcited by 1

50 most cited of 67 All 67 (JSON)