mlsanityData to 14 Oct 2026

Solving Robust MDPs through No-Regret Dynamics

arXiv 2305.19035 ↗First posted 30 May 2023cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Reinforcement Learning is a powerful framework for training agents to navigate different situations, but it is susceptible to changes in environmental dynamics. However, solving Markov Decision Processes that are robust to changes is difficult due to nonconvexity and size of action or state spaces. While most works have analyzed this problem by taking different assumptions on the problem, a general and efficient theoretical analysis is still missing. However, we generate a simple framework for improving robustness by solving a minimax iterative optimization problem where a policy player and an environmental dynamics player are playing against each other. Leveraging recent results in online nonconvex learning and techniques from improving policy gradient methods, we yield an algorithm that maximizes the robustness of the Value Function on the order of Oleft(frac1T^frac12right) where T is the number of iterations of the algorithm.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 62 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

33 of 62 references.

References · 33

  1. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 539
  2. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  3. 2017Training GANs with Optimismcited by 217
  4. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 177
  5. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  6. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 174
  7. 2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 154
  8. 2015Fast Convergence of Regularized Learning in Gamescited by 150
  9. 2019Global Optimality Guarantees For Policy Gradient Methodscited by 150
  10. 2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 135
  11. 2018Assessing Generalization in Deep Reinforcement Learningcited by 133
  12. 2018The Mechanics of n-Player Differentiable Gamescited by 118
21 more
  1. 2018Generalization and Regularization in DQNcited by 118
  2. 2020Learning to be Safe: Deep RL with a Safety Criticcited by 116
  3. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 96
  4. 2019Observational Overfitting in Reinforcement Learningcited by 91
  5. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 82
  6. 2019Robust Reinforcement Learning for Continuous Control with Model Misspecificationcited by 78
  7. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 69
  8. 2016Safe Policy Improvement by Minimizing Robust Baseline Regretcited by 53
  9. 2017Reinforcement Learning under Model Mismatchcited by 52
  10. 2021Robust Control Barrier-Value Functions for Safety-Critical Controlcited by 51
  11. 2019Online Non-Convex Learning: Following the Perturbed Leader is Optimalcited by 32
  12. 2023Robust Average-Reward Markov Decision Processescited by 22
  13. 2021No-Regret Dynamics in the Fenchel Game: A Unified Framework for Algorithmic Convex Optimizationcited by 20
  14. 2022Online Policy Optimization for Robust MDPcited by 19
  15. 2018Learning Robust Optionscited by 17
  16. 2020First-Order Methods for Wasserstein Distributionally Robust MDPcited by 17
  17. 2013Minimax Optimal Algorithms for Unconstrained Linear Optimizationcited by 12
  18. 2018Distributionally Robust Optimization for Sequential Decision Makingcited by 12
  19. 2022On Accelerated Perceptrons and Beyondcited by 6
  20. 2020Robust Reinforcement Learning using Least Squares Policy Iteration with Provable Performance Guaranteescited by 4
  21. 2020Soft-Robust Algorithms for Batch Reinforcement Learningcited by 3

Built on by

Not cited yet.