mlsanityData to 7 Oct 2026

Online Robust Reinforcement Learning with Model Uncertainty

arXiv 2109.14523 ↗First posted 29 Sep 2021cs.LGcited by 95

Not on the map. Why?

Abstract (arXiv)

Robust reinforcement learning (RL) is to find a policy that optimizes the worst-case performance over an uncertainty set of MDPs. In this paper, we focus on model-free robust RL, where the uncertainty set is defined to be centering at a misspecified MDP that generates a single sample trajectory sequentially and is assumed to be unknown. We develop a sample-based approach to estimate the unknown uncertainty set and design a robust Q-learning algorithm (tabular case) and robust TDC algorithm (function approximation setting), which can be implemented in an online and incremental fashion. For the robust Q-learning algorithm, we prove that it converges to the optimal robust Q function, and for the robust TDC algorithm, we prove that it converges asymptotically to some stationary points. Unlike the results in [Roy et al., 2017], our algorithms do not need any additional conditions on the discount factor to guarantee the convergence. We further characterize the finite-time error bounds of the two algorithms and show that both the robust Q-learning and robust TDC algorithms converge as fast as their vanilla counterparts(within a constant factor). Our numerical experiments further demonstrate the robustness of our algorithms. Our approach can be readily extended to robustify many other algorithms, e.g., TD, SARSA, and other GTD algorithms.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). Builds on 33 of its 67 references that are in the map. First posted on arXiv 29 Sep 2021.

Builds on

33 of 67 references.

References · 33

  1. 2017Robust Adversarial Reinforcement Learningcited by 439
  2. 2017Adversarial Attacks on Neural Network Policiescited by 314
  3. 2020On the Global Convergence Rates of Softmax Policy Gradient Methodscited by 258
  4. 2018A Finite Time Analysis of Temporal Difference Learning With Linear Function Approximationcited by 225
  5. 2020Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularizationcited by 170
  6. 2019Finite-Time Error Bounds For Linear Stochastic Approximation and TD Learningcited by 165
  7. 2017Tactics of Adversarial Attack on Deep Reinforcement Learning Agentscited by 163
  8. 2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 157
  9. 2017Robust Deep Reinforcement Learning with Adversarial Attackscited by 152
  10. 2016EPOpt: Learning Robust Neural Network Policies Using Model Ensemblescited by 136
  11. 2020A Finite Time Analysis of Two Time-Scale Actor Critic Methodscited by 120
  12. 2017Delving into adversarial attacks on deep policiescited by 104
21 more
  1. 2020Finite-Time Analysis of Asynchronous Stochastic Approximation and Q-Learningcited by 93
  2. 2020Sample Complexity of Asynchronous Q-Learning: Sharper Analysis and Variance Reductioncited by 93
  3. 2019Finite-Sample Analysis for SARSA with Linear Function Approximationcited by 85
  4. 2019Variance-reduced Q-learning is minimax optimalcited by 72
  5. 2020Robust Reinforcement Learning using Adversarial Populationscited by 64
  6. 2019On the Sample Complexity of Actor-Critic Method for Reinforcement Learning with Function Approximationcited by 54
  7. 2017Reinforcement Learning under Model Mismatchcited by 52
  8. 2020Finite Time Analysis of Linear Two-timescale Stochastic Approximation with Markovian Noisecited by 52
  9. 2019Wasserstein Robust Reinforcement Learningcited by 50
  10. 2019Finite-Time Performance Bounds and Adaptive Learning Rate Selection for Two Time-Scale Reinforcement Learningcited by 40
  11. 2020Finite-Sample Analysis of Proximal Gradient TD Algorithmscited by 32
  12. 2019Two Time-scale Off-Policy TD Learning: Non-asymptotic Analysis over Markovian Samplescited by 31
  13. 2019A Tale of Two-Timescale Reinforcement Learning with the Tightest Finite-Time Boundcited by 31
  14. 2020Sample Complexity Bounds for Two Timescale Value-based Reinforcement Learning Algorithmscited by 19
  15. 2019Finite-Time Analysis and Restarting Scheme for Linear Two-Time-Scale Stochastic Approximationcited by 18
  16. 2020Robust Reinforcement Learning with Wasserstein Constraintcited by 18
  17. 2020Finite-sample Analysis of Greedy-GQ with Linear Function Approximation under Markovian Noisecited by 17
  18. 2020Variance-Reduced Off-Policy TDC Learning: Non-Asymptotic Convergence Analysiscited by 12
  19. 2019Finite-Sample Analysis of Decentralized Temporal-Difference Learning with Linear Function Approximationcited by 9
  20. 2021Non-Asymptotic Analysis for Two Time-scale TDC with General Smooth Function Approximationcited by 8
  21. 2021Greedy-GQ with Variance Reduction: Finite-time Analysis and Improved Complexitycited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 14; 2023: 24; 2024: 19; 2025: 24; 2026: 14.

95 papers.

Sort:

By area:

  1. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  2. 2022Robust Reinforcement Learning using Offline Datacited by 95
  3. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexitycited by 85
  4. 2022Policy Gradient Method For Robust Reinforcement Learningcited by 67
  5. 2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coveragecited by 51
  6. 2023The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Modelcited by 50
  7. 2023Improved Sample Complexity Bounds for Distributionally Robust Reinforcement Learningcited by 44
  8. 2022Policy Gradient in Robust MDPs with Global Convergence Guaranteecited by 35
  9. 2023Natural Actor-Critic for Robust Reinforcement Learning with Function Approximationcited by 35
  10. 2023Robust Multi-Agent Reinforcement Learning with State Uncertaintycited by 35
  11. 2022First-order Policy Optimization for Robust Markov Decision Processcited by 31
  12. 2023Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlationcited by 25
38 more
  1. 2024Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithmscited by 24
  2. 2023Robust Average-Reward Markov Decision Processescited by 22
  3. 2023Model-Free Robust Average-Reward Reinforcement Learningcited by 20
  4. 2022Efficient Policy Iteration for Robust Markov Decision Processes via Regularizationcited by 19
  5. 2022Online Policy Optimization for Robust MDPcited by 19
  6. 2023Calibrating Multimodal Learningcited by 17
  7. 2023Distributionally Robust Model-based Reinforcement Learning with Large State Spacescited by 17
  8. 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learningcited by 17
  9. 2023Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexitycited by 16
  10. 2025RobustVLA: Robustness-Aware Reinforcement Post-Training for Vision-Language-Action Modelscited by 16
  11. 2023Single-Trajectory Distributionally Robust Reinforcement Learningcited by 15
  12. 2024Distributionally Robust Constrained Reinforcement Learning under Strong Dualitycited by 15
  13. 2022Robust Constrained Reinforcement Learningcited by 14
  14. 2023Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coveragecited by 12
  15. 2024Towards Robust Model-Based Reinforcement Learning Against Adversarial Corruptioncited by 12
  16. 2024Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processescited by 12
  17. 2023Solving Long-run Average Reward Robust MDPs via Stochastic Gamescited by 10
  18. 2024Near-Optimal Policy Identification in Robust Constrained Markov Decision Processes via Epigraph Formcited by 10
  19. 2024A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challengescited by 10
  20. 2023Adjustable Robust Reinforcement Learning for Online 3D Bin Packingcited by 9
  21. 2023MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operatorcited by 9
  22. 2024Model-Free Robust Reinforcement Learning with Sample Complexity Analysiscited by 9
  23. 2022Robust Reinforcement Learning in Continuous Control Tasks with Uncertainty Set Regularizationcited by 7
  24. 2023Bring Your Own (Non-Robust) Algorithm to Solve Robust MDPs by Estimating The Worst Kernelcited by 6
  25. 2024Constrained Reinforcement Learning Under Model Mismatchcited by 6
  26. 2024Certifiably Robust Policies for Uncertain Parametric Environmentscited by 6
  27. 2025Guaranteeing Out-Of-Distribution Detection in Deep RL via Transition Estimationcited by 6
  28. 2025Efficient Policy Optimization in Robust Constrained MDPs with Iteration Complexity Guaranteescited by 6
  29. 2025Confounding Robust Deep Reinforcement Learning: A Causal Approachcited by 6
  30. 2023An Efficient Solution to s-Rectangular Robust Markov Decision Processescited by 5
  31. 2023Multi-Agent Reinforcement Learning: Methods, Applications, Visionary Prospects, and Challengescited by 5
  32. 2024Roping in Uncertainty: Robustness and Regularization in Markov Gamescited by 5
  33. 2025Uncertainty in Action: Confidence Elicitation in Embodied Agentscited by 5
  34. 2025ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learningcited by 5
  35. 2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Dualitycited by 4
  36. 2024Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processescited by 3
  37. 2024Robust Model-Based Reinforcement Learning with an Adversarial Auxiliary Modelcited by 3
  38. 2025Dual Formulation for Non-Rectangular Lp Robust Markov Decision Processescited by 3

50 most cited of 95 All 95 (JSON)