Data to 7 Oct 2026

Reinforcement Learning under Model Mismatch

arXiv 1706.04711 ↗First posted 15 Jun 2017cs.LG stat.MLcited by 52

Not on the map. Why?

Abstract (arXiv)

We study reinforcement learning under model misspecification, where we do not have access to the true environment but only to a reasonably close approximation to it. We address this problem by extending the framework of robust MDPs to the model-free Reinforcement Learning setting, where we do not have access to the model parameters, but can only sample states from it. We define robust versions of Q-learning, SARSA, and TD-learning and prove convergence to an approximately optimal robust policy and approximate value function respectively. We scale up the robust algorithms to large MDPs via function approximation and prove convergence under two different settings. We prove convergence of robust approximate policy iteration and robust approximate value iteration for linear architectures (under mild assumptions). We also define a robust loss function, the mean squared robust projected Bellman error and give stochastic gradient descent algorithms that are guaranteed to converge to a local minimum.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Jun 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 14; 2023: 17; 2024: 5; 2025: 5; 2026: 5.

52 papers.

Sort:

By area:

2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
2021Online Robust Reinforcement Learning with Model Uncertaintycited by 95
2022Robust Reinforcement Learning using Offline Datacited by 95
2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexitycited by 85
2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 83
2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 82
2022Policy Gradient Method For Robust Reinforcement Learningcited by 67
2023The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Modelcited by 50
2021Twice regularized MDPs and the equivalence between robustness and regularizationcited by 46
2023Improved Sample Complexity Bounds for Distributionally Robust Reinforcement Learningcited by 44
2022Policy Gradient in Robust MDPs with Global Convergence Guaranteecited by 35
2023Natural Actor-Critic for Robust Reinforcement Learning with Function Approximationcited by 35

38 more

2022First-order Policy Optimization for Robust Markov Decision Processcited by 31
2023Policy Gradient for Rectangular Robust Markov Decision Processescited by 29
2023Robust Average-Reward Markov Decision Processescited by 22
2023Model-Free Robust Average-Reward Reinforcement Learningcited by 20
2023Distributionally Robust Model-based Reinforcement Learning with Large State Spacescited by 17
2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learningcited by 17
2022Robust Policy Learning over Multiple Uncertainty Setscited by 16
2023Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexitycited by 16
2023Single-Trajectory Distributionally Robust Reinforcement Learningcited by 15
2022Robust Constrained Reinforcement Learningcited by 14
2023Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coveragecited by 12
2024Towards Robust Model-Based Reinforcement Learning Against Adversarial Corruptioncited by 12
2022Robust Imitation Learning from Corrupted Demonstrationscited by 10
2023Solving Long-run Average Reward Robust MDPs via Stochastic Gamescited by 10
2022Robust Reinforcement Learning in Continuous Control Tasks with Uncertainty Set Regularizationcited by 7
2023First-order Policy Optimization for Robust Policy Evaluationcited by 6
2024Constrained Reinforcement Learning Under Model Mismatchcited by 6
2025Confounding Robust Deep Reinforcement Learning: A Causal Approachcited by 6
2021Disturbing Reinforcement Learning Agents with Corrupted Rewardscited by 4
2022Policy Learning for Robust Markov Decision Process with a Mismatched Generative Modelcited by 4
2022Environment Design for Inverse Reinforcement Learningcited by 4
2025Rectified Robust Policy Optimization for Model-Uncertain Constrained Reinforcement Learning without Strong Dualitycited by 4
2021Robust Reinforcement Learning under model misspecificationcited by 2
2021Online Algorithms and Policies Using Adaptive and Machine Learning Approachescited by 2
2026Robust Adversarial Policy Optimization Under Dynamics Uncertaintycited by 2
2022Robust Phi-Divergence MDPscited by 1
2023Twice Regularized Markov Decision Processes: The Equivalence between Robustness and Regularizationcited by 1
2025CAMP in the Odyssey: Provably Robust Reinforcement Learning with Certified Radius Maximizationcited by 1
2023Risk-Averse MDPs under Reward Ambiguitycited by 0
2023Risk-Sensitive and Robust Model-Based Reinforcement Learning and Planningcited by 0
2023On Practical Robust Reinforcement Learning: Practical Uncertainty Set and Double-Agent Algorithmcited by 0
2023Solving Robust MDPs through No-Regret Dynamicscited by 0
2024Noisy Zero-Shot Coordination: Breaking The Common Knowledge Assumption In Zero-Shot Coordination Gamescited by 0
2024Robust Offline Reinforcement Learning for Non-Markovian Decision Processescited by 0
2025Finite-Time Convergence of Single-Trajectory Chi-Square Robust Q-Learning With Linear Function Approximationcited by 0
2025Robust Exploratory Stopping under Ambiguity in Reinforcement Learningcited by 0
2026Distributionally Robust Cooperative Multi-Agent Reinforcement Learning via Robust Value Factorizationcited by 0
2026Robust Transfer Learning with Side Informationcited by 0

50 most cited of 52 All 52 (JSON)