Data to 7 Oct 2026

Robust Reinforcement Learning using Offline Data

arXiv 2208.05129 ↗First posted 10 Aug 2022cs.LG cs.AI stat.MLcited by 95

Not on the map. Why?

Cited by 24 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The goal of robust reinforcement learning (RL) is to learn a policy that is robust against the uncertainty in model parameters. Parameter uncertainty commonly occurs in many real-world RL applications due to simulator modeling errors, changes in the real-world system dynamics over time, and adversarial disturbances. Robust RL is typically formulated as a max-min problem, where the objective is to learn the policy that maximizes the value against the worst possible models that lie in an uncertainty set. In this work, we propose a robust RL algorithm called Robust Fitted Q-Iteration (RFQI), which uses only an offline dataset to learn the optimal robust policy. Robust RL with offline data is significantly more challenging than its non-robust counterpart because of the minimization over all models present in the robust Bellman operator. This poses challenges in offline data collection, optimization over the models, and unbiased estimation. In this work, we propose a systematic approach to overcome these challenges, resulting in our RFQI algorithm. We prove that RFQI learns a near-optimal robust policy under standard assumptions and demonstrate its superior performance on standard benchmark problems.

Key facts. Cited by 95 papers in the map (data through 7 Oct 2026). Builds on 40 of its 96 references that are in the map. First posted on arXiv 10 Aug 2022.

Builds on

40 of 96 references.

References · 40

2014Adam: A Method for Stochastic Optimizationcited by 36,960
2017Proximal Policy Optimization Algorithmscited by 14,441
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
2016OpenAI Gymcited by 1,684
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
2020MOPO: Model-based Offline Policy Optimizationcited by 630

28 more

2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
2017Robust Adversarial Reinforcement Learningcited by 439
2018Learning Models with Uniform Performance via Distributionally Robust Optimizationcited by 334
2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 235
2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 167
2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 135
2019Model-Based Reinforcement Learning with a Generative Model is Minimax Optimalcited by 124
2020Breaking the Sample Size Barrier in Model-Based Reinforcement Learning with a Generative Modelcited by 108
2018The Limits and Potentials of Deep Learning for Roboticscited by 104
2021Online Robust Reinforcement Learning with Model Uncertaintycited by 95
2021Sample Complexity of Robust Reinforcement Learning with a Generative Modelcited by 82
2019Robust Reinforcement Learning for Continuous Control with Model Misspecificationcited by 76
2017Reinforcement Learning under Model Mismatchcited by 52
2021Corruption-Robust Offline Reinforcement Learningcited by 51
2021Exponential Bellman Equation and Improved Regret Bounds for Risk-Sensitive Reinforcement Learningcited by 42
2019Policy Optimization for H2 Linear Control with H_∞ Robustness Guarantee: Implicit Regularization and Global Convergencecited by 39
2018Soft-Robust Actor-Critic Policy-Gradientcited by 37
2019Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPscited by 37
2015Distributionally Robust Counterpart in Markov Decision Processescited by 32
2019A Bayesian Approach to Robust Reinforcement Learningcited by 32
2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learningcited by 32
2014Variance-Constrained Actor-Critic Algorithms for Discounted and Average Reward MDPscited by 28
2021Towards Theoretical Understandings of Robust Markov Decision Processes: Sample Complexity and Asymptoticscited by 27
2022Robust Reinforcement Learning as a Stackelberg Game via Adaptively-Regularized Adversarial Trainingcited by 26
2014Empirical Q-Value Iterationcited by 7
2013Empirical Dynamic Programmingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 28; 2024: 21; 2025: 27; 2026: 14.

95 papers.

Sort:

By area:

2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
2023Double Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coveragecited by 51
2023The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Modelcited by 50
2023Improved Sample Complexity Bounds for Distributionally Robust Reinforcement Learningcited by 44
2022Policy Gradient in Robust MDPs with Global Convergence Guaranteecited by 35
2023Natural Actor-Critic for Robust Reinforcement Learning with Function Approximationcited by 35
2023Near-Minimax-Optimal Risk-Sensitive Reinforcement Learning with CVaRcited by 32
2022Distributionally Robust Offline Reinforcement Learning with Linear Function Approximationcited by 31
2023Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlationcited by 25
2023Robust Multi-Agent Reinforcement Learning via Adversarial Regularization: Theoretical Foundation and Stable Algorithmscited by 25
2024Distributionally Robust Off-Dynamics Reinforcement Learning: Provable Efficiency with Linear Function Approximationcited by 25
2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24

38 more

2024Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithmscited by 24
2025RobustVLA: On Robustness of Vision-Language-Action Model against Multi-Modal Perturbationscited by 23
2023Towards Minimax Optimality of Model-based Robust Reinforcement Learningcited by 21
2023Model-Free Robust Average-Reward Reinforcement Learningcited by 20
2023Survival Instinct in Offline Reinforcement Learningcited by 20
2024Corruption Robust Offline Reinforcement Learning with Human Feedbackcited by 20
2023Distributionally Robust Model-based Reinforcement Learning with Large State Spacescited by 17
2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learningcited by 17
2024Diffusion Model for Planning: A Systematic Literature Reviewcited by 17
2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimizationcited by 17
2023Soft Robust MDPs and Risk-Sensitive MDPs: Equivalence, Policy Gradient, and Sample Complexitycited by 16
2023Single-Trajectory Distributionally Robust Reinforcement Learningcited by 15
2024Distributionally Robust Constrained Reinforcement Learning under Strong Dualitycited by 15
2024Upper and Lower Bounds for Distributionally Robust Off-Dynamics Reinforcement Learningcited by 15
2023Robust Fitted-Q-Evaluation and Iteration under Sequentially Exogenous Unobserved Confounderscited by 12
2023Bridging Distributionally Robust Learning and Offline RL: An Approach to Mitigate Distribution Shift and Partial Data Coveragecited by 12
2023On the Opportunities and Challenges of Offline Reinforcement Learning for Recommender Systemscited by 10
2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RLcited by 10
2024A Comprehensive Survey on Inverse Constrained Reinforcement Learning: Definitions, Progress and Challengescited by 10
2022Personalized Reward Learning with Interaction-Grounded Learning (IGL)cited by 9
2023Adjustable Robust Reinforcement Learning for Online 3D Bin Packingcited by 9
2023MICRO: Model-Based Offline Reinforcement Learning with a Conservative Bellman Operatorcited by 9
2026Group Distributionally Robust Optimization-Driven Reinforcement Learning for LLM Reasoningcited by 9
2023Decision-Making Under Uncertainty: Beyond Probabilitiescited by 8
2024Nonparametric Bellman Mappings for Reinforcement Learning: Application to Robust Adaptive Filteringcited by 8
2024Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptionscited by 8
2024Robust Offline Reinforcement Learning with Linearly Structured f-Divergence Regularizationcited by 8
2022Dynamic selection of p-norm in linear adaptive filtering via online kernel-based reinforcement learningcited by 7
2023Optimal Transport for Measures with Noisy Tree Metriccited by 7
2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modelingcited by 7
2023Scalable Counterfactual Distribution Estimation in Multivariate Causal Modelscited by 6
2025Efficient Policy Optimization in Robust Constrained MDPs with Iteration Complexity Guaranteescited by 6
2025Confounding Robust Deep Reinforcement Learning: A Causal Approachcited by 6
2025ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learningcited by 5
2023Achieving the Asymptotically Optimal Sample Complexity of Offline Reinforcement Learning: A DRO-Based Approachcited by 3
2023Provably Efficient Offline Reinforcement Learning with Perturbed Data Sourcescited by 3
2023Model-based Offline Policy Optimization with Adversarial Networkcited by 3
2024Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processescited by 3

50 most cited of 95 All 95 (JSON)