mlsanityData to 7 Oct 2026

Corruption-Robust Offline Reinforcement Learning

arXiv 2106.06630 ↗First posted 11 Jun 2021cs.LGcited by 51

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We study the adversarial robustness in offline reinforcement learning. Given a batch dataset consisting of tuples (s, a, r, s'), an adversary is allowed to arbitrarily modify ε fraction of the tuples. From the corrupted dataset the learner aims to robustly identify a near-optimal policy. We first show that a worst-case Omega(dε) optimality gap is unavoidable in linear MDP of dimension d, even if the adversary only corrupts the reward element in a tuple. This contrasts with dimension-free results in robust supervised learning and best-known lower-bound in the online RL setting with corruption. Next, we propose robust variants of the Least-Square Value Iteration (LSVI) algorithm utilizing robust supervised learning oracles, which achieve near-matching performances in cases both with and without full data coverage. The algorithm requires the knowledge of ε to design the pessimism bonus in the no-coverage case. Surprisingly, in this case, the knowledge of ε is necessary, as we show that being adaptive to unknown ε is impossible.This again contrasts with recent results on corruption-robust online RL and implies that robust offline RL is a strictly harder problem.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 32 of its 51 references that are in the map. First posted on arXiv 11 Jun 2021.

Builds on

32 of 51 references.

Closest ancestors · 13

Same direction, newest first

  1. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  2. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  3. 2021Representation Matters: Offline Pretraining for Sequential Decision Making11 Feb
  4. 2021Near-Optimal Offline Reinforcement Learning via Double Variance Reduction2 Feb
  5. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  6. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
  7. 2020Near-Optimal Provable Uniform Convergence in Offline Policy Evaluation for Reinforcement Learning7 Jul
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020MOPO: Model-based Offline Policy Optimization27 May
  10. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  12. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
1 more
  1. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 19

  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  3. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  4. 2016Robust Estimators in High Dimensions without the Computational Intractabilitycited by 209
  5. 2019Provably Efficient Exploration in Policy Optimizationcited by 203
  6. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  7. 2016Agnostic Estimation of Mean and Covariancecited by 131
  8. 2018Sever: A Robust Meta-Algorithm for Stochastic Optimizationcited by 123
  9. 2019Recent Advances in Algorithmic High-Dimensional Robust Statisticscited by 100
  10. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  11. 2016Learning from Untrusted Datacited by 90
  12. 2019Learning Adversarial MDPs with Bandit Feedback and Unknown Transitioncited by 84
7 more
  1. 2018Efficient Algorithms for Outlier-Robust Regressioncited by 72
  2. 2019Corruption-robust exploration in episodic reinforcement learningcited by 72
  3. 2021Cautiously Optimistic Policy Optimization and Exploration with Linear Function Approximationcited by 53
  4. 2020Robust regression with covariate filtering: Heavy tails and adversarial contaminationcited by 49
  5. 2020Robust Linear Regression: Optimal Rates in Polynomial Timecited by 42
  6. 2021Robust Policy Gradient against Strong Data Corruptioncited by 30
  7. 2021Improved Corruption Robust Algorithms for Episodic Reinforcement Learningcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 11; 2023: 10; 2024: 10; 2025: 13; 2026: 4.

51 papers.

Sort:

By area:

  1. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  2. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  3. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 128
  4. 2022Robust Reinforcement Learning using Offline Datacited by 95
  5. 2021A Model Selection Approach for Corruption Robust Reinforcement Learningcited by 45
  6. 2022Leveraging Offline Data in Online Reinforcement Learningcited by 42
  7. 2022The Efficacy of Pessimism in Asynchronous Q-Learningcited by 41
  8. 2023Corruption-Robust Offline Reinforcement Learning with General Function Approximationcited by 30
  9. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangiancited by 29
  10. 2022COPA: Certifying Robust Policies for Offline Reinforcement Learning against Poisoning Attackscited by 25
  11. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
  12. 2024Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithmscited by 24
38 more
  1. 2023Survival Instinct in Offline Reinforcement Learningcited by 20
  2. 2024Corruption Robust Offline Reinforcement Learning with Human Feedbackcited by 20
  3. 2022Reward Poisoning Attacks on Offline Multi-Agent Reinforcement Learningcited by 19
  4. 2022Online Policy Optimization for Robust MDPcited by 19
  5. 2023Offline Primal-Dual Reinforcement Learning for Linear MDPscited by 17
  6. 2025RobustVLA: Robustness-Aware Reinforcement Post-Training for Vision-Language-Action Modelscited by 16
  7. 2022Byzantine-Robust Online and Offline Distributed Reinforcement Learningcited by 13
  8. 2023Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learningcited by 13
  9. 2023Offline Data Enhanced On-Policy Policy Gradient with Provable Guaranteescited by 11
  10. 2022Robust Imitation Learning from Corrupted Demonstrationscited by 10
  11. 2023On the Opportunities and Challenges of Offline Reinforcement Learning for Recommender Systemscited by 10
  12. 2023Offline Minimax Soft-Q-learning Under Realizability and Partial Coveragecited by 9
  13. 2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modelingcited by 7
  14. 2024Towards Robust Policy: Enhancing Offline Reinforcement Learning with Adversarial Attacks and Defensescited by 5
  15. 2024Pessimism Meets Risk: Risk-Sensitive Offline Reinforcement Learningcited by 5
  16. 2025ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learningcited by 5
  17. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shiftscited by 5
  18. 2024Universal Black-Box Reward Poisoning Attack against Offline Reinforcement Learningcited by 4
  19. 2024Corruption-Robust Offline Two-Player Zero-Sum Markov Gamescited by 4
  20. 2024Data Poisoning Attacks on Off-Policy Policy Evaluation Methodscited by 4
  21. 2025MOORL: A Framework for Integrating Offline-Online Reinforcement Learningcited by 4
  22. 2025Multi-level Certified Defense Against Poisoning Attacks in Offline Reinforcement Learningcited by 3
  23. 2025Hybrid Cross-domain Robust Reinforcement Learningcited by 3
  24. 2025Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruptioncited by 3
  25. 2025Policy Teaching via Data Poisoning in Learning from Human Preferencescited by 2
  26. 2025ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learningcited by 2
  27. 2022Optimal Estimation of Off-Policy Policy Gradient via Double Fitted Iterationcited by 1
  28. 2023Corruption-Robust Lipschitz Contextual Searchcited by 1
  29. 2024Robustness Evaluation of Offline Reinforcement Learning for Robot Control Against Action Perturbationscited by 1
  30. 2025Corruption-Tolerant Asynchronous Q-Learning with Near-Optimal Ratescited by 1
  31. 2026Action Shapley: A Training Data Selection Metric for World Model in Reinforcement Learningcited by 1
  32. 2023Robust Offline Reinforcement Learning -- Certify the Confidence Intervalcited by 0
  33. 2024MoMA: Model-based Mirror Ascent for Offline Reinforcement Learningcited by 0
  34. 2025Protocols for Verifying Smooth Strategies in Bandits and Gamescited by 0
  35. 2025Forecasting in Offline Reinforcement Learning for Non-stationary Environmentscited by 0
  36. 2025Sparse Offline Reinforcement Learning with Corruption Robustnesscited by 0
  37. 2026Learning Sequential Decisions from Multiple Sources via Group-Robust Markov Decision Processescited by 0
  38. 2026Corruption-robust Offline Multi-agent Reinforcement Learning From Human Feedbackcited by 0

50 most cited of 51 All 51 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Robustness of Offline Reinforcement Learning Under Data Corruption via Sharpness-Aware Minimization
  2. 2025CAWR: Corruption-Averse Advantage-Weighted Regression for Robust Policy Optimization
  3. 2024Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptions
  4. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity
  5. 2022Robust Offline Reinforcement Learning with Gradient Penalty and Constraint Relaxation
  6. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing
  7. 2024Robust Offline Reinforcement Learning for Non-Markovian Decision Processes
  8. 2022ARMOR: A Model-based Framework for Improving Arbitrary Baseline Policies with Offline Data