mlsanityData to 7 Oct 2026

Corruption-Robust Offline Reinforcement Learning with General Function Approximation

arXiv 2310.14550 ↗First posted 23 Oct 2023cs.LGcited by 30

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We investigate the problem of corruption robustness in offline reinforcement learning (RL) with general function approximation, where an adversary can corrupt each sample in the offline dataset, and the corruption level zeta≥0 quantifies the cumulative corruption amount over n episodes and H steps. Our goal is to find a policy that is robust to such corruption and minimizes the suboptimality gap with respect to the optimal policy for the uncorrupted Markov decision processes (MDPs). Drawing inspiration from the uncertainty-weighting technique from the robust online RL setting citephe2022nearly,ye2022corruptionrobust, we design a new uncertainty weight iteration procedure to efficiently compute on batched samples and propose a corruption-robust algorithm for offline RL. Notably, under the assumption of single policy coverage and the knowledge of zeta, our proposed algorithm achieves a suboptimality bound that is worsened by an additive factor of O(zeta (C(widehatF,μ)n)-1) due to the corruption. Here widehatF is the confidence set, and the dataset ZnH, and C(widehatF,μ) is a coefficient that depends on widehatF and the underlying data distribution μ. When specialized to linear MDPs, the corruption-dependent error term reduces to O(zeta d n-1) with d being the dimension of the feature map, which matches the existing lower bound for corrupted linear MDPs. This suggests that our analysis is tight in terms of the corruption-dependent term.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 51 of its 66 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

51 of 66 references.

Closest ancestors · 16

Same direction, newest first

  1. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  2. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  3. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Game31 May
  4. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters27 May
  5. 2022COPA: Certifying Robust Policies for Offline Reinforcement Learning against Poisoning Attacks16 Mar
  6. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimism17 Oct
  7. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  8. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning19 Aug
  9. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coverage13 Jul
  10. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  11. 2021Corruption-Robust Offline Reinforcement Learning11 Jun
  12. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
4 more
  1. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  2. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?22 Oct
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  4. 2019Policy Poisoning in Batch Reinforcement Learning and Control13 Oct

Other references · 35

  1. 2016Deep Exploration via Bootstrapped DQNcited by 518
  2. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 514
  3. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 290
  4. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 235
  5. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 222
  6. 2021The Statistical Complexity of Interactive Decision Makingcited by 208
  7. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  8. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 176
  9. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  10. 2018Stochastic bandits robust to adversarial corruptionscited by 118
  11. 2018Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendationcited by 101
  12. 2019Better Algorithms for Stochastic Bandits with Adversarial Corruptionscited by 91
23 more
  1. 2020Adaptive Reward-Poisoning Attacks against Reinforcement Learningcited by 86
  2. 2019Learning Adversarial MDPs with Bandit Feedback and Unknown Transitioncited by 84
  3. 2021Adapting to Misspecification in Contextual Banditscited by 81
  4. 2019Online Convex Optimization in Adversarial Markov Decision Processescited by 79
  5. 2020Stochastic Linear Bandits Robust to Adversarial Attackscited by 65
  6. 2023Uncertainty Estimation by Fisher Information-based Evidential Deep Learningcited by 58
  7. 2022Computationally Efficient Horizon-Free Reinforcement Learning for Linear Mixture MDPscited by 52
  8. 2021Policy Optimization in Adversarial MDPs: Improved Exploration via Dilated Bonusescited by 51
  9. 2022Nearly Optimal Algorithms for Linear Contextual Bandits with Adversarial Corruptionscited by 51
  10. 2020Simultaneously Learning Stochastic and Adversarial Episodic MDPs with Known Transitioncited by 48
  11. 2021Achieving Near Instance-Optimality and Minimax-Optimality in Stochastic and Adversarial Linear Bandits Simultaneouslycited by 48
  12. 2021A Model Selection Approach for Corruption Robust Reinforcement Learningcited by 45
  13. 2022Pessimistic Minimax Value Iteration: Provably Efficient Equilibrium Learning from Offline Datasetscited by 42
  14. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
  15. 2021Robust Stochastic Linear Contextual Bandits Under Adversarial Attackscited by 33
  16. 2022Corruption-Robust Algorithms with Uncertainty Weighting for Nonlinear Contextual Bandits and Markov Decision Processescited by 33
  17. 2022A General Framework for Sample-Efficient Function Approximation in Reinforcement Learningcited by 31
  18. 2020Stochastic Shortest Path with Adversarially Changing Costscited by 28
  19. 2021Finding the Stochastic Shortest Path with Low Regret: The Adversarial Cost and Unknown Transition Casecited by 28
  20. 2018Deep Reinforcement Learning for Dynamic Treatment Regimes on Medical Registry Datacited by 23
  21. 2021Linear Contextual Bandits with Adversarial Corruptionscited by 21
  22. 2023Robust Lipschitz Bandits to Adversarial Corruptionscited by 15
  23. 2018Mitigation of Policy Manipulation Attacks on Deep Q-Networks with Parameter-Space Noisecited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 13; 2026: 3.

30 papers.

Sort:

By area:

  1. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  2. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  3. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 32
  4. 2024No Regrets: Investigating and Improving Regret Approximations for Curriculum Discoverycited by 29
  5. 2024Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithmscited by 24
  6. 2024Corruption Robust Offline Reinforcement Learning with Human Feedbackcited by 20
  7. 2024Refining Minimax Regret for Unsupervised Environment Designcited by 16
  8. 2024Towards Robust Model-Based Reinforcement Learning Against Adversarial Corruptioncited by 12
  9. 2024Model-based RL as a Minimalist Approach to Horizon-Free and Second-Order Boundscited by 11
  10. 2024Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptionscited by 8
  11. 2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modelingcited by 7
  12. 2025On Corruption-Robustness in Performative Reinforcement Learningcited by 6
18 more
  1. 2024Uncertainty-Aware Reward-Free Exploration with General Function Approximationcited by 5
  2. 2025ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learningcited by 5
  3. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shiftscited by 5
  4. 2024Universal Black-Box Reward Poisoning Attack against Offline Reinforcement Learningcited by 4
  5. 2025Catoni Contextual Bandits are Robust to Heavy-tailed Rewardscited by 3
  6. 2025Multi-level Certified Defense Against Poisoning Attacks in Offline Reinforcement Learningcited by 3
  7. 2025Hybrid Cross-domain Robust Reinforcement Learningcited by 3
  8. 2025ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learningcited by 2
  9. 2025Near-Optimal Second-Order Guarantees for Model-Based Adversarial Imitation Learningcited by 2
  10. 2024Robustness Evaluation of Offline Reinforcement Learning for Robot Control Against Action Perturbationscited by 1
  11. 2025Daunce: Data Attribution through Uncertainty Estimationcited by 1
  12. 2025Corruption-Tolerant Asynchronous Q-Learning with Near-Optimal Ratescited by 1
  13. 2025Robust In-Context Reinforcement Learning Under Reward Poisoning Attackscited by 0
  14. 2025Enhancing Robustness of Offline Reinforcement Learning Under Data Corruption via Sharpness-Aware Minimizationcited by 0
  15. 2025Forecasting in Offline Reinforcement Learning for Non-stationary Environmentscited by 0
  16. 2026Corruption-robust Offline Multi-agent Reinforcement Learning From Human Feedbackcited by 0
  17. 2026Online Security Learning in Cooperative Multi-Agent Systems under Hidden Byzantine Attackscited by 0
  18. 2026RMB: Reward Model Boosting Mitigates Reward Hackingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Sparse Offline Reinforcement Learning with Corruption Robustness
  2. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruption
  3. 2024Minimax Optimal and Computationally Efficient Algorithms for Distributionally Robust Offline Reinforcement Learning
  4. 2024Robust Offline Reinforcement Learning for Non-Markovian Decision Processes
  5. 2023Achieving the Asymptotically Optimal Sample Complexity of Offline Reinforcement Learning: A DRO-Based Approach
  6. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity
  7. 2024Sample Complexity of Offline Distributionally Robust Linear Markov Decision Processes
  8. 2025Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruption