mlsanityData to 7 Oct 2026

Towards Robust Model-Based Reinforcement Learning Against Adversarial Corruption

arXiv 2402.08991 ↗First posted 14 Feb 2024stat.ML cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

This study tackles the challenges of adversarial corruption in model-based reinforcement learning (RL), where the transition dynamics can be corrupted by an adversary. Existing studies on corruption-robust RL mostly focus on the setting of model-free RL, where robust least-square regression is often employed for value function estimation. However, these techniques cannot be directly applied to model-based RL. In this paper, we focus on model-based RL and take the maximum likelihood estimation (MLE) approach to learn transition model. Our work encompasses both online and offline settings. In the online setting, we introduce an algorithm called corruption-robust optimistic MLE (CR-OMLE), which leverages total-variation (TV)-based information ratios as uncertainty weights for MLE. We prove that CR-OMLE achieves a regret of tildeO(√T + C), where C denotes the cumulative corruption level after T episodes. We also prove a lower bound to show that the additive dependence on C is optimal. We extend our weighting technique to the offline setting, and propose an algorithm named corruption-robust pessimistic MLE (CR-PMLE). Under a uniform coverage condition, CR-PMLE exhibits suboptimality worsened by O(C/n), nearly matching the lower bound. To the best of our knowledge, this is the first work on corruption-robust model-based RL algorithms with provable guarantees.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 43 of its 63 references that are in the map. First posted on arXiv 14 Feb 2024.

Builds on

43 of 63 references.

References · 43

  1. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  2. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  3. 2016Contextual Decision Processes with Low Bellman Rank are PAC-Learnablecited by 290
  4. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  5. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
  6. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  7. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 222
  8. 2021The Statistical Complexity of Interactive Decision Makingcited by 208
  9. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  10. 2019Reinforcement Learning in Feature Space: Matrix Bandit, Kernels, and Regret Boundcited by 199
  11. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 176
  12. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 128
31 more
  1. 2020Minimax-Optimal Off-Policy Evaluation with Linear Function Approximationcited by 120
  2. 2018Stochastic bandits robust to adversarial corruptionscited by 118
  3. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
  4. 2021Online Robust Reinforcement Learning with Model Uncertaintycited by 95
  5. 2019Better Algorithms for Stochastic Bandits with Adversarial Corruptionscited by 91
  6. 2020Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planningcited by 83
  7. 2020Stochastic Linear Bandits Robust to Adversarial Attackscited by 65
  8. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Gamecited by 53
  9. 2017Reinforcement Learning under Model Mismatchcited by 52
  10. 2022Nearly Optimal Algorithms for Linear Contextual Bandits with Adversarial Corruptionscited by 51
  11. 2023The Curious Price of Distributional Robustness in Reinforcement Learning with a Generative Modelcited by 50
  12. 2021Achieving Near Instance-Optimality and Minimax-Optimality in Stochastic and Adversarial Linear Bandits Simultaneouslycited by 48
  13. 2021A Model Selection Approach for Corruption Robust Reinforcement Learningcited by 45
  14. 2022GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyondcited by 45
  15. 2022VOQL: Towards Optimal Regret in Model-free RL with Nonlinear Function Approximationcited by 44
  16. 2022Pessimistic Minimax Value Iteration: Provably Efficient Equilibrium Learning from Offline Datasetscited by 42
  17. 2021Robust Stochastic Linear Contextual Bandits Under Adversarial Attackscited by 33
  18. 2022Corruption-Robust Algorithms with Uncertainty Weighting for Nonlinear Contextual Bandits and Markov Decision Processescited by 33
  19. 2022Model-based RL with Optimistic Posterior Sampling: Structural Conditions and Sample Complexitycited by 31
  20. 2022A General Framework for Sample-Efficient Function Approximation in Reinforcement Learningcited by 31
  21. 2023Corruption-Robust Offline Reinforcement Learning with General Function Approximationcited by 30
  22. 2023The Benefits of Being Distributional: Small-Loss Bounds for Reinforcement Learningcited by 27
  23. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
  24. 2021Linear Contextual Bandits with Adversarial Corruptionscited by 21
  25. 2023Towards Minimax Optimality of Model-based Robust Reinforcement Learningcited by 21
  26. 2013Online Learning in Markov Decision Processes with Adversarially Chosen Transition Probability Distributionscited by 18
  27. 2021Unifying Cardiovascular Modelling with Deep Reinforcement Learning for Uncertainty Aware Control of Sepsis Treatmentcited by 16
  28. 2023Robust Lipschitz Bandits to Adversarial Corruptionscited by 15
  29. 2023A Nearly Optimal and Low-Switching Algorithm for Reinforcement Learning with General Function Approximationcited by 14
  30. 2020Adversarial Attacks to Machine Learning-Based Smart Healthcare Systemscited by 12
  31. 2023Pessimistic Nonlinear Least-Squares Value Iteration for Offline Reinforcement Learningcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 7; 2026: 2.

12 papers.

  1. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 32
  2. 2024Distributionally Robust Reinforcement Learning with Interactive Data Collection: Fundamental Hardness and Near-Optimal Algorithmscited by 24
  3. 2025RobustVLA: Robustness-Aware Reinforcement Post-Training for Vision-Language-Action Modelscited by 16
  4. 2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modelingcited by 7
  5. 2025Wolfpack Adversarial Attack for Robust Multi-Agent Reinforcement Learningcited by 6
  6. 2026WMAttack: Automated Attack Search for Adversarial Evaluation of World-Model Agentscited by 6
  7. 2025ORVIT: Near-Optimal Online Distributionally Robust Reinforcement Learningcited by 5
  8. 2025Catoni Contextual Bandits are Robust to Heavy-tailed Rewardscited by 3
  9. 2025Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruptioncited by 3
  10. 2025ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learningcited by 2
  11. 2026ARB4WM: An Adversarial Robustness Benchmark for World Models in Continuous Controlcited by 2
  12. 2025Daunce: Data Attribution through Uncertainty Estimationcited by 1