mlsanityData to 7 Oct 2026

Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling

arXiv 2407.04285 ↗First posted 5 Jul 2024cs.LG cs.AIcited by 7

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Learning policy from offline datasets through offline reinforcement learning (RL) holds promise for scaling data-driven decision-making while avoiding unsafe and costly online interactions. However, real-world data collected from sensors or humans often contains noise and errors, posing a significant challenge for existing offline RL methods, particularly when the real-world data is limited. Our study reveals that prior research focusing on adapting predominant offline RL methods based on temporal difference learning still falls short under data corruption when the dataset is limited. In contrast, we discover that vanilla sequence modeling methods, such as Decision Transformer, exhibit robustness against data corruption, even without specialized modifications. To unlock the full potential of sequence modeling, we propose Robust Decision Rransformer (RDT) by incorporating three simple yet effective robust techniques: embedding dropout to improve the model's robustness against erroneous inputs, Gaussian weighted learning to mitigate the effects of corrupted labels, and iterative data correction to eliminate corrupted data from the source. Extensive experiments on MuJoCo, Kitchen, and Adroit tasks demonstrate RDT's superior performance under various data corruption scenarios compared to prior methods. Furthermore, RDT exhibits remarkable robustness in a more challenging setting that combines training-time data corruption with test-time observation perturbations. These results highlight the potential of sequence modeling for learning from noisy or corrupted offline datasets, thereby promoting the reliable application of offline RL in real-world scenarios. Our code is available at https://github.com/jiawei415/RobustDecisionTransformer.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 55 of its 65 references that are in the map. First posted on arXiv 5 Jul 2024.

Builds on

55 of 65 references.

Closest ancestors · 23

Same direction, newest first

  1. 2024Offline Reinforcement Learning from Datasets with Structured Non-Stationarity23 May
  2. 2024Corruption Robust Offline Reinforcement Learning with Human Feedback9 Feb
  3. 2023Corruption-Robust Offline Reinforcement Learning with General Function Approximation23 Oct
  4. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruption19 Oct
  5. 2023Accountability in Offline Reinforcement Learning: Explaining Decisions with a Corpus of Examples11 Oct
  6. 2023Survival Instinct in Offline Reinforcement Learning5 Jun
  7. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  8. 2023Anti-Exploration by Random Network Distillation31 Jan
  9. 2022Distributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity11 Aug
  10. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  11. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matters27 May
  12. 2022COPA: Certifying Robust Policies for Offline Reinforcement Learning against Poisoning Attacks16 Mar
11 more
  1. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  2. 2021RvS: What is Essential for Offline RL via Supervised Learning?20 Dec
  3. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  4. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  5. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  6. 2021Corruption-Robust Offline Reinforcement Learning11 Jun
  7. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning1 Feb
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020MOPO: Model-based Offline Policy Optimization27 May
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 32

  1. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  5. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  6. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
  7. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  8. 2014Training Deep Neural Networks on Noisy Labels with Bootstrappingcited by 322
  9. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  10. 2020Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observationscited by 235
  11. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  12. 2015A Theoretically Grounded Application of Dropout in Recurrent Neural Networkscited by 161
20 more
  1. 2017Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samplescited by 147
  2. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
  3. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  4. 2021Robust Reinforcement Learning on State Observations with Learned Optimal Adversarycited by 133
  5. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  6. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  7. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  8. 2022Robust Reinforcement Learning using Offline Datacited by 95
  9. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  10. 2023Elastic Decision Transformercited by 66
  11. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
  12. 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
  13. 2024Q-value Regularized Transformer for Offline Reinforcement Learningcited by 28
  14. 2023Future-conditioned Unsupervised Pretraining for Decision Transformercited by 26
  15. 2024Reinformer: Max-Return Sequence Modeling for Offline RLcited by 25
  16. 2023Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learningcited by 24
  17. 2022UniMASK: Unified Inference in Sequential Decision Problemscited by 23
  18. 2023GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Modelscited by 20
  19. 2024Towards Robust Model-Based Reinforcement Learning Against Adversarial Corruptioncited by 12
  20. 2023Decision Transformer under Random Frame Droppingcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 1.

7 papers.

  1. 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agentscited by 213
  2. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shiftscited by 5
  3. 2025ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learningcited by 2
  4. 2025In-Context Curiosity: Distilling Exploration for Decision-Pretrained Transformers on Bandit Taskscited by 0
  5. 2025Robust Adversarial Reinforcement Learning in Stochastic Games via Sequence Modelingcited by 0
  6. 2025Enhancing Robustness of Offline Reinforcement Learning Under Data Corruption via Sharpness-Aware Minimizationcited by 0
  7. 2026Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasetscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Robust Policy Expansion for Offline-to-Online RL under Diverse Data Corruption
  2. 2025Sparse Offline Reinforcement Learning with Corruption Robustness
  3. 2025Expert or not? assessing data quality in offline reinforcement learning
  4. 2024Uncertainty-based Offline Variational Bayesian Reinforcement Learning for Robustness under Diverse Data Corruptions
  5. 2023SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learning
  6. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  7. 2025Collapsing Sequence-Level Data-Policy Coverage via Poisoning Attack in Offline Reinforcement Learning
  8. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets