mlsanityData to 7 Oct 2026

Simple Ingredients for Offline Reinforcement Learning

arXiv 2403.13097 ↗First posted 19 Mar 2024cs.LG cs.AIcited by 4

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning algorithms have proven effective on datasets highly connected to the target downstream task. Yet, leveraging a novel testbed (MOOD) in which trajectories come from heterogeneous sources, we show that existing methods struggle with diverse data: their performance considerably deteriorates as data collected for related but different tasks is simply added to the offline buffer. In light of this finding, we conduct a large empirical study where we formulate and test several hypotheses to explain this failure. Surprisingly, we find that scale, more than algorithmic considerations, is the key factor influencing performance. We show that simple methods like AWAC and IQL with increased network size overcome the paradoxical failure modes from the inclusion of additional data in MOOD, and notably outperform prior state-of-the-art algorithms on the canonical D4RL benchmark.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 44 of its 57 references that are in the map. First posted on arXiv 19 Mar 2024.

Builds on

44 of 57 references.

Closest ancestors · 13

Same direction, newest first

  1. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets6 Oct
  2. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting22 Jun
  3. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learning16 May
  4. 2023Extreme Q-Learning: MaxEnt RL without Entropy5 Jan
  5. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning31 Jan
  6. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  7. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  8. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  9. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  10. 2020Critic Regularized Regression26 Jun
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
1 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2016Density estimation using Real NVPcited by 2,029
  5. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  9. 2015Prioritized Experience Replaycited by 1,070
  10. 2018Exploration by Random Network Distillationcited by 966
  11. 2018DeepMind Control Suitecited by 935
  12. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
19 more
  1. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  2. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  3. 2016Improving Variational Inference with Inverse Autoregressive Flowcited by 636
  4. 2015Generating Sentences from a Continuous Spacecited by 607
  5. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  6. 2015MADE: Masked Autoencoder for Distribution Estimationcited by 306
  7. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
  8. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  9. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 157
  10. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  11. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  12. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
  13. 2021AW-Opt: Learning Robotic Skills with Imitation and Reinforcement at Scalecited by 43
  14. 2021Training Larger Networks for Deep Reinforcement Learningcited by 37
  15. 2022MoCapAct: A Multi-Task Dataset for Simulated Humanoid Controlcited by 24
  16. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matchingcited by 24
  17. 2021Towards Deeper Deep Reinforcement Learning with Spectral Normalizationcited by 19
  18. 2021Learning Pessimism for Robust and Efficient Off-Policy Reinforcement Learningcited by 18
  19. 2022Learning on the Job: Self-Rewarding Offline-to-Online Finetuning for Industrial Insertion of Novel Connectors from Visioncited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 51
  2. 2024Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learningcited by 2
  3. 2025Alert: Learning Trigger Functions for Early Classification of Time Series using Deep-RLcited by 1
  4. 2026JumpStart Your Policy Learning with Lessons from 160,000 Training Runscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Offline Reinforcement Learning Hands-On
  2. 2023Improving and Benchmarking Offline Reinforcement Learning Algorithms
  3. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  4. 2025A Clean Slate for Offline Reinforcement Learning
  5. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  6. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
  7. 2021A Dataset Perspective on Offline Reinforcement Learning
  8. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning