mlsanityData to 14 Oct 2026

The Generalization Gap in Offline Reinforcement Learning

arXiv 2312.05742 ↗First posted 10 Dec 2023cs.LG cs.AIcited by 26

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Despite recent progress in offline learning, these methods are still trained and tested on the same environment. In this paper, we compare the generalization abilities of widely used online and offline learning methods such as online reinforcement learning (RL), offline RL, sequence modeling, and behavioral cloning. Our experiments show that offline learning algorithms perform worse on new environments than online learning ones. We also introduce the first benchmark for evaluating generalization in offline learning, collecting datasets of varying sizes and skill-levels from Procgen (2D video games) and WebShop (e-commerce websites). The datasets contain trajectories for a limited number of game levels or natural language instructions and at test time, the agent has to generalize to new levels or instructions. Our experiments reveal that existing offline learning algorithms struggle to match the performance of online RL on both train and test environments. Behavioral cloning is a strong baseline, outperforming state-of-the-art offline RL and sequence modeling approaches when trained on data from multiple environments and tested on new ones. Finally, we find that increasing the diversity of the data, rather than its size, improves performance on new environments for all offline learning algorithms. Our study demonstrates the limited generalization of current offline learning algorithms highlighting the need for more research in this area.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 107 of its 130 references that are in the map. First posted on arXiv 10 Dec 2023.

Builds on

107 of 130 references.

Closest ancestors · 22

Same direction, newest first

  1. 2023Extreme Q-Learning: MaxEnt RL without Entropy5 Jan
  2. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes28 Nov
  3. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observations9 Jun
  4. 2022When does return-conditioned supervised learning work for offline reinforcement learning?2 Jun
  5. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  6. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  7. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learning31 Jan
  8. 2022The Challenges of Exploration for Offline Reinforcement Learning27 Jan
  9. 2021Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functions29 Nov
  10. 2021RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learning4 Nov
  11. 2021The Difficulty of Passive Learning in Deep Reinforcement Learning26 Oct
  12. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
10 more
  1. 2021BRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learning2 Oct
  2. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning19 Aug
  3. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  4. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism22 Mar
  5. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning1 Feb
  6. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  9. 2019Behavior Regularized Offline Reinforcement Learning26 Nov
  10. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 85

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  7. 2015Trust Region Policy Optimizationcited by 2,580
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  9. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  10. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  11. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  12. 2022A Generalist Agentcited by 808
38 more
  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  2. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  3. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  4. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  5. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  6. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  7. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 478
  8. 2020Reinforcement Learning with Augmented Datacited by 468
  9. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  10. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  11. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  12. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  13. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 274
  14. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 270
  15. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  16. 2020Decoupling Representation Learning from Reinforcement Learningcited by 269
  17. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  18. 2015Contextual Markov Decision Processescited by 220
  19. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 215
  20. 2022Multi-Game Decision Transformerscited by 214
  21. 2023Landmark Attention: Random-Access Infinite Context Length for Transformerscited by 213
  22. 2020The NetHack Learning Environmentcited by 198
  23. 2020Prioritized Level Replaycited by 198
  24. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 182
  25. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 174
  26. 2022Evolving Curricula with Regret-Based Environment Designcited by 173
  27. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 170
  28. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 137
  29. 2018Assessing Generalization in Deep Reinforcement Learningcited by 133
  30. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  31. 2018Generalization and Regularization in DQNcited by 118
  32. 2021Learning One Representation to Optimize All Rewardscited by 115
  33. 2021Replay-Guided Adversarial Environment Designcited by 111
  34. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 109
  35. 2020Invariant Causal Prediction for Block MDPscited by 105
  36. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 105
  37. 2019Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learningcited by 104
  38. 2019Generalization in Reinforcement Learning with Selective Noise Injection and Information Bottleneckcited by 101

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 10; 2026: 8.

26 papers.

Sort:

By area:

  1. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 89
  2. 2024Benchmarking Mobile Device Control Agents across Diverse Configurationscited by 55
  3. 2025Pre-Trained Video Generative Models as World Simulatorscited by 44
  4. 2025Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learningcited by 13
  5. 2025A Clean Slate for Offline Reinforcement Learningcited by 11
  6. 2024D2PO: Discriminator-Guided DPO with Response Evaluation Modelscited by 9
  7. 2024Provable Domain Adaptation for Offline Reinforcement Learning with Limited Samplescited by 5
  8. 2025Provable Zero-Shot Generalization in Offline Reinforcement Learningcited by 3
  9. 2024Can virtual staining for high-throughput screening generalize?cited by 2
  10. 2024Permutation Invariant Learning with High-Dimensional Particle Filterscited by 1
  11. 2024The Synergy of LLMs & RL Unlocks Offline Learning of Generalizable Language-Conditioned Policies with Low-fidelity Datacited by 1
  12. 2024Robustness Evaluation of Offline Reinforcement Learning for Robot Control Against Action Perturbationscited by 1
14 more
  1. 2025How Ensembles of Distilled Policies Improve Generalisation in Reinforcement Learningcited by 1
  2. 2025Offline vs. Online Learning in Model-based RL: Lessons for Data Collection Strategiescited by 1
  3. 2025AbideGym: Turning Static RL Worlds into Adaptive Challengescited by 1
  4. 2025Blindfolded Experts Generalize Better: Insights from Robotic Manipulation and Videogamescited by 1
  5. 2026VLGOR: Visual-Language Knowledge Guided Offline Reinforcement Learning for Generalizable Agentscited by 1
  6. 2026Offline Policy Optimization with Posterior Samplingcited by 1
  7. 2025Synthetic Data is Sufficient for Zero-Shot Visual Generalization from Offline Datacited by 0
  8. 2025LLM-Driven Policy Diffusion: Enhancing Generalization in Offline Reinforcement Learningcited by 0
  9. 2026120 Minutes and a Laptop: Minimalist Image-goal Navigation via Unsupervised Exploration and Offline RLcited by 0
  10. 2026Generalization in offline RL: The structure is more important than the amount of pessimismcited by 0
  11. 2026Improving greenhouse fruit-production control by integrating reinforcement learning into short-horizon model predictive controlcited by 0
  12. 2026Out-of-Distribution Generalisation with Sequence Models in Offline Multi-Agent Reinforcement Learningcited by 0
  13. 2026Improving Generalization and Robustness in Offline Reinforcement Learning via Boundary-Aware Data Augmentationcited by 0
  14. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Offline Reinforcement Learning Hands-On
  2. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  3. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  4. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?
  5. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  6. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  7. 2022Domain Generalization for Robust Model-Based Offline Reinforcement Learning
  8. 2021Representation Matters: Offline Pretraining for Sequential Decision Making