mlsanityData to 7 Oct 2026

D4RL: Datasets for Deep Data-Driven Reinforcement Learning

arXiv 2004.07219 ↗First posted 15 Apr 2020cs.LG stat.MLcited by 1,406

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

The offline reinforcement learning (RL) setting (also known as full batch RL), where a policy is learned from a static dataset, is compelling as progress enables RL methods to take advantage of large, previously-collected datasets, much like how the rise of large datasets has fueled results in supervised learning. However, existing online RL benchmarks are not tailored towards the offline setting and existing offline RL benchmarks are restricted to data generated by partially-trained agents, making progress in offline RL difficult to measure. In this work, we introduce benchmarks specifically designed for the offline setting, guided by key properties of datasets relevant to real-world applications of offline RL. With a focus on dataset collection, examples of such properties include: datasets generated via hand-designed controllers and human demonstrators, multitask datasets where an agent performs different tasks in the same environment, and datasets collected with mixtures of policies. By moving beyond simple benchmark tasks and data collected by partially-trained RL agents, we reveal important and unappreciated deficiencies of existing algorithms. To facilitate research, we have released our benchmark tasks and datasets with a comprehensive evaluation of existing algorithms, an evaluation protocol, and open-source examples. This serves as a common starting point for the community to identify shortcomings in existing offline RL methods and a collaborative route for progress in this emerging area.

Key facts. Cited by 1,406 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 21 of its 42 references that are in the map. First posted on arXiv 15 Apr 2020.

Builds on

21 of 42 references.

Closest ancestors · 2

Same direction, newest first

  1. 2019Behavior Regularized Offline Reinforcement Learning26 Nov
  2. 2019An Optimistic Perspective on Offline Reinforcement Learning10 Jul

Other references · 19

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  3. 2016OpenAI Gymcited by 1,684
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  7. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  8. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  9. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  10. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  11. 2019Learning Latent Plans from Playcited by 310
  12. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
7 more
  1. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  2. 2019On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shiftcited by 174
  3. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  4. 2019Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?cited by 140
  5. 2019Scaling data-driven robotics with reward sketching and batch reinforcement learningcited by 18
  6. 2018The AdobeIndoorNav Dataset: Towards Deep Reinforcement Learning based Real-world Indoor Robot Visual Navigationcited by 6
  7. 2017Batch Reinforcement Learning on the Industrial Benchmark: First Experiencescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 15; 2021: 107; 2022: 178; 2023: 294; 2024: 299; 2025: 283; 2026: 230.

1,406 papers.

Sort:

By area:

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  4. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  5. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  6. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  7. 2022A Generalist Agentcited by 802
  8. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  9. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  10. 2024Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusioncited by 634
  11. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  12. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
38 more
  1. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  2. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  3. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  4. 2021Implicit Behavioral Cloningcited by 442
  5. 2022Dataset Distillation by Matching Training Trajectoriescited by 437
  6. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  7. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  8. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  9. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 357
  10. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  11. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  12. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  13. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  14. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  15. 2024Diffusion Policy Policy Optimizationcited by 255
  16. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  17. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  18. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  19. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  20. 2023Affordances from Human Videos as a Versatile Representation for Roboticscited by 221
  21. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  22. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  23. 2022Online Decision Transformercited by 194
  24. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  25. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  26. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  27. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
  28. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  29. 2025Flow Q-Learningcited by 162
  30. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  31. 2021Offline RL Without Off-Policy Evaluationcited by 150
  32. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 149
  33. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Plannerscited by 147
  34. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  35. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learningcited by 145
  36. 2025Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RLcited by 144
  37. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  38. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137

50 most cited of 1,406 All 1,406 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020Critic Regularized Regression
  2. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
  3. 2022Leveraging Offline Data in Online Reinforcement Learning
  4. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  5. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  6. 2025Scaling Offline RL via Efficient and Expressive Shortcut Models
  7. 2022Data Valuation for Offline Reinforcement Learning
  8. 2021Measuring Data Quality for Dataset Selection in Offline Reinforcement Learning