mlsanityData to 14 Oct 2026

RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning

arXiv 2006.13888 ↗First posted 24 Jun 2020cs.LG stat.MLcited by 57

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline methods for reinforcement learning have a potential to help bridge the gap between reinforcement learning research and real-world applications. They make it possible to learn policies from offline datasets, thus overcoming concerns associated with online data collection in the real-world, including cost, safety, or ethical concerns. In this paper, we propose a benchmark called RL Unplugged to evaluate and compare offline RL methods. RL Unplugged includes data from a diverse range of domains including games (e.g., Atari benchmark) and simulated motor control problems (e.g., DM Control Suite). The datasets include domains that are partially or fully observable, use continuous or discrete actions, and have stochastic vs. deterministic dynamics. We propose detailed evaluation protocols for each domain in RL Unplugged and provide an extensive analysis of supervised learning and offline RL methods using these protocols. We will release data for all our tasks and open-source all algorithms presented in this paper. We hope that our suite of benchmarks will increase the reproducibility of experiments and make it possible to study challenging tasks with a limited computational budget, thus making RL research both more systematic and more accessible across the community. Moving forward, we view RL Unplugged as a living benchmark suite that will evolve and grow with datasets contributed by the research community and ourselves. Our project page is available on https://git.io/JJUhd.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 53 references that are in the map. First posted on arXiv 24 Jun 2020.

Builds on

36 of 53 references.

Closest ancestors · 4

Same direction, newest first

  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  2. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  3. 2019Behavior Regularized Offline Reinforcement Learning26 Nov
  4. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 32

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  6. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  7. 2018DeepMind Control Suitecited by 956
  8. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  10. 2017Deep Reinforcement Learning that Matterscited by 754
  11. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  12. 2019Solving Rubik's Cube with a Robot Handcited by 734
20 more
  1. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  2. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  3. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 353
  4. 2018Maximum a Posteriori Policy Optimisationcited by 328
  5. 2019Challenges of Real-World Reinforcement Learningcited by 291
  6. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  7. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  8. 2016COCO-Text: Dataset and Benchmark for Text Detection and Recognition in Natural Imagescited by 236
  9. 2018Distributed Distributional Deterministic Policy Gradientscited by 199
  10. 2017Emergence of Locomotion Behaviours in Rich Environmentscited by 165
  11. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  12. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 144
  13. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 122
  14. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  15. 2018Neural probabilistic motor primitives for humanoid controlcited by 97
  16. 2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
  17. 2020An empirical investigation of the challenges of real-world reinforcement learningcited by 77
  18. 2019Making Efficient Use of Demonstrations to Solve Hard Exploration Problemscited by 45
  19. 2018Hierarchical visuomotor control of humanoidscited by 30
  20. 2019Deep neuroethology of a virtual rodentcited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 5; 2021: 30; 2022: 11; 2023: 5; 2024: 2; 2025: 2; 2026: 2.

57 papers.

Sort:

By area:

  1. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  2. 2021Implicit Behavioral Cloningcited by 452
  3. 2022Dataset Distillation by Matching Training Trajectoriescited by 442
  4. 2022Behavior Transformers: Cloning k modes with one stonecited by 304
  5. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 154
  6. 2021Offline RL Without Off-Policy Evaluationcited by 151
  7. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 114
  8. 2022Robot Learning on the Job: Human-in-the-Loop Autonomy and Learning During Deploymentcited by 108
  9. 2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 95
  10. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  11. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  12. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
38 more
  1. 2021d3rlpy: An Offline Deep Reinforcement Learning Librarycited by 84
  2. 2021Towards Instance-Optimal Offline Reinforcement Learning with Pessimismcited by 78
  3. 2021Demonstration-Guided Reinforcement Learning with Learned Skillscited by 76
  4. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 73
  5. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  6. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  7. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  8. 2022Challenges and Opportunities in Offline Reinforcement Learning from Visual Observationscited by 63
  9. 2021Plan Better Amid Conservatism: Offline Multi-Agent Reinforcement Learning with Actor Rectificationcited by 61
  10. 2021Vector Quantized Models for Planningcited by 48
  11. 2021TRAIL: Near-Optimal Imitation Learning with Suboptimal Datacited by 39
  12. 2023Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversationscited by 38
  13. 2022Skill-based Meta-Reinforcement Learningcited by 37
  14. 2021Provable Representation Learning for Imitation with Contrastive Fourier Featurescited by 34
  15. 2021Autoregressive Dynamics Models for Offline Policy Evaluation and Optimizationcited by 33
  16. 2021Regularized Behavior Value Estimationcited by 32
  17. 2021RLDS: an Ecosystem to Generate, Share and Use Datasets in Reinforcement Learningcited by 31
  18. 2021A Dataset Perspective on Offline Reinforcement Learningcited by 27
  19. 2022The Challenges of Exploration for Offline Reinforcement Learningcited by 26
  20. 2022An Empirical Study of Implicit Regularization in Deep Offline RLcited by 24
  21. 2021Value Iteration in Continuous Actions, States and Timecited by 18
  22. 2021TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrationscited by 18
  23. 2021Hyperparameter Selection for Imitation Learningcited by 17
  24. 2021Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functionscited by 17
  25. 2022Real World Offline Reinforcement Learning with Realistic Data Sourcecited by 17
  26. 2020DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPscited by 16
  27. 2022Federated Offline Reinforcement Learningcited by 13
  28. 2025Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learningcited by 12
  29. 2023NetHack is Hard to Hackcited by 11
  30. 2023AlphaStar Unplugged: Large-Scale Offline Reinforcement Learningcited by 9
  31. 2020Offline Reinforcement Learning Hands-Oncited by 6
  32. 2021Continuous-Time Fitted Value Iteration for Robust Policiescited by 6
  33. 2022DraftRec: Personalized Draft Recommendation for Winning in Multi-Player Online Battle Arena Gamescited by 6
  34. 2023Robotic Manipulation Datasets for Offline Compositional Reinforcement Learningcited by 6
  35. 2021RL4RS: A Real-World Dataset for Reinforcement Learning based Recommender Systemcited by 5
  36. 2021Discovering an Aid Policy to Minimize Student Evasion Using Offline Reinforcement Learningcited by 3
  37. 2021Measuring Data Quality for Dataset Selection in Offline Reinforcement Learningcited by 3
  38. 2021Pessimistic Model Selection for Offline Deep Reinforcement Learningcited by 3

50 most cited of 57 All 57 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Improving and Benchmarking Offline Reinforcement Learning Algorithms
  2. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  3. 2025A Clean Slate for Offline Reinforcement Learning
  4. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learning
  5. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  6. 2021A Minimalist Approach to Offline Reinforcement Learning
  7. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  8. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies