mlsanityData to 7 Oct 2026

Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems

arXiv 2005.01643 ↗First posted 4 May 2020cs.LG cs.AI stat.MLcited by 1,873

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In this tutorial article, we aim to provide the reader with the conceptual tools needed to get started on research on offline reinforcement learning algorithms: reinforcement learning algorithms that utilize previously collected data, without additional online data collection. Offline reinforcement learning algorithms hold tremendous promise for making it possible to turn large datasets into powerful decision making engines. Effective offline reinforcement learning methods would be able to extract policies with the maximum possible utility out of the available data, thereby allowing automation of a wide range of decision-making domains, from healthcare and education to robotics. However, the limitations of current algorithms make this difficult. We will aim to provide the reader with an understanding of these challenges, particularly in the context of modern deep reinforcement learning methods, and describe some potential solutions that have been explored in recent work to mitigate these challenges, along with recent applications, and a discussion of perspectives on open problems in the field.

Key facts. Cited by 1,873 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 119 of its 193 references that are in the map. First posted on arXiv 4 May 2020.

Builds on

119 of 193 references.

Closest ancestors · 4

Same direction, newest first

  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  2. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  3. 2019Behavior Regularized Offline Reinforcement Learning26 Nov
  4. 2019An Optimistic Perspective on Offline Reinforcement Learning10 Jul

Other references · 115

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  5. 2015Continuous control with deep reinforcement learningcited by 3,279
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  7. 2015Trust Region Policy Optimizationcited by 2,503
  8. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  9. 2019Invariant Risk Minimizationcited by 1,832
  10. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  11. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  12. 2016End to End Learning for Self-Driving Carscited by 1,000
38 more
  1. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  2. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  3. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  4. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  6. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  7. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  8. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  9. 2016Deep Exploration via Bootstrapped DQNcited by 518
  10. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
  11. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 503
  12. 2019Model-Based Reinforcement Learning for Ataricited by 495
  13. 2014Semi-Supervised Learning with Deep Generative Modelscited by 494
  14. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  15. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  16. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 373
  17. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  18. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  19. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  20. 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 333
  21. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  22. 2017Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuningcited by 288
  23. 2016Data-Efficient Off-Policy Policy Evaluation for Reinforcement Learningcited by 285
  24. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  25. 2018Sim-to-Real: Learning Agile Locomotion For Quadruped Robotscited by 277
  26. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  27. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  28. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  29. 2018Learning to Drive in a Daycited by 266
  30. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  31. 2019Causality for Machine Learningcited by 242
  32. 2016CAD2RL: Real Single-Image Flight without a Single Real Imagecited by 227
  33. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 226
  34. 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 224
  35. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  36. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 208
  37. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 203
  38. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187

50 most cited of 115 All 115 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 17; 2021: 205; 2022: 267; 2023: 350; 2024: 367; 2025: 347; 2026: 320.

1,873 papers.

Sort:

By area:

  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  3. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  4. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  5. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  6. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  7. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  8. 2023Augmented Language Models: a Surveycited by 398
  9. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  10. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  11. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  12. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
38 more
  1. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  2. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  3. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  4. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 307
  5. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  6. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  7. 2025π*0.6: a VLA That Learns From Experiencecited by 280
  8. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 275
  9. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 267
  10. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  11. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  12. 2023Pretraining Language Models with Human Preferencescited by 245
  13. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  14. 2024ReFT: Reasoning with Reinforced Fine-Tuningcited by 236
  15. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  16. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  17. 2022Multi-Game Decision Transformerscited by 212
  18. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 211
  19. 2021Towards Causal Representation Learningcited by 207
  20. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  21. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  22. 2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 172
  23. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
  24. 2021A Survey of Zero-shot Generalisation in Deep Reinforcement Learningcited by 164
  25. 2025Flow Q-Learningcited by 162
  26. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  27. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  28. 2023Learning to Act from Actionless Videos through Dense Correspondencescited by 159
  29. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
  30. 2021Offline RL Without Off-Policy Evaluationcited by 150
  31. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Plannerscited by 147
  32. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  33. 2022Prompting Decision Transformer for Few-Shot Policy Generalizationcited by 143
  34. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  35. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  36. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 137
  37. 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
  38. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136

50 most cited of 1,873 All 1,873 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020MOPO: Model-based Offline Policy Optimization
  2. 2022Data Valuation for Offline Reinforcement Learning
  3. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
  4. 2023Guiding Online Reinforcement Learning with Action-Free Offline Pretraining
  5. 2025Offline Reinforcement Learning in Large State Spaces: Algorithms and Guarantees
  6. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  7. 2021Interpretable performance analysis towards offline reinforcement learning: A dataset perspective
  8. 2025On the Statistical Complexity for Offline and Low-Adaptive Reinforcement Learning with Structures