mlsanityData to 7 Oct 2026

Offline Reinforcement Learning Hands-On

arXiv 2011.14379 ↗First posted 29 Nov 2020cs.LGcited by 6

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline Reinforcement Learning (RL) aims to turn large datasets into powerful decision-making engines without any online interactions with the environment. This great promise has motivated a large amount of research that hopes to replicate the success RL has experienced in simulation settings. This work ambitions to reflect upon these efforts from a practitioner viewpoint. We start by discussing the dataset properties that we hypothesise can characterise the type of offline methods that will be the most successful. We then verify these claims through a set of experiments and designed datasets generated from environments with both discrete and continuous action spaces. We experimentally validate that diversity and high-return examples in the data are crucial to the success of offline RL and show that behavioural cloning remains a strong contender compared to its contemporaries. Overall, this work stands as a tutorial to help people build their intuition on today's offline RL methods and their applicability.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 14 of its 18 references that are in the map. First posted on arXiv 29 Nov 2020.

Builds on

14 of 18 references.

Closest ancestors · 8

Same direction, newest first

  1. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  2. 2020Critic Regularized Regression26 Jun
  3. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning24 Jun
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  7. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  8. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 6

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647

Built on by

Cited by year

Citing papers per year: 2021: 3; 2022: 0; 2023: 2; 2024: 0; 2025: 1.

6 papers.

  1. 2021A Dataset Perspective on Offline Reinforcement Learningcited by 27
  2. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutionscited by 26
  3. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matchingcited by 24
  4. 2021On Multi-objective Policy Optimization as a Tool for Reinforcement Learning: Case Studies in Offline RL and Finetuningcited by 4
  5. 2021Offline Policy Comparison under Limited Historical Agent-Environment Interactionscited by 3
  6. 2025State Diversity Matters in Offline Behavior Distillationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?
  2. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  3. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  4. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory
  5. 2021A Minimalist Approach to Offline Reinforcement Learning
  6. 2023The Generalization Gap in Offline Reinforcement Learning
  7. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  8. 2024Benchmarks for Reinforcement Learning with Biased Offline Data and Imperfect Simulators