mlsanityData to 14 Oct 2026

A Minimalist Approach to Offline Reinforcement Learning

arXiv 2106.06860 ↗First posted 12 Jun 2021cs.LG cs.AI stat.MLcited by 899

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) defines the task of learning from a fixed batch of data. Due to errors in value estimation from out-of-distribution actions, most offline RL algorithms take the approach of constraining or regularizing the policy with the actions contained in the dataset. Built on pre-existing RL algorithms, modifications to make an RL algorithm work offline comes at the cost of additional complexity. Offline RL algorithms introduce new hyperparameters and often leverage secondary components such as generative models, while adjusting the underlying RL algorithm. In this paper we aim to make a deep RL algorithm work while making minimal changes. We find that we can match the performance of state-of-the-art offline RL algorithms by simply adding a behavior cloning term to the policy update of an online RL algorithm and normalizing the data. The resulting algorithm is a simple to implement and tune baseline, while more than halving the overall run time by removing the additional computational overhead of previous methods.

Key facts. Cited by 899 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 45 of its 64 references that are in the map. First posted on arXiv 12 Jun 2021.

Builds on

45 of 64 references.

Closest ancestors · 10

Same direction, newest first

  1. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  2. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  3. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  4. 2020Critic Regularized Regression26 Jun
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  8. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  9. 2019Behavior Regularized Offline Reinforcement Learning26 Nov
  10. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 35

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2015Continuous control with deep reinforcement learningcited by 3,407
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  7. 2016OpenAI Gymcited by 1,736
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  10. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  11. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  12. 2017Deep Reinforcement Learning that Matterscited by 754
23 more
  1. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  2. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  3. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 664
  4. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  5. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  6. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 284
  7. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  8. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 144
  9. 2019Behaviour Suite for Reinforcement Learningcited by 132
  10. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 126
  11. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  12. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  13. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 91
  14. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  15. 2018Evaluating Reinforcement Learning Algorithms in Observational Health Settingscited by 60
  16. 2018The Mirage of Action-Dependent Baselines in Reinforcement Learningcited by 48
  17. 2018Horizon: Facebook's Open Source Applied Reinforcement Learning Platformcited by 48
  18. 2019Making Efficient Use of Demonstrations to Solve Hard Exploration Problemscited by 45
  19. 2018Reinforced Imitation: Sample Efficient Deep Reinforcement Learning for Map-less Navigation by Leveraging Prior Demonstrationscited by 38
  20. 2020Offline Policy Selection under Uncertaintycited by 31
  21. 2019Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Dense and Sparse Reward Environmentscited by 17
  22. 2021Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learningcited by 15
  23. 2020Shaping Rewards for Reinforcement Learning with Imperfect Demonstrations using Generative Modelscited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 24; 2022: 114; 2023: 175; 2024: 214; 2025: 171; 2026: 201.

899 papers.

Sort:

By area:

  1. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  2. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
  3. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  4. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  5. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  6. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  7. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 231
  8. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  9. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  10. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 197
  11. 2022Online Decision Transformercited by 196
  12. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 185
38 more
  1. 2025Flow Q-Learningcited by 184
  2. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  3. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  4. 2023Efficient Diffusion Policies for Offline Reinforcement Learningcited by 157
  5. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 131
  6. 2022Imitation Is Not Enough: Robustifying Imitation with Reinforcement Learning for Challenging Driving Scenarioscited by 129
  7. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
  8. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  9. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 121
  10. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 119
  11. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 119
  12. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 116
  13. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 116
  14. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  15. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learningcited by 115
  16. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  17. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learningcited by 111
  18. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 106
  19. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 104
  20. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  21. 2023Synthetic Experience Replaycited by 101
  22. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 100
  23. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 96
  24. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 91
  25. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 89
  26. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 89
  27. 2021d3rlpy: An Offline Deep Reinforcement Learning Librarycited by 84
  28. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 84
  29. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  30. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 79
  31. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  32. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 77
  33. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 76
  34. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 76
  35. 2023MADiff: Offline Multi-agent Learning with Diffusion Modelscited by 75
  36. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 75
  37. 2022Supported Policy Optimization for Offline Reinforcement Learningcited by 74
  38. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 73

50 most cited of 899 All 899 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2021Offline RL Without Off-Policy Evaluation
  2. 2021The Least Restriction for Offline Reinforcement Learning
  3. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation
  4. 2020MOPO: Model-based Offline Policy Optimization
  5. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  6. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  7. 2020MOReL : Model-Based Offline Reinforcement Learning
  8. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory