mlsanityData to 7 Oct 2026

Efficient Offline Reinforcement Learning: First Imitate, then Improve

arXiv 2406.13376 ↗First posted 19 Jun 2024cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Supervised imitation-based approaches are often favored over off-policy reinforcement learning approaches for learning policies offline, since their straightforward optimization objective makes them computationally efficient and stable to train. However, their performance is fundamentally limited by the behavior policy that collected the dataset. Off-policy reinforcement learning provides a promising approach for improving on the behavior policy, but training is often computationally inefficient and unstable due to temporal-difference bootstrapping. In this paper, we propose a best-of-both approach by pre-training with supervised learning before improving performance with off-policy reinforcement learning. Specifically, we demonstrate improved efficiency by pre-training an actor with behavior cloning and a critic with a supervised Monte-Carlo value error. We find that we are able to substantially improve the training time of popular off-policy algorithms on standard benchmarks, and also achieve greater stability. Code is available at: https://github.com/AdamJelley/EfficientOfflineRL

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 44 of its 54 references that are in the map. First posted on arXiv 19 Jun 2024.

Builds on

44 of 54 references.

Closest ancestors · 13

Same direction, newest first

  1. 2023Improving Offline RL by Blending Heuristics1 Jun
  2. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learning16 May
  3. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions30 Mar
  4. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  5. 2022When does return-conditioned supervised learning work for offline reinforcement learning?2 Jun
  6. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?12 Apr
  7. 2021RvS: What is Essential for Offline RL via Supervised Learning?20 Dec
  8. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  9. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  10. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
1 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 31

  1. 2016Layer Normalizationcited by 5,579
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  8. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  10. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  11. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  12. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
19 more
  1. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  2. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  3. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  4. 2017Count-Based Exploration with Neural Density Modelscited by 255
  5. 2024Reinforcement learningcited by 236
  6. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 172
  7. 2018Self-Imitation Learningcited by 135
  8. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  9. 2019Training Agents using Upside-Down Reinforcement Learningcited by 106
  10. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 65
  11. 2022You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environmentscited by 62
  12. 2021Tactical Optimism and Pessimism for Deep Reinforcement Learningcited by 55
  13. 2022Dichotomy of Control: Separating What You Can Control from What You Cannotcited by 47
  14. 2018An Off-policy Policy Gradient Theorem Using Emphatic Weightingscited by 34
  15. 2022Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Errorcited by 31
  16. 2021A Generalized Projected Bellman Error for Off-policy Value Estimation in Reinforcement Learningcited by 26
  17. 2022UniMASK: Unified Inference in Sequential Decision Problemscited by 23
  18. 2016Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tighteningcited by 21
  19. 2022Monte Carlo Augmented Actor-Critic for Sparse Reward Deep Reinforcement Learning from Suboptimal Demonstrationscited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learning
  2. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  3. 2021Curriculum Offline Imitation Learning
  4. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  5. 2021Offline RL Without Off-Policy Evaluation
  6. 2024SelfBC: Self Behavior Cloning for Offline Reinforcement Learning
  7. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  8. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Models