Rainbow-DemoRL: Combining Improvements in Demonstration-Augmented Reinforcement Learning
Not on the map. Why?
Abstract (arXiv)
Several approaches have been proposed to improve the sample efficiency of online reinforcement learning (RL) by leveraging demonstrations collected offline. The offline data can be used directly as transitions to optimize RL objectives, or offline policy and value functions can first be learned from the data and then used for online finetuning or to provide reference actions. While each of these strategies has shown compelling results, it is unclear which method has the most impact on sample efficiency, whether these approaches can be combined, and if there are cumulative benefits. We classify existing demonstration-augmented RL approaches into three categories and perform an extensive empirical study of their strengths, weaknesses, and combinations to isolate the contribution of each strategy and determine effective hybrid combinations for sample-efficient online RL. Our analysis reveals that directly reusing offline data and initializing with behavior cloning consistently outperform more complex offline RL pretraining methods for improving online sample efficiency.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 22 of its 28 references that are in the map. First posted on arXiv 28 Mar 2026.
Builds on
22 of 28 references.
References · 22
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
- 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
- 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
- 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
- 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
- 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
- 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 278
- 2018Residual Reinforcement Learning for Robot Controlcited by 242
- 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
10 more
- 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 153
- 2018Residual Policy Learningcited by 146
- 2023A Survey of Imitation Learning: Algorithms, Recent Developments, and Challengescited by 145
- 2023Real-World Humanoid Locomotion with Reinforcement Learningcited by 132
- 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
- 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 58
- 2023Imitation Bootstrapped Reinforcement Learningcited by 54
- 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 42
- 2024Learning Robust Autonomous Navigation and Locomotion for Wheeled-Legged Robotscited by 32
- 2024Contextualized Hybrid Ensemble Q-learning: Learning Fast with Control Priorscited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2020Offline Reinforcement Learning Hands-On
- 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
- 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
- 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
- 2025Active Advantage-Aligned Online Reinforcement Learning with Offline Data
- 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
- 2022Offline Reinforcement Learning with Adaptive Behavior Regularization
- 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning