mlsanityData to 7 Oct 2026

Fill in the Blanks: Accelerating Q-Learning with a Handful of Demonstrations in Sparse Reward Settings

arXiv 2510.24432 ↗First posted 28 Oct 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) in sparse-reward environments remains a significant challenge due to the lack of informative feedback. We propose a simple yet effective method that uses a small number of successful demonstrations to initialize the value function of an RL agent. By precomputing value estimates from offline demonstrations and using them as targets for early learning, our approach provides the agent with a useful prior over promising actions. The agent then refines these estimates through standard online interaction. This hybrid offline-to-online paradigm significantly reduces the exploration burden and improves sample efficiency in sparse-reward settings. Experiments on benchmark tasks demonstrate that our method accelerates convergence and outperforms standard baselines, even with minimal or suboptimal demonstration data.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 19 of its 26 references that are in the map. First posted on arXiv 28 Oct 2025.

Builds on

19 of 26 references.

References · 19

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  6. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  7. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  8. 2018Behavioral Cloning from Observationcited by 482
  9. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  10. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  11. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  12. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
7 more
  1. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  2. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  3. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  4. 2020Munchausen Reinforcement Learningcited by 68
  5. 2020Goal-GAN: Multimodal Trajectory Prediction Based on Goal Position Estimationcited by 57
  6. 2023Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learningcited by 28
  7. 2025SR-Reward: Taking The Path More Traveledcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Data
  2. 2025DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning
  3. 2023Accelerating Exploration with Unlabeled Prior Data
  4. 2021Online reinforcement learning with sparse rewards through an active inference capsule
  5. 2022Reinforcement learning with Demonstrations from Mismatched Task under Sparse Reward
  6. 2026Rainbow-DemoRL: Combining Improvements in Demonstration-Augmented Reinforcement Learning
  7. 2022Bayesian Q-learning With Imperfect Expert Demonstrations
  8. 2025Sparse-Reg: Improving Sample Complexity in Offline Reinforcement Learning using Sparsity