mlsanityData to 14 Oct 2026

Reinforcement Learning via Implicit Imitation Guidance

arXiv 2506.07505 ↗First posted 9 Jun 2025cs.LG cs.AIcited by 12

On the map: Offline RL and imitation learning from data → Imitation learning methods

also in 1 more
Abstract (arXiv)

We study the problem of sample efficient reinforcement learning, where prior data such as demonstrations are provided for initialization in lieu of a dense reward signal. A natural approach is to incorporate an imitation learning objective, either as regularization during training or to acquire a reference policy. However, imitation learning objectives can ultimately degrade long-term performance, as it does not directly align with reward maximization. In this work, we propose to use prior data solely for guiding exploration via noise added to the policy, sidestepping the need for explicit behavior cloning constraints. The key insight in our framework, Data-Guided Noise (DGN), is that demonstrations are most useful for identifying which actions should be explored, rather than forcing the policy to take certain actions. Our approach achieves up to 2-3x improvement over prior reinforcement learning from offline data methods across seven simulated continuous control tasks.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 36 references that are in the map. First posted on arXiv 9 Jun 2025.

Builds on

29 of 36 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Imitation Bootstrapped Reinforcement Learning3 Nov
  2. 2022Watch and Match: Supercharging Imitation with Regularized Optimal Transport30 Jun
  3. 2019Go-Explore: a New Approach for Hard-Exploration Problems30 Jan

Other references · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  3. 2018Exploration by Random Network Distillationcited by 993
  4. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
  5. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  6. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  7. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  8. 2020Planning to Explore via Self-Supervised World Modelscited by 368
  9. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  10. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  11. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  12. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
14 more
  1. 2019Self-Supervised Exploration via Disagreementcited by 265
  2. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  3. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  4. 2019Efficient Exploration via State Marginal Matchingcited by 188
  5. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 174
  6. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 167
  7. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  8. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 138
  9. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 104
  10. 2022MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrationscited by 62
  11. 2019Making Efficient Use of Demonstrations to Solve Hard Exploration Problemscited by 45
  12. 2018Computational Theories of Curiosity-Driven Learningcited by 11
  13. 2019Learning latent state representation for speeding up explorationcited by 8
  14. 2023Offline Retraining for Online RL: Decoupled Policy Learning to Mitigate Exploration Biascited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 9.

12 papers.

  1. 2025Self-Improving Vision-Language-Action Models with Data Generation via Residual RLcited by 78
  2. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 34
  3. 2025Value Flowscited by 15
  4. 2026EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Modelscited by 11
  5. 2026TQL: Scaling Q-Functions with Transformers by Preventing Attention Collapsecited by 9
  6. 2026Fine-tuning is Not Enough: A Parallel Framework for Collaborative Imitation and Reinforcement Learning in End-to-end Autonomous Drivingcited by 4
  7. 2026FASTER: Value-Guided Sampling for Fast RLcited by 4
  8. 2026Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?cited by 2
  9. 2026Q-Learning With World Modelscited by 2
  10. 2026Reinforcement Learning for Real-Time Vision-Language-Action Policiescited by 2
  11. 2026Incremental Residual Reinforcement Learning Toward Real-World Learning for Social Navigationcited by 1
  12. 2026APC-RL: Exceeding Data-Driven Behavior Priors with Adaptive Policy Compositioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasets
  2. 2023Good Better Best: Self-Motivated Imitation Learning for noisy Demonstrations
  3. 2025Prior-Guided Diffusion Planning for Offline Reinforcement Learning
  4. 2025Uncertainty-Based Smooth Policy Regularisation for Reinforcement Learning with Few Demonstrations
  5. 2020Universal Value Density Estimation for Imitation Learning and Goal-Conditioned Reinforcement Learning
  6. 2025CCDP: Composition of Conditional Diffusion Policies with Guided Sampling
  7. 2023A Strong Baseline for Batch Imitation Learning
  8. 2022Causal Imitation Learning under Temporally Correlated Noise