mlsanityData to 7 Oct 2026

When Life Gives You BC, Make Q-functions: Extracting Q-values from Behavior Cloning for On-Robot Reinforcement Learning

arXiv 2605.05172 ↗First posted 6 May 2026cs.RO cs.AIcited by 3

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Behavior Cloning (BC) has emerged as a highly effective paradigm for robot learning. However, BC lacks a self-guided mechanism for online improvement after demonstrations have been collected. Existing offline-to-online learning methods often cause policies to replace previously learned good actions due to a distribution mismatch between offline data and online learning. In this work, we propose Q2RL, Q-Estimation and Q-Gating from BC for Reinforcement Learning, an algorithm for efficient offline-to-online learning. Our method consists of two parts: (1) Q-Estimation extracts a Q-function from a BC policy using a few interaction steps with the environment, followed by online RL with (2) Q-Gating, which switches between BC and RL policy actions based on their respective Q-values to collect samples for RL policy training. Across manipulation tasks from D4RL and robomimic benchmarks, Q2RL outperforms SOTA offline-to-online learning baselines on success rate and time to convergence. Q2RL is efficient enough to be applied in an on-robot RL setting, learning robust policies for contact-rich and high precision manipulation tasks such as pipe assembly and kitting, in 1-2 hours of online interaction, achieving success rates of up to 100% and up to 3.75x improvement against the original BC policy. Code and video are available at https://pages.rai-inst.com/q2rl_website/

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 43 references that are in the map. First posted on arXiv 6 May 2026.

Builds on

36 of 43 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data10 Dec
  2. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  6. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2022Flow Matching for Generative Modelingcited by 5,426
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  5. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  6. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  7. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  8. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  9. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  10. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  11. 2017Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperationcited by 286
  12. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 245
18 more
  1. 2018Residual Reinforcement Learning for Robot Controlcited by 242
  2. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 191
  3. 2024Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learningcited by 153
  4. 2018Residual Policy Learningcited by 146
  5. 2020Learning to Generalize Across Long-Horizon Tasks from Human Demonstrationscited by 114
  6. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learningcited by 113
  7. 2024FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learningcited by 79
  8. 2022Learning and Retrieval from Prior Data for Skill-based Imitation Learningcited by 77
  9. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 60
  10. 2023Imitation Bootstrapped Reinforcement Learningcited by 54
  11. 2025RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learningcited by 47
  12. 2017Estimating Mixture Entropy with Pairwise Distancescited by 34
  13. 2025Streaming Flow Policy: Simplifying diffusion/flow-matching policies by treating action trajectories as flow trajectoriescited by 34
  14. 2022The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Modelscited by 31
  15. 2018Learning under Misspecified Objective Spacescited by 14
  16. 2024On-Robot Reinforcement Learning with Goal-Contrastive Rewardscited by 7
  17. 2025Accelerating Residual Reinforcement Learning with Uncertainty Estimationcited by 6
  18. 2025Quantization-Free Autoregressive Action Transformercited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planningcited by 4
  2. 2026A Minimalist Retargeting-Guided Reinforcement Learning Recipe for Dexterous Manipulationcited by 1
  3. 2026Find Something You Can't Do: Agentic Real-World Reinforcement Learning for Self-Improving VLA Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions
  2. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policies
  3. 2019Integrating Behavior Cloning and Reinforcement Learning for Improved Performance in Dense and Sparse Reward Environments
  4. 2020Offline Learning from Demonstrations and Unlabeled Experience
  5. 2023Improving Behavioural Cloning with Positive Unlabeled Learning
  6. 2026RankQ: Offline-to-Online Reinforcement Learning via Self-Supervised Action Ranking
  7. 2021Quantile Filtered Imitation Learning
  8. 2025Should We Ever Prefer Decision Transformer for Offline Reinforcement Learning?