ReF-HIL: Shaping the Critic around Human Action Neighborhoods for Efficient Human-in-the-Loop Reinforcement Learning
On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation
Abstract (arXiv)
Human-in-the-loop reinforcement learning (HIL-RL) offers a promising route to efficient training of robotic manipulation policies by combining autonomous learning with human demonstrations and online corrections. However, insufficient use of successful human experience in value learning prolongs costly real-world training, while persistent imitation penalties can limit value-driven policy improvement. To address these limitations, we propose ReF-HIL, an efficient HIL-RL framework that uses human guidance to accelerate the learning process. Human-Reference-Guided Value Shaping learns an independent value reference from successful human experience to guide online value learning, while incorporating local corrective feedback. A Human Action Fence defines a learned human-action neighborhood, allowing value-driven optimization for better performance without imitation penalties inside while constraining policy and value updates outside. Experiments on five diverse and challenging real-world manipulation tasks demonstrate improved overall learning efficiency and higher success rates compared with the evaluated baselines. Specifically, ReF-HIL reaches 90% autonomous success in only 18-63 minutes of active training and achieves final success rates of 91.7-100%. These results highlight the potential of human-guided reinforcement learning to acquire reliable manipulation skills efficiently in the real world. Project website: https://anonymous.4open.science/w/ReF-HIL-7762/
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 27 of its 32 references that are in the map. First posted on arXiv 29 Sep 2026.
Builds on
27 of 32 references.
Closest ancestors · 7
Same direction, newest first
- 2026GAINS: Leveraging Inconsistent Human Intervention Signals in Reinforcement Learning16 Aug
- 2026Max-Q Selective Imitation for Human-in-the-Loop Online Robot Learning15 Aug
- 2026Preference-Calibrated Human-in-the-Loop Reinforcement Learning for Robotic Manipulation2 Jun
- 2026OHP-RL: Online Human Preference as Guidance in Reinforcement Learning for Robot Manipulation15 May
- 2025Real-world Reinforcement Learning from Suboptimal Interventions30 Dec
- 2024Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning29 Oct
- 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes7 Aug
Other references · 20
- 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
- 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
- 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
- 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
- 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
- 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
- 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 266
- 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
- 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 138
- 2022Supported Policy Optimization for Offline Reinforcement Learningcited by 74
- 2023Imitation Bootstrapped Reinforcement Learningcited by 64
- 2022Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimizationcited by 49
8 more
- 2020Behavior Priors for Efficient Reinforcement Learningcited by 41
- 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 34
- 2023RLIF: Interactive Imitation Learning as Reinforcement Learningcited by 30
- 2021Prioritized Experience-based Reinforcement Learning with Human Guidance for Autonomous Drivingcited by 22
- 2025Learning from Active Human Involvement through Proxy Value Propagationcited by 19
- 2019Fixed-Horizon Temporal Difference Methods for Stable Reinforcement Learningcited by 13
- 2024Reinforcement Learning From Imperfect Corrective Actions And Proxy Rewardscited by 3
- 2025From Action Labels to Sets: Rethinking Action Supervision for Imitation Learning from Corrective Feedbackcited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Res-HIL: Human-Guided Residual Reinforcement Learning for Sample-Efficient Dexterous Manipulation
- 2025Human-in-the-loop Online Rejection Sampling for Robotic Manipulation
- 2026HiL-ResRL: A Model-Agnostic Finetuning Adapter via Human-in-the-loop Residual Reinforcement Learning
- 2026From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention
- 2026E2HiL: Entropy-Guided Sample Selection for Efficient Real-World Human-in-the-Loop Reinforcement Learning
- 2026PRISM: Personalized Refinement of Imitation Skills for Manipulation via Human Instructions
- 2026EvoHIL: Self-Evolving Reward and Flow-Matched Policy Optimization for Robust Human-in-the-Loop Reinforcement Learning
- 2026Efficient Real-World Online Reinforcement Learning for Robot Manipulation via Centralized Training and Critic Decomposition