# E2HiL: Entropy-Guided Sample Selection for Efficient Real-World Human-in-the-Loop Reinforcement Learning

arXiv: https://arxiv.org/abs/2601.19969 · HTML: https://mlsanity.com/paper/2601.19969/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 33 of its 48 references that are in the map.
- First posted on arXiv 27 Jan 2026.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 30 of 33)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models](https://mlsanity.com/paper/2505.22617/) (2025)
- [Efficient Online Reinforcement Learning with Offline Data](https://mlsanity.com/paper/2302.02948/) (2023)
- [Maximum a Posteriori Policy Optimisation](https://mlsanity.com/paper/1806.06920/) (2018)
- [Reasoning with Exploration: An Entropy Perspective](https://mlsanity.com/paper/2506.14758/) (2025)
- [HG-DAgger: Interactive Imitation Learning with Human Experts](https://mlsanity.com/paper/1810.02890/) (2018)
- [Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning](https://mlsanity.com/paper/2303.05479/) (2023)
- [Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning](https://mlsanity.com/paper/2410.21845/) (2024)
- [Maximum Entropy RL (Provably) Solves Some Robust RL Problems](https://mlsanity.com/paper/2103.06257/) (2021)
- [Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes](https://mlsanity.com/paper/2408.03539/) (2024)
- [VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning](https://mlsanity.com/paper/2505.18719/) (2025)
- [The Effect of Sampling Temperature on Problem Solving in Large Language Models](https://mlsanity.com/paper/2402.05201/) (2024)
- [ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy](https://mlsanity.com/paper/2502.05450/) (2025)
- [SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning](https://mlsanity.com/paper/2509.09674/) (2025)
- [Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation](https://mlsanity.com/paper/2403.03949/) (2024)
- [A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms](https://mlsanity.com/paper/1907.03146/) (2019)
- [Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods](https://mlsanity.com/paper/2404.00282/) (2024)
- [Model-Based Reinforcement Learning via Meta-Policy Optimization](https://mlsanity.com/paper/1809.05214/) (2018)
- [Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Intervention](https://mlsanity.com/paper/2104.11203/) (2021)
- [Real-World Robot Applications of Foundation Models: A Review](https://mlsanity.com/paper/2402.05741/) (2024)
- [RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learning](https://mlsanity.com/paper/2510.14830/) (2025)
- [Self-Supervised Online Reward Shaping in Sparse-Reward Environments](https://mlsanity.com/paper/2103.04529/) (2021)
- [Fast Rates for Maximum Entropy Exploration](https://mlsanity.com/paper/2303.08059/) (2023)
- [NORA-1.5: A Vision-Language-Action Model Trained using World Model- and Action-based Preference Rewards](https://mlsanity.com/paper/2511.14659/) (2025)
- [Guided Uncertainty-Aware Policy Optimization: Combining Learning and Model-Based Strategies for Sample-Efficient Policy Learning](https://mlsanity.com/paper/2005.10872/) (2020)
- [Empowering Embodied Visual Tracking with Visual Foundation Models and Offline RL](https://mlsanity.com/paper/2404.09857/) (2024)
- [What Matters in Learning from Large-Scale Datasets for Robot Manipulation](https://mlsanity.com/paper/2506.13536/) (2025)
- [Empowering Embodied Manipulation: A Bimanual-Mobile Robot Manipulation Dataset for Household Tasks](https://mlsanity.com/paper/2405.18860/) (2024)
- [VLA-Reasoner: Empowering Vision-Language-Action Models with Reasoning via Online Monte Carlo Tree Search](https://mlsanity.com/paper/2509.22643/) (2025)
- [RLingua: Improving Reinforcement Learning Sample Efficiency in Robotic Manipulations With Large Language Models](https://mlsanity.com/paper/2403.06420/) (2024)

## Built on by (most cited first, 5 of 5)

- [UniIntervene: Agentic Intervention for Efficient Real-World Reinforcement Learning](https://mlsanity.com/paper/2606.12372/) (2026)
- [Preference-Calibrated Human-in-the-Loop Reinforcement Learning for Robotic Manipulation](https://mlsanity.com/paper/2606.03949/) (2026)
- [Facet-0: A Robotic Foundation Model for Contact-Rich Precise Manipulation](https://mlsanity.com/paper/2609.01596/) (2026)
- [Res-HIL: Human-Guided Residual Reinforcement Learning for Sample-Efficient Dexterous Manipulation](https://mlsanity.com/paper/2609.30023/) (2026)
- [UniIntervene++: An Adaptive Intervention Agent for Efficient Real-World Reinforcement Learning](https://mlsanity.com/paper/2610.03620/) (2026)

## Abstract (arXiv)

Human-in-the-loop guidance has emerged as an effective approach for accelerating online reinforcement learning (RL) in real-world manipulation. However, existing human-in-the-loop RL (HiL-RL) frameworks often suffer from low sample efficiency, requiring substantial human interventions to achieve convergence and thereby leading to high labor costs. To address this, we propose a sample-efficient real-world human-in-the-loop RL framework named E2HiL, which requires fewer human interventions by actively selecting informative samples. Specifically, stable reduction of policy entropy enables improved trade-off between exploration and exploitation with higher sample efficiency. We first build influence functions of different samples on the policy entropy, which is efficiently estimated by the covariance of action probabilities and soft advantages of policies. Then we select samples with moderate values of influence functions, where shortcut samples that induce sharp entropy drops and noisy samples with negligible effect are pruned. Extensive experiments across 10 real-world manipulation tasks, spanning multiple embodiments and learning frameworks, demonstrate that E2HiL improves success rates by 24.9% while reducing human interventions by 9.3% compared to state-of-the-art HiL-RL baselines. These results validate its effectiveness as a policy- and embodiment-agnostic plug-and-play module for efficient real-world RL. The project page can be found at https://e2hil.github.io/.