mlsanityData to 14 Oct 2026

Selective Progress-Aware Querying for Human-in-the-Loop Reinforcement Learning

arXiv 2509.20541 ↗First posted 24 Sep 2025cs.ROcited by 1

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Human feedback can greatly accelerate robot learning, but in real-world settings, such feedback is costly and limited. Existing human-in-the-loop reinforcement learning (HiL-RL) methods often assume abundant feedback, limiting their practicality for physical robot deployment. In this work, we introduce SPARQ, a progress-aware query policy that requests feedback only when learning stagnates or worsens, thereby reducing unnecessary oracle calls. We evaluate SPARQ on a simulated UR5 cube-picking task in PyBullet, comparing against three baselines: no feedback, random querying, and always querying. Our experiments show that SPARQ achieves near-perfect task success, matching the performance of always querying while consuming about half the feedback budget. It also provides more stable and efficient learning than random querying, and significantly improves over training without feedback. These findings suggest that selective, progress-based query strategies can make HiL-RL more efficient and scalable for robots operating under realistic human effort constraints.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 9 of its 16 references that are in the map. First posted on arXiv 24 Sep 2025.

Builds on

9 of 16 references.

References · 9

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2018HG-DAgger: Interactive Imitation Learning with Human Expertscited by 266
  4. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  5. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 123
  6. 2021ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learningcited by 108
  7. 2024Diff-DAgger: Uncertainty Estimation with Diffusion Policy for Robotic Manipulationcited by 33
  8. 2020Soliciting Human-in-the-Loop User Feedback for Interactive Machine Learning Reduces User Trust and Impressions of Model Accuracycited by 20
  9. 2020Cost-effective Interactive Attention Learning with Neural Attention Processescited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026E2HiL: Entropy-Guided Sample Selection for Efficient Real-World Human-in-the-Loop Reinforcement Learningcited by 5

Similar, not cited

Close in wording, no citation either way

  1. 2024Adaptive Querying for Reward Learning from Human Feedback
  2. 2026Elastic Queries Reinforcement Learning: Self-Aware Policy Execution for VLA Models
  3. 2024Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks
  4. 2023Breadcrumbs to the Goal: Goal-Conditioned Exploration from Human-in-the-Loop Feedback
  5. 2026Max-Q Selective Imitation for Human-in-the-Loop Online Robot Learning
  6. 2024Reinforcement Learning from Human Feedback with Active Queries
  7. 2023Query-Policy Misalignment in Preference-Based Reinforcement Learning
  8. 2025Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback