mlsanityData to 14 Oct 2026

Active Query Selection for Crowd-Based Reinforcement Learning

arXiv 2508.19132 ↗First posted 26 Aug 2025cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Preference-based reinforcement learning has gained prominence as a strategy for training agents in environments where the reward signal is difficult to specify or misaligned with human intent. However, its effectiveness is often limited by the high cost and low availability of reliable human input, especially in domains where expert feedback is scarce or errors are costly. To address this, we propose a novel framework that combines two complementary strategies: probabilistic crowd modelling to handle noisy, multi-annotator feedback, and active learning to prioritize feedback on the most informative agent actions. We extend the Advise algorithm to support multiple trainers, estimate their reliability online, and incorporate entropy-based query selection to guide feedback requests. We evaluate our approach in a set of environments that span both synthetic and real-world-inspired settings, including 2D games (Taxi, Pacman, Frozen Lake) and a blood glucose control task for Type 1 Diabetes using the clinically approved UVA/Padova simulator. Our preliminary results demonstrate that agents trained with feedback on uncertain trajectories exhibit faster learning in most tasks, and we outperform the baselines for the blood glucose control task.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 14 of its 30 references that are in the map. First posted on arXiv 26 Aug 2025.

Builds on

14 of 30 references.

References · 14

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Deep reinforcement learning from human preferencescited by 3,912
  3. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  4. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  5. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 123
  6. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 123
  7. 2017Deep learning from crowdscited by 105
  8. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 89
  9. 2024How to Evaluate Reward Models for RLHFcited by 88
  10. 2017Learning how to Active Learn: A Deep Reinforcement Learning Approachcited by 79
  11. 2018DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedbackcited by 41
  12. 2020Learning from Crowds by Modeling Common Confusionscited by 32
2 more
  1. 2023Learning to Identify Critical States for Reinforcement Learning from Videoscited by 9
  2. 2024Crowd-PrefRL: Preference-Based Reward Learning from Crowdscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Reinforcement Learning from Human Feedback with Active Queries
  2. 2025PB2: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning
  3. 2025Contextual bandits with entropy-based human feedback
  4. 2025Efficient Reinforcement Learning from Human Feedback via Bayesian Preference Inference
  5. 2024Cost-Effective Proxy Reward Model Construction with On-Policy and Active Learning
  6. 2023End-to-end Offline Reinforcement Learning for Glycemia Control
  7. 2023Sample Efficient Preference Alignment in LLMs via Active Exploration
  8. 2025Efficient Preference-Based Reinforcement Learning: Randomized Exploration Meets Experimental Design