mlsanityData to 7 Oct 2026

Adaptive Querying for Reward Learning from Human Feedback

arXiv 2412.07990 ↗First posted 11 Dec 2024cs.RO cs.AI cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Learning from human feedback is a popular approach to train robots to adapt to user preferences and improve safety. Existing approaches typically consider a single querying (interaction) format when seeking human feedback and do not leverage multiple modes of user interaction with a robot. We examine how to learn a penalty function associated with unsafe behaviors using multiple forms of human feedback, by optimizing both the query state and feedback format. Our proposed adaptive feedback selection is an iterative, two-phase approach which first selects critical states for querying, and then uses information gain to select a feedback format for querying across the sampled critical states. The feedback format selection also accounts for the cost and probability of receiving feedback in a certain format. Our experiments in simulation demonstrate the sample efficiency of our approach in learning to avoid undesirable behaviors. The results of our user study with a physical robot highlight the practicality and effectiveness of adaptive feedback selection in seeking informative, user-aligned feedback that accelerate learning. Experiment videos, code and appendices are found on our website: https://tinyurl.com/AFS-learning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 53 references that are in the map. First posted on arXiv 11 Dec 2024.

Builds on

25 of 53 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Reinforcement Learning from Diverse Human Preferences27 Jan
  2. 2022The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types23 Aug
  3. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learning13 Apr
  4. 2018Reward learning from human preferences and demonstrations in Atari15 Nov

Other references · 21

  1. 2016Concrete Problems in AI Safetycited by 1,582
  2. 2017Inverse Reward Designcited by 246
  3. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 135
  4. 2023RoboCLIP: One Demonstration is Enough to Learn Robot Policiescited by 128
  5. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  6. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  7. 2020Learning Reward Functions from Diverse Sources of Human Feedback: Optimally Integrating Demonstrations and Preferencescited by 71
  8. 2023"No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomycited by 60
  9. 2020Avoiding Side Effects By Considering Future Taskscited by 32
  10. 2020Bayesian Robust Optimization for Imitation Learningcited by 28
  11. 2016Identifying Unknown Unknowns in the Open World: Representations and Policies for Guided Explorationcited by 26
  12. 2020Reinforcement learning with human advice: a surveycited by 23
9 more
  1. 2023Incremental Learning of Humanoid Robot Behavior from Natural Interaction and Large Language Modelscited by 19
  2. 2024Trajectory Improvement and Reward Learning from Comparative Language Feedbackcited by 17
  3. 2023Guide Your Agent with Adaptive Multimodal Rewardscited by 10
  4. 2023Multimodal Pretrained Models for Verifiable Sequential Decision-Making: Planning, Grounding, and Perceptioncited by 8
  5. 2024Coherence-Driven Multimodal Safety Dialogue with Active Learning for Embodied Agentscited by 7
  6. 2021Learning from Ambiguous Demonstrations with Self-Explanation Guided Reinforcement Learningcited by 5
  7. 2024IDIL: Imitation Learning of Intent-Driven Expert Behaviorcited by 5
  8. 2024Safe Reinforcement Learning with Free-form Natural Language Constraints and Pre-Trained Language Modelscited by 4
  9. 2021Mitigating Negative Side Effects via Environment Shapingcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Active Reward Learning from Online Preferences
  2. 2023Boosting Feedback Efficiency of Interactive Reinforcement Learning by Adaptive Learning from Scores
  3. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  4. 2022Learning from Physical Human Feedback: An Object-Centric One-Shot Adaptation Method
  5. 2025Data-adaptive Safety Rules for Training Reward Models
  6. 2025Reinforcement Learning from Multi-level and Episodic Human Feedback
  7. 2022Feedback-efficient Active Preference Learning for Socially Aware Robot Navigation
  8. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective