mlsanityData to 7 Oct 2026

Active Reward Learning from Multiple Teachers

arXiv 2303.00894 ↗First posted 2 Mar 2023cs.LG cs.AIcited by 13

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward learning algorithms utilize human feedback to infer a reward function, which is then used to train an AI system. This human feedback is often a preference comparison, in which the human teacher compares several samples of AI behavior and chooses which they believe best accomplishes the objective. While reward learning typically assumes that all feedback comes from a single teacher, in practice these systems often query multiple teachers to gather sufficient training data. In this paper, we investigate this disparity, and find that algorithmic evaluation of these different sources of feedback facilitates more accurate and efficient reward learning. We formally analyze the value of information (VOI) when reward learning from teachers with varying levels of rationality, and define and evaluate an algorithm that utilizes this VOI to actively select teachers to query for feedback. Surprisingly, we find that it is often more informative to query comparatively irrational teachers. By formalizing this problem and deriving an analytical solution, we hope to facilitate improvement in reward learning approaches to aligning AI behavior with human values.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 25 of its 41 references that are in the map. First posted on arXiv 2 Mar 2023.

Builds on

25 of 41 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022The Expertise Problem: Learning from Specialized Feedback12 Nov
  2. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learning24 May
  3. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  4. 2021Choice Set Misspecification in Reward Inference19 Jan
  5. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  6. 2019Deep Reinforcement Learning from Policy-Dependent Human Feedback12 Feb
  7. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  8. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 17

  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  3. 2017Residual Attention Network for Image Classificationcited by 385
  4. 2017Inverse Reward Designcited by 246
  5. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 119
  6. 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
  7. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  8. 2019Asking Easy Questions: A User-Friendly Approach to Active Reward Learningcited by 67
  9. 2019Learning Reward Functions by Integrating Human Demonstrations and Preferencescited by 64
  10. 2019On the Feasibility of Learning, Rather than Assuming, Human Biases for Reward Inferencecited by 49
  11. 2015Learning the Preferences of Ignorant, Inconsistent Agentscited by 45
  12. 2020Adapting a Kidney Exchange Algorithm to Align with Human Valuescited by 36
5 more
  1. 2022Misspecification in Inverse Reinforcement Learningcited by 27
  2. 2021Human irrationality: both bad and good for reward inferencecited by 25
  3. 2020News Recommender System: A review of recent progress, challenges, and opportunitiescited by 24
  4. 2019Literal or Pedagogic Human? Analyzing Human Model Misspecification in Objective Learningcited by 15
  5. 2018Active Inverse Reward Designcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 5; 2025: 2; 2026: 2.

13 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  2. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  3. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  4. 2024DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agentscited by 68
  5. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  6. 2023Decision Making for Human-in-the-loop Robotic Agents via Uncertainty-Aware Reinforcement Learningcited by 9
  7. 2024Pareto-Optimal Learning from Preferences with Hidden Contextcited by 8
  8. 2023Active teacher selection for reward learningcited by 7
  9. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learningcited by 6
  10. 2023Minimum Levels of Interpretability for Artificial Moral Agentscited by 5
  11. 2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4
  12. 2026Adaptive Pluralistic Alignment: A pipeline for dynamic artificial democracycited by 2
1 more
  1. 2025Supervised Reward Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026To Go Far, Go Together: Diverse Preferences Induce a Curriculum for Reward Optimization
  2. 2025Conservative Bias in Multi-Teacher Learning: Why Agents Prefer Low-Reward Advisors
  3. 2026Exploring Re-inforcement Learning via Human Feedback under User Heterogeneity
  4. 2020Learning Rewards from Linguistic Feedback
  5. 2025Reward Learning from Multiple Feedback Types
  6. 2024Dual Active Learning for Reinforcement Learning from Human Feedback
  7. 2025Reinforcement Learning from Multi-level and Episodic Human Feedback
  8. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework