mlsanityData to 14 Oct 2026

The Expertise Problem: Learning from Specialized Feedback

arXiv 2211.06519 ↗First posted 12 Nov 2022cs.LGcited by 20

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) is a powerful technique for training agents to perform difficult-to-specify tasks. However, human feedback can be noisy, particularly when human teachers lack relevant knowledge or experience. Levels of expertise vary across teachers, and a given teacher may have differing levels of expertise for different components of a task. RLHF algorithms that learn from multiple teachers therefore face an expertise problem: the reliability of a given piece of feedback depends both on the teacher that it comes from and how specialized that teacher is on relevant components of the task. Existing state-of-the-art RLHF algorithms assume that all evaluations come from the same distribution, obscuring this inter- and intra-human variance, and preventing them from accounting for or taking advantage of variations in expertise. We formalize this problem, implement it as an extension of an existing RLHF benchmark, evaluate the performance of a state-of-the-art RLHF algorithm, and explore techniques to improve query and teacher selection. Our key contribution is to demonstrate and characterize the expertise problem, and to provide an open-source implementation for testing future solutions.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 20 of its 29 references that are in the map. First posted on arXiv 12 Nov 2022.

Builds on

20 of 29 references.

Closest ancestors · 4

Same direction, newest first

  1. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  2. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  3. 2018Scalable agent alignment via reward modeling: a research direction19 Nov
  4. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 16

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2020Learning to summarize from human feedbackcited by 2,588
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  6. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  7. 2018DeepMind Control Suitecited by 956
  8. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  9. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  10. 2020Reward-rational (implicit) choice: A unifying formalism for reward learningcited by 120
  11. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  12. 2020Learning from Suboptimal Demonstration via Self-Supervised Reward Regressioncited by 84
4 more
  1. 2022X-Risk Analysis for AI Researchcited by 60
  2. 2022Imitation Learning by Estimating Expertise of Demonstratorscited by 50
  3. 2021Confidence-Aware Imitation Learning from Demonstrations with Varying Optimalitycited by 44
  4. 2021Learning from Imperfect Demonstrations from Agents with Varying Dynamicscited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 5; 2025: 5; 2026: 3.

20 papers.

Sort:

By area:

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
  2. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  3. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  4. 2024Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedbackcited by 21
  5. 2023Safety and Fairness for Content Moderation in Generative Modelscited by 16
  6. 2023Perspectives on the Social Impacts of Reinforcement Learning with Human Feedbackcited by 15
  7. 2025RLHS: Mitigating Misalignment in RLHF with Hindsight Simulationcited by 15
  8. 2023Active Reward Learning from Multiple Teacherscited by 13
  9. 2024A Survey On Enhancing Reinforcement Learning in Complex Environments: Insights from Human and LLM Feedbackcited by 9
  10. 2024Pareto-Optimal Learning from Preferences with Hidden Contextcited by 8
  11. 2023Active teacher selection for reward learningcited by 7
  12. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learningcited by 6
8 more
  1. 2023Factual and Personalized Recommendations using Language Models and Reinforcement Learningcited by 4
  2. 2025Interactive Groupwise Comparison for Reinforcement Learning from Human Feedbackcited by 4
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4
  4. 2024When Can Proxies Improve the Sample Complexity of Preference Learning?cited by 1
  5. 2025Contemplative Artificial Intelligencecited by 1
  6. 2023H-COAL: Human Correction of AI-Generated Labels for Biomedical Named Entity Recognitioncited by 0
  7. 2026Mitigating Cognitive Bias in RLHF by Altering Rationalitycited by 0
  8. 2026Finding the Signal in the Spam: Jointly Learning Rewards and Worker Reliability from Pairwise Comparisonscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Dual Active Learning for Reinforcement Learning from Human Feedback
  2. 2023A Survey of Reinforcement Learning from Human Feedback
  3. 2024CANDERE-COACH: Reinforcement Learning from Noisy Feedback
  4. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Framework
  5. 2023RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedback
  6. 2021Improving Reinforcement Learning with Human Assistance: An Argument for Human Subject Studies with HIPPO Gym
  7. 2021Reinforcement Learning with Feedback from Multiple Humans with Diverse Skills
  8. 2025Reinforcement Learning from Human Feedback