mlsanityData to 7 Oct 2026

Crowd-PrefRL: Preference-Based Reward Learning from Crowds

arXiv 2401.10941 ↗First posted 17 Jan 2024cs.HC cs.LG cs.SIcited by 7

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Preference-based reinforcement learning (RL) provides a framework to train AI agents using human feedback through preferences over pairs of behaviors, enabling agents to learn desired behaviors when it is difficult to specify a numerical reward function. While this paradigm leverages human feedback, it typically treats the feedback as given by a single human user. However, different users may desire multiple AI behaviors and modes of interaction. Meanwhile, incorporating preference feedback from crowds (i.e. ensembles of users) in a robust manner remains a challenge, and the problem of training RL agents using feedback from multiple human users remains understudied. In this work, we introduce a conceptual framework, Crowd-PrefRL, that integrates preference-based RL approaches with techniques from unsupervised crowdsourcing to enable training of autonomous system behaviors from crowdsourced feedback. We show preliminary results suggesting that Crowd-PrefRL can learn reward functions and agent policies from preference feedback provided by crowds of unknown expertise and reliability. We also show that in most cases, agents trained with Crowd-PrefRL outperform agents trained with majority-vote preferences or preferences from any individual user, especially when the spread of user error rates among the crowd is large. Results further suggest that our method can identify the presence of minority viewpoints within the crowd in an unsupervised manner.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 20 of its 36 references that are in the map. First posted on arXiv 17 Jan 2024.

Builds on

20 of 36 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF13 Dec
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  3. 2023Breadcrumbs to the Goal: Goal-Conditioned Exploration from Human-in-the-Loop Feedback20 Jul
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  6. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 14

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2018DeepMind Control Suitecited by 935
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  6. 2023Towards Measuring the Representation of Subjective Global Opinions in Language Modelscited by 424
  7. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  8. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  9. 2017Multi-Modal Imitation Learning from Unstructured Demonstrations using Generative Adversarial Netscited by 64
  10. 2023Group Preference Optimization: Few-Shot Alignment of Large Language Modelscited by 61
  11. 2022Imitation Learning by Estimating Expertise of Demonstratorscited by 50
  12. 2021Learning Reward Functions from Scale Feedbackcited by 24
2 more
  1. 2013Ranking and combining multiple predictors without labeled datacited by 22
  2. 2021Batch Reinforcement Learning from Crowdscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 2.

7 papers.

  1. 2024Corruption Robust Offline Reinforcement Learning with Human Feedbackcited by 20
  2. 2024Scalable Interactive Machine Learning for Future Command and Controlcited by 2
  3. 2024Learning From Crowdsourced Noisy Labels: A Signal Processing Perspectivecited by 2
  4. 2025Human-Inspired Multi-Level Reinforcement Learningcited by 1
  5. 2025Active Query Selection for Crowd-Based Reinforcement Learningcited by 0
  6. 2026Implicit Safety Alignment from Crowd Preferencescited by 0
  7. 2026BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024PrefCLM: Enhancing Preference-based Reinforcement Learning with Crowdsourced Large Language Models ↗
  2. 2023Reinforcement Learning from Diverse Human Preferences
  3. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation
  4. 2025Pref-GUIDE: Continual Policy Learning from Real-Time Human Feedback via Preference-Based Learning
  5. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  6. 2025PB2: Preference Space Exploration via Population-Based Methods in Preference-Based Reinforcement Learning
  7. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
  8. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint