mlsanityData to 7 Oct 2026

Aligning Crowd Feedback via Distributional Preference Reward Modeling

arXiv 2402.09764 ↗First posted 15 Feb 2024cs.AIcited by 26

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Deep Reinforcement Learning is widely used for aligning Large Language Models (LLM) with human preference. However, the conventional reward modelling is predominantly dependent on human annotations provided by a select cohort of individuals. Such dependence may unintentionally result in skewed models that reflect the inclinations of these annotators, thereby failing to adequately represent the wider population's expectations. We propose the Distributional Preference Reward Model (DPRM), a simple yet effective framework to align large language models with diverse human preferences. To this end, we characterize multiple preferences by a categorical distribution and introduce a Bayesian updater to accommodate shifted or new preferences. On top of that, we design an optimal-transportation-based loss to calibrate DPRM to align with the preference distribution. Finally, the expected reward is utilized to fine-tune an LLM policy to generate responses favoured by the population. Our experiments show that DPRM significantly enhances the alignment of LLMs with population preference, yielding more accurate, unbiased, and contextually appropriate responses.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 32 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

22 of 32 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 19

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  10. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2019When Does Label Smoothing Help?cited by 985
7 more
  1. 2021Ethical and social risks of harm from Language Modelscited by 957
  2. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  3. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  4. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  5. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  6. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  7. 2023The False Promise of Imitating Proprietary LLMscited by 254

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 11; 2026: 3.

26 papers.

Sort:

By area:

  1. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 136
  2. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  3. 2024Quantile Regression for Distributional Reward Models in RLHFcited by 59
  4. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  5. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 36
  6. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  7. 2025Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Datasetcited by 32
  8. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 23
  9. 2024No Preference Left Behind: Group Distributional Preference Optimizationcited by 21
  10. 2025Direct Alignment with Heterogeneous Preferencescited by 19
  11. 2024Geometric-Averaged Preference Optimization for Soft Preference Labelscited by 18
  12. 2024Beyond the Binary: Capturing Diverse Preferences With Reward Regularizationcited by 11
14 more
  1. 2025Pluralistic Off-policy Evaluation and Alignmentcited by 11
  2. 2025LIVS: A Pluralistic Alignment Dataset for Inclusive Public Spacescited by 8
  3. 2025Translate Policy to Language: Flow Matching Generated Rewards for LLM Explanationscited by 6
  4. 2024Semantic Loss Guided Data Efficient Supervised Fine Tuning for Safe Responses in LLMscited by 4
  5. 2024Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignmentcited by 4
  6. 2025Beyond Single-Point Judgment: Distribution Alignment for LLM-as-a-Judgecited by 4
  7. 2025Safety Must Precede the Deployment of Open-Ended AIcited by 3
  8. 2026WildReward: Learning Reward Models from In-the-Wild Human Interactionscited by 3
  9. 2024Learning From Crowdsourced Noisy Labels: A Signal Processing Perspectivecited by 2
  10. 2025MOSLIM:Align with diverse preferences in prompts through reward classificationcited by 2
  11. 2024PMoL: Parameter Efficient MoE for Preference Mixing of LLM Alignmentcited by 1
  12. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferencescited by 1
  13. 2026MixDPO: Modeling Preference Strength for Pluralistic Alignmentcited by 1
  14. 2026Diffusion Reward Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization
  2. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
  3. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models
  4. 2025Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibrium
  5. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  6. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  7. 2024MaxMin-RLHF: Alignment with Diverse Human Preferences
  8. 2024Aligning language models with human preferences