mlsanityData to 14 Oct 2026

On Diversified Preferences of Large Language Model Alignment

arXiv 2312.07401 ↗First posted 12 Dec 2023cs.AIcited by 26

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning large language models (LLMs) with human preferences has been recognized as the key to improving LLMs' interaction quality. However, in this pluralistic world, human preferences can be diversified due to annotators' different tastes, which hinders the effectiveness of LLM alignment methods. This paper presents the first quantitative analysis of the experimental scaling law for reward models with varying sizes, from 1.3 billion to 7 billion parameters, trained with human feedback exhibiting diverse preferences. Our analysis reveals that the impact of diversified human preferences depends on both model size and data size. Larger models with sufficient capacity mitigate the negative effects of diverse preferences, while smaller models struggle to accommodate them. To mitigate the impact of diverse preferences, we introduce a new metric, Expected Calibration Error (ECE), to evaluate RMs and show their obvious positive correlation with the alignment performance of LLMs. Furthermore, we propose a Multi-Objective Reward learning method (MORE) to enhance the calibration performance of RMs on shared preferences. Through experiments on four models and five human preference datasets, we find the calibration error can be adopted as a key metric for evaluating RMs and MORE can obtain superior alignment performance.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 48 of its 74 references that are in the map. First posted on arXiv 12 Dec 2023.

Builds on

48 of 74 references.

Closest ancestors · 9

Same direction, newest first

  1. 2023Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game14 Nov
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  3. 2023Group Preference Optimization: Few-Shot Alignment of Large Language Models17 Oct
  4. 2023Aligning Language Models with Human Preferences via a Bayesian Approach9 Oct
  5. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  6. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback24 May
  7. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  8. 2022Training language models to follow instructions with human feedback4 Mar
  9. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 39

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  5. 2017On Calibration of Modern Neural Networkscited by 4,960
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
27 more
  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  2. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 872
  3. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 821
  4. 2021Domain Generalization: A Surveycited by 764
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  6. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  7. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  8. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  9. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  10. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 375
  11. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 328
  12. 2021Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective Annotationscited by 314
  13. 2023Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspectivecited by 280
  14. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  15. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  16. 2023How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Compositioncited by 223
  17. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  18. 2021Learning to Diversify for Single Domain Generalizationcited by 170
  19. 2020Residual Energy-Based Models for Text Generationcited by 95
  20. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 93
  21. 2023On the Calibration of Large Language Models and Alignmentcited by 73
  22. 2021Agreeing to Disagree: Annotating Offensive Language Datasets with Annotators' Disagreementcited by 72
  23. 2019Sticking to the Facts: Confident Decoding for Faithful Data-to-Text Generationcited by 62
  24. 2023Everyone's Voice Matters: Quantifying Annotation Disagreement Using Demographic Informationcited by 52
  25. 2023Everyone Deserves A Reward: Learning Customized Human Preferencescited by 43
  26. 2021Which Samples Should be Learned First: Easy or Hard?cited by 12
  27. 2023On the Power of Adaptive Weighted Aggregation in Heterogeneous Federated Learning and Beyondcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 6; 2026: 10.

26 papers.

Sort:

By area:

  1. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 80
  2. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 72
  3. 2024Personality Alignment of Large Language Modelscited by 36
  4. 2024SHED: Shapley-Based Automated Dataset Refinement for Instruction Fine-Tuningcited by 19
  5. 2025Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personascited by 19
  6. 2025Direct Alignment with Heterogeneous Preferencescited by 19
  7. 2024Reward Modeling with Ordinal Feedback: Wisdom of the Crowdcited by 11
  8. 2024Contextual Moral Value Alignment Through Context-Based Aggregationcited by 10
  9. 2025Medical Reasoning in the Era of LLMs: A Systematic Review of Enhancement Techniques and Applicationscited by 10
  10. 2024Regularized Conditional Diffusion Model for Multi-Task Preference Alignmentcited by 9
  11. 2024Atoxia: Red-teaming Large Language Models with Target Toxic Answerscited by 7
  12. 2025Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidancecited by 7
14 more
  1. 2024BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalizationcited by 6
  2. 2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4
  3. 2026Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skillcited by 3
  4. 2026GD2PO: Mitigating Multi-Reward Conflicts via Group-Dynamic reward-Decoupled Policy Optimizationcited by 3
  5. 2024From Efficiency to Equity: Measuring Fairness in Preference Learningcited by 2
  6. 2026Reducing Hallucinations in LLMs via Factuality-Aware Preference Learningcited by 2
  7. 2026Structuring the Space of Sociotechnical Alignmentcited by 2
  8. 2025Is On-Policy Data always the Best Choice for Direct Preference Optimization-based LM Alignment?cited by 1
  9. 2026Spectral Souping: A Unified Framework for Online Preference Alignmentcited by 1
  10. 2025SMRC: Aligning Large Language Models with Student Reasoning for Mathematical Error Correctioncited by 0
  11. 2026PolyAlign: Conditional Human-Distribution Alignmentcited by 0
  12. 2026AlignDiff: Exploiting Model-Intrinsic Information for Better Preference Data Selectioncited by 0
  13. 2026PlurVA-LLM-2026 Shared Task Track-1: Pluralistic Value Alignment in LLMs via Multilingual Fine-Tuning and Threshold Calibrationcited by 0
  14. 2026Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Samplingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  2. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  3. 2024MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time
  4. 2025Towards Understanding Valuable Preference Data for Large Language Model Alignment
  5. 2026Preference Data Selection for Mitigating the Alignment Tax in Large Language Models
  6. 2025Larger or Smaller Reward Margins to Select Preferences for Alignment?
  7. 2025Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
  8. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier