mlsanityData to 14 Oct 2026

Pairwise Calibrated Rewards for Pluralistic Alignment

arXiv 2506.06298 ↗First posted 17 May 2025cs.LG cs.AIcited by 14

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Current alignment pipelines presume a single, universal notion of desirable behavior. However, human preferences often diverge across users, contexts, and cultures. As a result, disagreement collapses into the majority signal and minority perspectives are discounted. To address this, we propose reflecting diverse human preferences through a distribution over multiple reward functions, each inducing a distinct aligned policy. The distribution is learned directly from pairwise preference without annotator identifiers or predefined groups. Instead, annotator disagreements are treated as informative soft labels. Our central criterion is pairwise calibration: for every pair of candidate responses, the proportion of reward functions preferring one response matches the fraction of annotators with that preference. We prove that even a small outlier-free ensemble can accurately represent diverse preference distributions. Empirically, we introduce and validate a practical training heuristic to learn such ensembles, and demonstrate its effectiveness through improved calibration, implying a more faithful representation of pluralistic values.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 62 of its 78 references that are in the map. First posted on arXiv 17 May 2025.

Builds on

62 of 78 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Direct Alignment with Heterogeneous Preferences22 Feb
  2. 2025Clone-Robust AI Alignment16 Jan
  3. 2024VPO: Leveraging the Number of Votes in Preference Optimization30 Oct
  4. 2024Margin Matching Preference Optimization: Enhanced Model Alignment with Granular Feedback4 Oct
  5. 2024Geometric-Averaged Preference Optimization for Soft Preference Labels10 Sep
  6. 2024From Distributional to Overton Pluralism: Investigating Large Language Model Alignment25 Jun
  7. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences12 Jun
  8. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling2 Jun
  9. 2024Preference Learning Algorithms Do Not Learn Preference Rankings29 May
  10. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization26 May
  11. 2024MallowsPO: Fine-Tune Your LLM with Preference Dispersions23 May
  12. 2023Aligning Language Models with Human Preferences via a Bayesian Approach9 Oct
1 more
  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 49

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  6. 2022Red Teaming Language Models with Language Modelscited by 1,026
  7. 2019When Does Label Smoothing Help?cited by 988
  8. 2021Ethical and social risks of harm from Language Modelscited by 966
  9. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  11. 2023Whose Opinions Do Language Models Reflect?cited by 773
  12. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 759
37 more
  1. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  2. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  3. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  4. 2024HelpSteer2: Open-source dataset for training top-performing reward modelscited by 236
  5. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 199
  6. 2024SOAP: Improving and Stabilizing Shampoo using Adamcited by 192
  7. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  8. 2024A Roadmap to Pluralistic Alignmentcited by 184
  9. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 171
  10. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 155
  11. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 144
  12. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 133
  13. 2021On Releasing Annotator-Level Labels and Information in Datasetscited by 126
  14. 2022Evaluating and Inducing Personality in Pre-trained Language Modelscited by 123
  15. 2020A Distributional Approach to Controlled Text Generationcited by 119
  16. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 113
  17. 2024Survey of Cultural Awareness in Language Models: Text and Beyondcited by 103
  18. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 100
  19. 2023Aligning Language Models to User Opinionscited by 99
  20. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 87
  21. 2023DICES Dataset: Diversity in Conversational AI Evaluation for Safetycited by 86
  22. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 74
  23. 2024PERSONA: A Reproducible Testbed for Pluralistic Alignmentcited by 73
  24. 2024PersonalLLM: Tailoring LLMs to Individual Preferencescited by 60
  25. 2025Reinforcement Learning from Human Feedbackcited by 58
  26. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  27. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Valuescited by 54
  28. 2025Randomness, Not Representation: The Unreliability of Evaluating Cultural Alignment in LLMscited by 54
  29. 2024Axioms for AI Alignment from Human Feedbackcited by 52
  30. 2025Evaluating the Diversity and Quality of LLM Generated Contentcited by 51
  31. 2023Generative Social Choicecited by 43
  32. 2023The History and Risks of Reinforcement Learning and Human Feedbackcited by 42
  33. 2023AI Alignment and Social Choice: Fundamental Limitations and Policy Implicationscited by 36
  34. 2025Value Profiles for Encoding Human Variationcited by 29
  35. 2024Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedbackcited by 26
  36. 2024Adaptive Preference Scaling for Reinforcement Learning with Human Feedbackcited by 20
  37. 2024Cultural Palette: Pluralising Culture Alignment via Multi-agent Palettecited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 12.

14 papers.

Sort:

By area:

  1. 2026How RLHF Amplifies Sycophancycited by 23
  2. 2025Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Frameworkcited by 4
  3. 2026AI Alignment From Social Choice Perspectivescited by 2
  4. 2025When Distance Distracts: Representation Distance Bias in BT-Loss for Reward Modelscited by 1
  5. 2026Robust AI Evaluation through Maximal Lotteriescited by 1
  6. 2026From Weights to Words: Expressing and Editing Preference Model Inferences in Natural Languagecited by 1
  7. 2026SNAP: A Self-Consistent Agreement Principle with Application to Robust Computationcited by 0
  8. 2026Back to Blackwell: Closing the Loop on Intransitivity in Multi-Objective Preference Fine-Tuningcited by 0
  9. 2026Variance-aware Reward Modeling with Anchor Guidancecited by 0
  10. 2026Calibrated Preference Learning: The Case of Label Rankingcited by 0
  11. 2026Algorithmic Impact Reveals the Hidden Social Choice Structure of Alignmentcited by 0
  12. 2026Inference-Time Nash Alignmentcited by 0
2 more
  1. 2026Groupwise Distortion Guarantees for Preference-Based Alignmentcited by 0
  2. 2026Learning Individual Scientific Taste with Privileged Hindsight Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MixDPO: Modeling Preference Strength for Pluralistic Alignment
  2. 2026Who Should We Listen to More? Welfare-Aware Preference Acquisition for Pluralistic Alignment
  3. 2025Pluralistic Off-policy Evaluation and Alignment
  4. 2024MaxMin-RLHF: Alignment with Diverse Human Preferences
  5. 2025REWARD CONSISTENCY: Improving Multi-Objective Alignment from a Data-Centric Perspective
  6. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?
  7. 2025The Reward Model Selection Crisis in Personalized Alignment
  8. 2026Hidden Consensus:Preference-Validity Compression in Human Feedback