mlsanityData to 14 Oct 2026

Beyond RLHF and NLHF: Population-Proportional Alignment under an Axiomatic Framework

arXiv 2506.05619 ↗First posted 5 Jun 2025cs.AI cs.LGcited by 4

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Conventional preference learning methods often prioritize opinions held more widely when aggregating preferences from multiple evaluators. This may result in policies that are biased in favor of some types of opinions or groups and susceptible to strategic manipulation. To address this issue, we develop a novel preference learning framework capable of aligning aggregate opinions and policies proportionally with the true population distribution of evaluator preferences. Grounded in social choice theory, our approach infers the feasible set of evaluator population distributions directly from pairwise comparison data. Using these estimates, the algorithm constructs a policy that satisfies foundational axioms from social choice theory, namely monotonicity and Pareto efficiency, as well as our newly-introduced axioms of population-proportional alignment and population-bounded manipulability. Moreover, we propose a soft-max relaxation method that smoothly trades off population-proportional alignment with the selection of the Condorcet winner (which beats all other options in pairwise comparisons). Finally, we validate the effectiveness and scalability of our approach through experiments on both tabular recommendation tasks and large language model alignment.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 57 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

42 of 57 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Pairwise Calibrated Rewards for Pluralistic Alignment17 May
  2. 2024No Preference Left Behind: Group Distributional Preference Optimization28 Dec
  3. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences12 Jun
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 38

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2024Qwen2.5 Technical Reportcited by 4,942
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2020Learning to summarize from human feedbackcited by 2,588
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  7. 2023Instruction Tuning with GPT-4cited by 686
  8. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  9. 2023Nash Learning from Human Feedbackcited by 224
  10. 2024A Roadmap to Pluralistic Alignmentcited by 184
  11. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 144
  12. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 133
26 more
  1. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 123
  2. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 100
  3. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 95
  4. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 74
  5. 2020Active Preference-Based Gaussian Process Regression for Reward Learningcited by 71
  6. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  7. 2024Axioms for AI Alignment from Human Feedbackcited by 52
  8. 2014Justified Representation in Approval-Based Committee Votingcited by 42
  9. 2020Proportional Participatory Budgeting with Additive Utilitiescited by 37
  10. 2023AI Alignment and Social Choice: Fundamental Limitations and Policy Implicationscited by 36
  11. 2024Provable Pluralistic Alignment: Multi-Party RLHF under Offline Human Feedbackcited by 32
  12. 2024Truthful Aggregation of LLMs with an Application to Online Advertisingcited by 29
  13. 2024Mapping Social Choice Theory to RLHFcited by 27
  14. 2020Participatory Budgeting: Models and Approachescited by 23
  15. 2022PrefRec: Recommender Systems with Human Preferences for Reinforcing Long-term User Engagementcited by 21
  16. 2016Learning Dynamic Robot-to-Human Object Handover from Human Feedbackcited by 19
  17. 2024Mechanism Design for LLM Fine-tuning with Multiple Reward Modelscited by 18
  18. 2025Jackpot! Alignment as a Maximal Lotterycited by 17
  19. 2022Optimized Distortion and Proportional Fairness in Votingcited by 12
  20. 2024Online Learning from Strategic Human Feedback in LLM Fine-Tuningcited by 10
  21. 2025Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matchingcited by 10
  22. 2017Proportionally Representative Participatory Budgeting: Axioms and Algorithmscited by 9
  23. 2019Proportionally Representative Participatory Budgeting with Ordinal Preferencescited by 9
  24. 2025Navigating the Social Welfare Frontier: Portfolios for Multi-objective Reinforcement Learningcited by 5
  25. 2025Strategyproof Reinforcement Learning from Human Feedbackcited by 5
  26. 2024A Unified Linear Programming Framework for Offline Reward Learning from Human Demonstrations and Feedbackcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026RVPO: Risk-Sensitive Alignment via Variance Regularizationcited by 3
  2. 2026AI Alignment From Social Choice Perspectivescited by 2
  3. 2026AI Alignment through a Game-theoretic Lens: A Surveycited by 0
  4. 2026Groupwise Distortion Guarantees for Preference-Based Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?
  2. 2025Semiparametric Preference Optimization: Your Language Model is Secretly a Single-Index Model
  3. 2026Democratic Preference Alignment via Sortition-Weighted RLHF
  4. 2026How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamics
  5. 2024Representative Social Choice: From Learning Theory to AI Alignment
  6. 2026Response Time Enhances Alignment with Heterogeneous Preferences
  7. 2025Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theory
  8. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences