mlsanityData to 7 Oct 2026

Direct Alignment with Heterogeneous Preferences

arXiv 2502.16320 ↗First posted 22 Feb 2025cs.AI cs.LGcited by 19

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Alignment with human preferences is commonly framed using a universal reward function, even though human preferences are inherently heterogeneous. We formalize this heterogeneity by introducing user types and examine the limits of the homogeneity assumption. We show that aligning to heterogeneous preferences with a single policy is best achieved using the average reward across user types. However, this requires additional information about annotators. We examine improvements under different information settings, focusing on direct alignment methods. We find that minimal information can yield first-order improvements, while full feedback from each user type leads to consistent learning of the optimal policy. Surprisingly, however, no sample-efficient consistent direct loss exists in this latter setting. These results reveal a fundamental tension between consistency and sample efficiency in direct policy alignment.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 57 of its 83 references that are in the map. First posted on arXiv 22 Feb 2025.

Builds on

57 of 83 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Clone-Robust AI Alignment16 Jan
  2. 2024PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences12 Jun
  3. 2024Preference Learning Algorithms Do Not Learn Preference Rankings29 May
  4. 2024MallowsPO: Fine-Tune Your LLM with Preference Dispersions23 May
  5. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  6. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  7. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
  8. 2024Aligning Crowd Feedback via Distributional Preference Reward Modeling15 Feb
  9. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  10. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  11. 2023On Diversified Preferences of Large Language Model Alignment12 Dec
  12. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
2 more
  1. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization5 Oct
  2. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 43

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  7. 2023Whose Opinions Do Language Models Reflect?cited by 759
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  9. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  10. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  11. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  12. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 193
31 more
  1. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  2. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  3. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  4. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 141
  5. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  6. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  7. 2024Personalized Language Modeling from Personalized Human Feedbackcited by 124
  8. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 95
  9. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  10. 2024How to Evaluate Reward Models for RLHFcited by 84
  11. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 71
  12. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  13. 2024PersonalLLM: Tailoring LLMs to Individual Preferencescited by 56
  14. 2021Whose Ground Truth? Accounting for Individual and Collective Identities Underlying Dataset Annotationcited by 51
  15. 2024Axioms for AI Alignment from Human Feedbackcited by 50
  16. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?cited by 47
  17. 2024Can Language Models Reason about Individualistic Human Values and Preferences?cited by 34
  18. 2024Provable Pluralistic Alignment: Multi-Party RLHF under Offline Human Feedbackcited by 31
  19. 2022The Challenge of Understanding What Users Want: Inconsistent Preferences and Engagement Optimizationcited by 28
  20. 2024Mapping Social Choice Theory to RLHFcited by 26
  21. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Modelscited by 23
  22. 2023A density estimation perspective on learning from pairwise human preferencescited by 23
  23. 2025Personalized Preference Fine-tuning of Diffusion Modelscited by 22
  24. 2025Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personascited by 17
  25. 2021Distortion in Social Choice Problems: The First 15 Years and Beyondcited by 16
  26. 2024Pareto-Optimal Learning from Preferences with Hidden Contextcited by 8
  27. 2024Personalized Adaptation via In-Context Preference Learningcited by 8
  28. 2024Representative Social Choice: From Learning Theory to AI Alignmentcited by 8
  29. 2024Policy Aggregationcited by 8
  30. 2022On the identifiability of mixtures of ranking modelscited by 7
  31. 2024Adaptive Alignment: Dynamic Preference Adjustments via Multi-Objective Reinforcement Learning for Pluralistic AIcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 14.

19 papers.

Sort:

By area:

  1. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?cited by 20
  2. 2025Pairwise Calibrated Rewards for Pluralistic Alignmentcited by 12
  3. 2025Moral Change or Noise? On Problems of Aligning AI With Temporally Unstable Human Feedbackcited by 5
  4. 2025Emergent Alignment via Competitioncited by 3
  5. 2025The Limits of Preference Data for Post-Trainingcited by 1
  6. 2026Power and Limitations of Aggregation in Compound AI Systemscited by 1
  7. 2026Personalized Group Relative Policy Optimization for Heterogenous Preference Alignmentcited by 1
  8. 2026Linear Social Choice with Few Queries: A Moment-Based Approachcited by 1
  9. 2026SNAP: A Self-Consistent Agreement Principle with Application to Robust Computationcited by 0
  10. 2026Open Problems in Differentiable Social Choice: Learning Mechanisms, Decisions, and Alignmentcited by 0
  11. 2026Back to Blackwell: Closing the Loop on Intransitivity in Multi-Objective Preference Fine-Tuningcited by 0
  12. 2026When to Ask a Question: Understanding Communication Strategies in Generative AI Toolscited by 0
7 more
  1. 2026The End Justifies the Mean: A Linear Ranking Rule for Proportional Sequential Decisionscited by 0
  2. 2026Federated Variational Preference Alignment with Gumbel-Softmax Prior for Personalized User Preferencescited by 0
  3. 2026PAFO: Pareto Fairness Optimization for Personalized Reward Modelingcited by 0
  4. 2026AI Alignment From Social Choice Perspectivescited by 0
  5. 2026Algorithmic Impact Reveals the Hidden Social Choice Structure of Alignmentcited by 0
  6. 2026Inference-Time Nash Alignmentcited by 0
  7. 2026Multiple latent orderings better predict language model preferencescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MixDPO: Modeling Preference Strength for Pluralistic Alignment
  2. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  3. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  4. 2026Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspective
  5. 2026Response Time Enhances Alignment with Heterogeneous Preferences
  6. 2025The Reward Model Selection Crisis in Personalized Alignment
  7. 2025The Alignment Game: A Theory of Long-Horizon Alignment Through Recursive Curation
  8. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment