mlsanityData to 7 Oct 2026

MaxMin-RLHF: Alignment with Diverse Human Preferences

arXiv 2402.08925 ↗First posted 14 Feb 2024cs.CL cs.AI cs.LGcited by 121

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) aligns language models to human preferences by employing a singular reward model derived from preference data. However, such an approach overlooks the rich diversity of human preferences inherent in data collected from multiple users. In this work, we first derive an impossibility result of alignment with single reward RLHF, thereby highlighting its insufficiency in representing diverse human preferences. To provide an equitable solution to the problem, we learn a mixture of preference distributions via an expectation-maximization algorithm and propose a MaxMin alignment objective for policy learning inspired by the Egalitarian principle in social choice theory to better represent diverse human preferences. We elucidate the connection of our proposed approach to distributionally robust optimization and general utility RL, thereby highlighting the generality and robustness of our proposed solution. We present comprehensive experimental results on small-scale (GPT-2) and large-scale language models (with Tulu2-7B) and show the efficacy of the proposed approach in the presence of diversity among human preferences. Our algorithm achieves an average improvement of more than 16% in win-rates over conventional RLHF algorithms and improves the win-rate (accuracy) for minority groups by over 33% without compromising the performance of majority groups, showcasing the robustness and fairness of our approach. We remark that our findings in this work are not only limited to language models but also extend to reinforcement learning in general.

Key facts. Cited by 121 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 15 of its 45 references that are in the map. First posted on arXiv 14 Feb 2024.

Builds on

15 of 45 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  2. 2023Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problems24 Jul
  3. 2023Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism29 May
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  5. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 10

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Whose Opinions Do Language Models Reflect?cited by 759
  3. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  4. 2023Instruction Tuning with GPT-4cited by 676
  5. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  6. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  7. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  8. 2023Aligning Large Language Models with Human: A Surveycited by 31
  9. 2023Unified Off-Policy Learning to Rank: a Reinforcement Learning Perspectivecited by 4
  10. 2018Towards Coherent and Cohesive Long-form Text Generationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 24; 2025: 45; 2026: 52.

121 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  2. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 136
  3. 2024Modular Pluralism: Pluralistic Alignment via Multi-LLM Collaborationcited by 111
  4. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  5. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 71
  6. 2025Language Model Fine-Tuning on Scaled Survey Data for Predicting Distributions of Public Opinionscited by 64
  7. 2024Aligning Language Models to Explicitly Handle Ambiguitycited by 55
  8. 2025Co-Evolving LLM Coder and Unit Tester via Reinforcement Learningcited by 51
  9. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
  10. 2024Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Modelscited by 27
  11. 2025Diverse Preference Learning for Capabilities and Alignmentcited by 25
  12. 2024One-Shot Safety Alignment for Large Language Models via Optimal Dualizationcited by 24
38 more
  1. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  2. 2025Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibriumcited by 22
  3. 2025BED-LLM: Intelligent Information Gathering with LLMs and Bayesian Experimental Designcited by 21
  4. 2025Distortion of AI Alignment: Does Preference Optimization Optimize for Preferences?cited by 20
  5. 2024Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioningcited by 18
  6. 2025Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personascited by 17
  7. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimizationcited by 17
  8. 2025Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimizationcited by 16
  9. 2025PluralLLM: Pluralistic Alignment in LLMs via Federated Learningcited by 14
  10. 2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hackingcited by 13
  11. 2024Reward Modeling with Ordinal Feedback: Wisdom of the Crowdcited by 11
  12. 2025Whose View of Safety? A Deep DIVE Dataset for Pluralistic Alignment of Text-to-Image Modelscited by 11
  13. 2025MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignmentcited by 10
  14. 2025Embedding-to-Prefix: Parameter-Efficient Personalization for Pre-Trained Large Language Modelscited by 10
  15. 2026Where does output diversity collapse in post-training?cited by 10
  16. 2024Fairness in Reinforcement Learning: A Surveycited by 9
  17. 2024Does Cross-Cultural Alignment Change the Commonsense Morality of Language Models?cited by 9
  18. 2025When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learningcited by 9
  19. 2025SafeSearch: Do Not Trade Safety for Utility in LLM Search Agentscited by 9
  20. 2024Pareto-Optimal Learning from Preferences with Hidden Contextcited by 8
  21. 2025Alignment of large language models with constrained learningcited by 8
  22. 2025Operationalizing Pluralistic Values in Large Language Model Alignment Reveals Trade-offs in Safety, Inclusivity, and Model Behaviorcited by 8
  23. 2024Can DPO Learn Diverse Human Values? A Theoretical Scaling Lawcited by 7
  24. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
  25. 2025Benchmarking Overton Pluralism in LLMscited by 7
  26. 2026Agentic AI Scientists Are Not Built For Autonomous Scientific Discoverycited by 7
  27. 2025Toward Scientific Reasoning in LLMs: Training from Expert Discussions via Reinforcement Learningcited by 6
  28. 2024Challenges and Future Directions of Data-Centric AI Alignmentcited by 5
  29. 2024Simultaneous Reward Distillation and Preference Learning: Get You a Language Model Who Can Do Bothcited by 5
  30. 2025Navigating the Social Welfare Frontier: Portfolios for Multi-objective Reinforcement Learningcited by 5
  31. 2025Strategyproof Reinforcement Learning from Human Feedbackcited by 5
  32. 2025Shallow Preference Signals: Large Language Model Aligns Even Better with Truncated Data?cited by 5
  33. 2025Do Large Language Models (Really) Need Statistical Foundations?cited by 5
  34. 2025Distribution Shift Alignment Helps LLMs Simulate Survey Response Distributionscited by 5
  35. 2024Value Alignment from Unstructured Textcited by 4
  36. 2025Scalable Valuation of Human Feedback through Provably Robust Model Alignmentcited by 4
  37. 2026Steering LLMs for Culturally Localized Generationcited by 4
  38. 2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4

50 most cited of 121 All 121 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  2. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  3. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization
  4. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning
  5. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  6. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  7. 2023Contrastive Preference Learning: Learning from Human Feedback without RL
  8. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models