mlsanityData to 14 Oct 2026

MallowsPO: Fine-Tune Your LLM with Preference Dispersions

arXiv 2405.14953 ↗First posted 23 May 2024cs.LG cs.AI stat.MLcited by 18

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimization (DPO) has recently emerged as a popular approach to improve reinforcement learning with human feedback (RLHF), leading to better techniques to fine-tune large language models (LLM). A weakness of DPO, however, lies in its lack of capability to characterize the diversity of human preferences. Inspired by Mallows' theory of preference ranking, we develop in this paper a new approach, the MallowsPO. A distinct feature of this approach is a dispersion index, which reflects the dispersion of human preference to prompts. We show that existing DPO models can be reduced to special cases of this dispersion index, thus unified with MallowsPO. More importantly, we demonstrate (empirically) how to use this dispersion index to enhance the performance of DPO in a broad array of benchmark tasks, from synthetic bandit selection to controllable generations and dialogues, while maintaining great generalization capabilities. MallowsPO is also compatible with other SOTA offline preference optimization methods, boosting nearly 2% extra LC win rate when used as a plugin for fine-tuning Llama3-Instruct.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 37 of its 57 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

37 of 57 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Human Alignment of Large Language Models through Online Preference Optimisation13 Mar
  2. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  3. 2024Direct Preference Optimization with an Offset16 Feb
  4. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  5. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  6. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 217 Nov
  7. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  9. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  10. 2023Preference Ranking Optimization for Human Alignment30 Jun
  11. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  9. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  10. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  11. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  12. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
14 more
  1. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  2. 2016Deep Reinforcement Learning for Dialogue Generationcited by 288
  3. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  4. 2017Automated Curriculum Learning for Neural Networkscited by 257
  5. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 229
  6. 2023Nash Learning from Human Feedbackcited by 224
  7. 2014Predictive Entropy Search for Efficient Global Optimization of Black-box Functionscited by 204
  8. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  9. 2024Secrets of RLHF in Large Language Models Part II: Reward Modelingcited by 171
  10. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Modelscited by 119
  11. 2018Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networkscited by 114
  12. 2022There Are a Thousand Hamlets in a Thousand People's Eyes: Enhancing Knowledge-grounded Dialogue with Personal Memorycited by 16
  13. 2020Learning Mixtures of Permutations: Groups of Pairwise Comparisons and Combinatorial Method of Momentscited by 6
  14. 2022Distribution Aware Metrics for Conditional Natural Language Generationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 9; 2026: 6.

18 papers.

Sort:

By area:

  1. 2025A Survey of Direct Preference Optimizationcited by 36
  2. 2025TokUR: Token-Level Uncertainty Estimation for Large Language Model Reasoningcited by 24
  3. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  4. 2024RainbowPO: A Unified Framework for Combining Improvements in Preference Optimizationcited by 19
  5. 2025Direct Alignment with Heterogeneous Preferencescited by 19
  6. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPOcited by 19
  7. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  8. 2025Pairwise Calibrated Rewards for Pluralistic Alignmentcited by 14
  9. 2025ComPO: Preference Alignment via Comparison Oraclescited by 13
  10. 2025Learning to summarize user information for personalized reinforcement learning from human feedbackcited by 10
  11. 2026An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shiftcited by 2
  12. 2025Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modelingcited by 1
6 more
  1. 2026MixDPO: Modeling Preference Strength for Pluralistic Alignmentcited by 1
  2. 2026Steerable Cultural Preference Optimization of Reward Modelscited by 1
  3. 2025Mallows Model with Learned Distance Metrics: Sampling and Maximum Likelihood Estimationcited by 0
  4. 2026VisualDeltas: Learning Preferences from Visual Quality Perturbationscited by 0
  5. 2026A Zeroth-Order Paradigm for LLM Preference Alignmentcited by 0
  6. 2026BA-DPO: Bias-Adjusted Direct Preference Optimization for Language Model Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Uni-DPO: A Unified Paradigm for Dynamic Preference Optimization of LLMs
  2. 2024New Desiderata for Direct Preference Optimization
  3. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications
  4. 2025Pre-DPO: Improving Data Utilization in Direct Preference Optimization Using a Guiding Reference Model
  5. 2024Minor DPO reject penalty to increase training robustness
  6. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  7. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
  8. 2025When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasets