mlsanityData to 14 Oct 2026

Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts

arXiv 2412.04628 ↗First posted 5 Dec 2024cs.LG cs.AI cs.CLcited by 9

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimization (DPO) has become a popular approach for aligning language models using pairwise preferences. However, in practical post-training pipelines, on-policy generation typically yields multiple candidate responses per prompt, which are scored by a reward model to guide learning. In this setting, we propose Multi-Preference Optimization (MPO), a generalization of DPO that optimizes over entire sets of responses by extending the Bradley-Terry model to groupwise comparisons between chosen and rejected sets. To further enhance learning, MPO employs deviation-based weighting, which emphasizes outlier responses that deviate most from the mean reward, effectively inducing a self-paced curriculum. We theoretically prove that MPO reduces alignment bias at a rate of Oleft(frac1√nright) with respect to the number of responses per query. Empirically, MPO achieves state-of-the-art performance on the UltraFeedback benchmark and yields up to ~ 17.5% improvement over the state-of-the-art baseline in length-controlled win rate on AlpacaEval2, establishing a new baseline for preference-based alignment

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 38 of its 50 references that are in the map. First posted on arXiv 5 Dec 2024.

Builds on

38 of 50 references.

Closest ancestors · 16

Same direction, newest first

  1. 2024Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Level17 Jun
  2. 2024Online DPO: Online Direct Preference Optimization with Fast-Slow Chasing8 Jun
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  4. 2024Token-level Direct Preference Optimization18 Apr
  5. 2024sDPO: Don't Use Your Data All at Once28 Mar
  6. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  7. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  8. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  9. 2024Noise Contrastive Alignment of Language Models with Explicit Rewards8 Feb
  10. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  11. 2023Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game14 Nov
  12. 2023Zephyr: Direct Distillation of LM Alignment25 Oct
4 more
  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  2. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  3. 2023Preference Ranking Optimization for Human Alignment30 Jun
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 22

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2017Deep reinforcement learning from human preferencescited by 3,912
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2015Trust Region Policy Optimizationcited by 2,580
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  10. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  11. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  12. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
10 more
  1. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  2. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  3. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  4. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  5. 2024Iterative Reasoning Preference Optimizationcited by 246
  6. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  7. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
  8. 2024On Softmax Direct Preference Optimization for Recommendationcited by 64
  9. 2024Following Length Constraints in Instructionscited by 46
  10. 2024Contrastive Perplexity for Controlled Generation: An Application in Detoxifying Large Language Modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 6.

9 papers.

  1. 2025M2PO: Multi-Perspective Multi-Pair Preference Optimization for Machine Translationcited by 4
  2. 2025TB or Not TB: Coverage-Driven Direct Preference Optimization for Verilog Stimulus Generationcited by 4
  3. 2026Direct Preference Optimization with Rating Information: Practical Algorithms and Provable Gainscited by 2
  4. 2026Beyond Pairs: Your Language Model is Secretly Optimizing a Preference Graphcited by 2
  5. 2024REFA: Reference Free Alignment for multi-preference optimizationcited by 0
  6. 2026Provably avoiding over-optimization in Direct Preference Optimization without knowing the data distributioncited by 0
  7. 2026Learning to Rank Caption Chains for Video-Text Alignmentcited by 0
  8. 2026GroupDPO: Memory-Efficient Group-Wise Direct Preference Optimizationcited by 0
  9. 2026SportsGrounder: Proposal-Aided Interleaved Grounding for Dense Sports Video Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Direct Preference Optimization with an Offset
  2. 2024Triple Preference Optimization: Achieving Better Alignment using a Single Step Optimization
  3. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization
  4. 2024Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization
  5. 2025BPO: Revisiting Preference Modeling in Direct Preference Optimization
  6. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO
  7. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts
  8. 2025Less is More: Improving LLM Alignment via Preference Data Selection