mlsanityData to 7 Oct 2026

MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment

arXiv 2502.18699 ↗First posted 25 Feb 2025cs.CL cs.LG stat.MEcited by 10

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) has shown promise in aligning large language models (LLMs). Yet its reliance on a singular reward model often overlooks the diversity of human preferences. Recent approaches address this limitation by leveraging multi-dimensional feedback to fine-tune corresponding reward models and train LLMs using reinforcement learning. However, the process is costly and unstable, especially given the competing and heterogeneous nature of human preferences. In this paper, we propose Mixing Preference Optimization (MPO), a post-processing framework for aggregating single-objective policies as an alternative to both multi-objective RLHF (MORLHF) and MaxMin-RLHF. MPO avoids alignment from scratch. Instead, it log-linearly combines existing policies into a unified one with the weight of each policy computed via a batch stochastic mirror descent. Empirical results demonstrate that MPO achieves balanced performance across diverse preferences, outperforming or matching existing models with significantly reduced computational costs.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 51 references that are in the map. First posted on arXiv 25 Feb 2025.

Builds on

40 of 51 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Decoding-Time Language Model Alignment with Multiple Objectives27 Jun
  2. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences23 May
  3. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb
  4. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment15 Feb
  5. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization5 Oct
  6. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  7. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  11. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  12. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
21 more
  1. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  2. 2016Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Lojasiewicz Conditioncited by 526
  3. 2023Aligning Text-to-Image Models using Human Feedbackcited by 418
  4. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  5. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  6. 2023Human Preference Score: Better Aligning Text-to-Image Models with Human Preferencecited by 284
  7. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 271
  8. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  9. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 231
  10. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  11. 2022TimeLMs: Diachronic Language Models from Twittercited by 129
  12. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 121
  13. 2019Deep Reinforcement Learning for Multi-objective Optimizationcited by 87
  14. 2024RLHF from Heterogeneous Feedback via Personalization and Preference Aggregationcited by 71
  15. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learningcited by 57
  16. 2020Biased Stochastic First-Order Methods for Conditional Stochastic Optimization and Applications in Meta Learningcited by 46
  17. 2020Understanding Learned Reward Functionscited by 37
  18. 2024Provable Pluralistic Alignment: Multi-Party RLHF under Offline Human Feedbackcited by 31
  19. 2019Sample Complexity of Sample Average Approximation for Conditional Stochastic Optimizationcited by 19
  20. 2024Multi-level Monte-Carlo Gradient Methods for Stochastic Optimization with Biased Oraclescited by 4
  21. 2024Landscape of Policy Optimization for Finite Horizon MDPs with General State and Actioncited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 27
  2. 2025Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weightingcited by 23
  3. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
  4. 2026Toward Personalized LLM-Powered Agents: Foundations, Evaluation, and Future Directionscited by 10
  5. 2026Token-Level LLM Collaboration via FusionRoutecited by 9
  6. 2025Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedbackcited by 7
  7. 2026Reinforcement Learning from Human Feedback: A Statistical Perspectivecited by 4
  8. 2026Beyond Compromise: Pareto-Lenient Consensus for Efficient Multi-Preference LLM Alignmentcited by 2
  9. 2026Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuningcited by 2
  10. 2026Towards Pareto-Optimal Tool-Integrated Agents with Pareto Ranking Policy Optimizationcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models
  2. 2025Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences
  3. 2024Hybrid Preference Optimization for Alignment: Provably Faster Convergence Rates by Combining Offline Preferences with Online Exploration
  4. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model
  5. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization
  6. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  7. 2025PEO: Improving Bi-Factorial Preference Alignment with Post-Training Policy Extrapolation
  8. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models