mlsanityData to 14 Oct 2026

MoPLEx: Estimating Plackett-Luce Mixture Models for Multi-Objective Alignment

arXiv 2608.25200 ↗First posted 25 Aug 2026cs.LG cs.AI cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

We study learning a mixture of k Plackett-Luce models from multi-way ranking responses from annotators that may represent heterogeneous underlying preferences. This problem has many applications in AI alignment and preference optimization. Prior work has studied mixtures of Bradley-Terry models from pairwise comparisons. However, estimating a mixture of multi-way ranking models can become theoretically unidentifiable when k exceeds m/2, where m is the ranking length. We design an efficient algorithm to address this issue by first augmenting the rankings to a larger size (e.g., generating comparisons from a base model), followed by a gradient-based estimation to reduce inference cost (in the input embedding space). With this procedure in mind, we then fit a mixture of Plackett-Luce (PL) models via an expectation-maximization-style iteration, or MoPLEx in short. We conduct extensive experiments to verify this algorithm. First, we find that the gradient-based approximation estimates true probabilities with less than 5% error on models with up to 34 billion parameters. Second, MoPLEx improves clustering and ranking accuracy by an average of 43.7% and 15.2% over baselines using a single PL model or a mixture of Bradley-Terry models, on UltraFeedback and PERSONA datasets. These results demonstrate the effectiveness of MoPLEx for tackling multi-way rankings following heterogeneous preferences through measuring alignment via gradients.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 21 of its 35 references that are in the map. First posted on arXiv 25 Aug 2026.

Builds on

21 of 35 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Permutative Preference Alignment from Listwise Ranking of Human Judgments6 Oct
  2. 2024Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences23 May
  3. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewards28 Feb

Other references · 18

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  3. 2024MaxMin-RLHF: Alignment with Diverse Human Preferencescited by 123
  4. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 100
  5. 2020Precise High-Dimensional Asymptotics for Quantifying Heterogeneous Transferscited by 27
  6. 2024Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignmentcited by 27
  7. 2024Scalable Multitask Learning Using Gradient-based Estimation of Task Affinitycited by 20
  8. 2016Learning Mixtures of Plackett-Luce Modelscited by 19
  9. 2023Boosting Multitask Learning on Graphs through Higher-Order Task Affinitiescited by 15
  10. 2018Efficiently Learning Mixtures of Mallows Modelscited by 13
  11. 2024Scalable Fine-tuning from Multiple Data Sources: A First-Order Approximation Approachcited by 13
  12. 2025MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learningcited by 11
6 more
  1. 2025Efficient Ensemble for Fine-tuning Language Models on Multiple Datasetscited by 10
  2. 2022Influence Functions for Sequence Tagging Modelscited by 9
  3. 2023Learning Rich Rankingscited by 8
  4. 2025Linear-Time Demonstration Selection for In-Context Learning via Gradient Estimationcited by 8
  5. 2025Scalable Multi-Objective and Meta Reinforcement Learning via Gradient Estimationcited by 8
  6. 2024Statistical Models of Top-k Partial Orderscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Ranking Inferences Based on the Top Choice of Multiway Comparisons
  2. 2023Spectral Ranking Inferences based on General Multiway Comparisons
  3. 2026How Many Repeated Pairwise Comparisons Are Needed for Ranking under Heterogeneity?
  4. 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Models
  5. 2025Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modeling
  6. 2026Distributionally Robust Listwise Preference Optimization
  7. 2026From "Weak" Signals to Strong Models: Preference Delta Aggregation with LoRA Merging
  8. 2026Prompt-Dependent Ranking of Large Language Models with Uncertainty Quantification