mlsanityData to 7 Oct 2026

Beyond Pairwise: Empowering LLM Alignment With Ranked Choice Modeling

arXiv 2510.23631 ↗First posted 24 Oct 2025cs.LG cs.AI stat.MEcited by 1

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Alignment of large language models (LLMs) has predominantly relied on pairwise preference optimization, where annotators select the better of two responses to a prompt. While simple, this approach overlooks the opportunity to learn from richer forms of human feedback, such as multiway comparisons and top-k rankings. We introduce Ranked Choice Preference Optimization (RCPO), a unified framework that bridges preference optimization with (ranked) choice modeling via maximum likelihood estimation. RCPO supports both utility-based and rank-based models, subsumes several pairwise methods (such as DPO and SimPO) as special cases, and provides principled training objectives for richer feedback formats. We instantiate this framework with two representative models (Multinomial Logit and Mallows-RMJ). Experiments on Llama-3-8B-Instruct, Gemma-2-9B-it, and Mistral-7B-Instruct across in-distribution and out-of-distribution settings show that RCPO consistently outperforms competitive baselines. RCPO shows that directly leveraging ranked preference data, combined with the right choice models, yields more effective alignment. It offers an extensible foundation for incorporating (ranked) choice modeling into LLM training.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 79 references that are in the map. First posted on arXiv 24 Oct 2025.

Builds on

42 of 79 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025AlphaPO: Reward Shape Matters for LLM Alignment7 Jan
  2. 2024Permutative Preference Alignment from Listwise Ranking of Human Judgments6 Oct
  3. 2024RainbowPO: A Unified Framework for Combining Improvements in Preference Optimization5 Oct
  4. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey17 Sep
  5. 2024MallowsPO: Fine-Tune Your LLM with Preference Dispersions23 May
  6. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  7. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  8. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  9. 2024LiPO: Listwise Preference Optimization through Learning-to-Rank2 Feb
  10. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  12. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
3 more
  1. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  2. 2023Preference Ranking Optimization for Human Alignment30 Jun
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  8. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  10. 2023StarCoder: may the source be with you!cited by 790
  11. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  12. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
15 more
  1. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  2. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  3. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  4. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  5. 2014Predictive Entropy Search for Efficient Global Optimization of Black-box Functionscited by 201
  6. 2024Collective Constitutional AI: Aligning a Language Model with Public Inputcited by 120
  7. 2024Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedbackcited by 95
  8. 2024On Softmax Direct Preference Optimization for Recommendationcited by 64
  9. 2024Axioms for AI Alignment from Human Feedbackcited by 50
  10. 2024Beyond Preferences in AI Alignmentcited by 46
  11. 2023AI Alignment and Social Choice: Fundamental Limitations and Policy Implicationscited by 35
  12. 2024Automated Multi-level Preference for MLLMscited by 29
  13. 2024Mapping Social Choice Theory to RLHFcited by 26
  14. 2025K-order Ranking Preference Optimization for Large Language Modelscited by 6
  15. 2022On A Mallows-type Model For (Ranked) Choicescited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Mediocrity is the key for LLM as a Judge Anchor Selectioncited by 6

Similar, not cited

Close in wording, no citation either way

  1. 2024Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts
  2. 2025ComPO: Preference Alignment via Comparison Oracles
  3. 2026A Zeroth-Order Paradigm for LLM Preference Alignment
  4. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  5. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  6. 2024Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization
  7. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  8. 2025Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization