mlsanityData to 7 Oct 2026

LiPO: Listwise Preference Optimization through Learning-to-Rank

arXiv 2402.01878 ↗First posted 2 Feb 2024cs.CL cs.LGcited by 94

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Aligning language models (LMs) with curated human feedback is critical to control their behaviors in real-world applications. Several recent policy optimization methods, such as DPO and SLiC, serve as promising alternatives to the traditional Reinforcement Learning from Human Feedback (RLHF) approach. In practice, human feedback often comes in a format of a ranked list over multiple responses to amortize the cost of reading prompt. Multiple responses can also be ranked by reward models or AI feedback. There lacks such a thorough study on directly fitting upon a list of responses. In this work, we formulate the LM alignment as a textitlistwise ranking problem and describe the LiPO framework, where the policy can potentially learn more effectively from a ranked list of plausible responses given the prompt. This view draws an explicit connection to Learning-to-Rank (LTR), where most existing preference optimization work can be mapped to existing ranking objectives. Following this connection, we provide an examination of ranking objectives that are not well studied for LM alignment with DPO and SLiC as special cases when list size is two. In particular, we highlight a specific method, LiPO-λ, which leverages a state-of-the-art textitlistwise ranking objective and weights each preference pair in a more advanced manner. We show that LiPO-λ can outperform DPO variants and SLiC by a clear margin on several preference alignment tasks with both curated and real rankwise preference data.

Key facts. Cited by 94 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 21 of its 47 references that are in the map. First posted on arXiv 2 Feb 2024.

Builds on

21 of 47 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  2. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  3. 2023Preference Ranking Optimization for Human Alignment30 Jun
  4. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 16

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  7. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  8. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  9. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  10. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 302
  11. 2016Unbiased Learning-to-Rank with Biased Feedbackcited by 155
  12. 2013A Theoretical Analysis of NDCG Type Ranking Measurescited by 119
4 more
  1. 2015Simple, Robust and Optimal Ranking from Pairwise Comparisonscited by 68
  2. 2023RewriteLM: An Instruction-Tuned Large Language Model for Text Rewritingcited by 67
  3. 2020Non-Clicks Mean Irrelevant? Propensity Ratio Scoring As a Correctioncited by 10
  4. 2022Regression Compatible Listwise Objectives for Calibrated Ranking with Binary Relevancecited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 36; 2025: 35; 2026: 23.

94 papers.

Sort:

By area:

  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  2. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  3. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  4. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  5. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 75
  6. 2024Building Math Agents with Multi-Turn Iterative Preference Learningcited by 66
  7. 2024Learn Your Reference Model for Real Good Alignmentcited by 56
  8. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  9. 2024Curry-DPO: Enhancing Alignment using Curriculum Learning & Ranked Preferencescited by 39
  10. 2024Prompt Optimization with Human Feedbackcited by 35
  11. 2025A Survey of Direct Preference Optimizationcited by 35
  12. 2024Alignment of Diffusion Models: Fundamentals, Challenges, and Futurecited by 34
38 more
  1. 2024Scalable Ranked Preference Optimization for Text-to-Image Generationcited by 33
  2. 2024LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Modelscited by 29
  3. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimizationcited by 27
  4. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Rewardcited by 25
  5. 2024Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignmentcited by 24
  6. 2024Filtered Direct Preference Optimizationcited by 24
  7. 2025Reward Models in Deep Reinforcement Learning: A Surveycited by 24
  8. 2024A Survey on Human Preference Learning for Large Language Modelscited by 23
  9. 2025GRPO-λ: Credit Assignment improves LLM Reasoningcited by 21
  10. 2024f-PO: Generalizing Preference Optimization with f-divergence Minimizationcited by 19
  11. 2024FlashMask: Efficient and Rich Mask Extension of FlashAttentioncited by 18
  12. 2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPOcited by 18
  13. 2025MPO: Multilingual Safety Alignment via Reward Gap Optimizationcited by 18
  14. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  15. 2024Boost Your Human Image Generation Model via Direct Preference Optimizationcited by 16
  16. 2025Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimizationcited by 16
  17. 2024Balancing Enhancement, Harmlessness, and General Capabilities: Enhancing Conversational LLMs with Direct RLHFcited by 15
  18. 2024Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimizationcited by 15
  19. 2025Advancing LLM Safe Alignment with Safety Representation Rankingcited by 15
  20. 2024Offline Model-Based Optimization by Learning to Rankcited by 13
  21. 2025Perception in Reflectioncited by 13
  22. 2024Reward Modeling with Ordinal Feedback: Wisdom of the Crowdcited by 11
  23. 2025ComPO: Preference Alignment via Comparison Oraclescited by 11
  24. 2024Hummer: Towards Limited Competitive Preference Datasetcited by 10
  25. 2024ROSE: A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuningcited by 10
  26. 2025POLO: Preference-Guided Multi-Turn Reinforcement Learning for Lead Optimizationcited by 10
  27. 2025R-WoM: Retrieval-augmented World Model For Computer-use Agentscited by 10
  28. 2024Aligning Diffusion Models with Noise-Conditioned Perceptioncited by 9
  29. 2024SEER: Self-Aligned Evidence Extraction for Retrieval-Augmented Generationcited by 9
  30. 2025PerPO: Perceptual Preference Optimization via Discriminative Rewardingcited by 9
  31. 2025In-context Ranking Preference Optimizationcited by 9
  32. 2024Direct Preference Optimization for Suppressing Hallucinated Prior Exams in Radiology Report Generationcited by 7
  33. 2024Permutative Preference Alignment from Listwise Ranking of Human Judgmentscited by 7
  34. 2025LLM Alignment as Retriever Optimization: An Information Retrieval Perspectivecited by 7
  35. 2025Towards Better Optimization For Listwise Preference in Diffusion Modelscited by 7
  36. 2024Triple Preference Optimization: Achieving Better Alignment using a Single Step Optimizationcited by 6
  37. 2025Improving Protein Sequence Design through Designability Preference Optimizationcited by 6
  38. 2024LAMPO: Large Language Models as Preference Machines for Few-shot Ordinal Classificationcited by 5

50 most cited of 94 All 94 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model
  2. 2025Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models
  3. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  4. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  5. 2024Accelerated Preference Optimization for Large Language Model Alignment
  6. 2024LIRE: listwise reward enhancement for preference alignment
  7. 2025MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment
  8. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignment