mlsanityData to 14 Oct 2026

Preference learning made easy: Everything should be understood through win rate

arXiv 2502.10505 ↗First posted 14 Feb 2025cs.LG cs.CL stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

Preference learning, or the task of aligning generative models to preference comparison data, has yet to reach the conceptual maturity of classification, density estimation, etc. To close this gap, this work presents a framework to understand preference learning starting from the sampling distribution of pairwise preference data. First, we prove that the only evaluation of a generative model that respects both preferences and prevalences in the data distribution is a form of win rate, justifying win rate as the focal point to understand preference learning. We then analyze preference learning methods as win rate optimization (WRO) or non-WRO. We present novel instances of WRO beyond existing examples (RLHF, NLHF) and identify two key theoretical benefits of all such methods. We prove that common non-WRO methods like DPO and SFT on preferred samples lack these properties and suggest ways to mitigate such theoretical limitations. We also show that WRO underperforms in practice due optimization difficulties and that optimization success predicts performance better than choices which affect the objective's solution. Our analysis highlights best practices for existing methods and provides recommendations for future research, guided by the principle that one should either align non-WRO methods more closely with WRO or improve the optimization of WRO objectives.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 45 of its 58 references that are in the map. First posted on arXiv 14 Feb 2025.

Builds on

45 of 58 references.

References · 45

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  8. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 721
  9. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  10. 2024Self-Rewarding Language Modelscited by 681
  11. 2024Large Language Models: A Surveycited by 525
  12. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
33 more
  1. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  2. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  3. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  4. 2013Black Box Variational Inferencecited by 319
  5. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  6. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  7. 2019Monte Carlo Gradient Estimation in Machine Learningcited by 283
  8. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 264
  9. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  10. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  11. 2023Nash Learning from Human Feedbackcited by 224
  12. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  13. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 178
  14. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 157
  15. 2023Self-Alignment with Instruction Backtranslationcited by 151
  16. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedbackcited by 144
  17. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 143
  18. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 133
  19. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 121
  20. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  21. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 91
  22. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimizationcited by 78
  23. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 67
  24. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  25. 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48
  26. 2021Calibration and Consistency of Adversarial Surrogate Lossescited by 47
  27. 2020Markovian Score Climbing: Variational Inference with KL(p||q)cited by 46
  28. 2023Vanishing Gradients in Reinforcement Finetuning of Language Modelscited by 27
  29. 2024Transforming and Combining Rewards for Aligning Large Language Modelscited by 25
  30. 2024A Survey on Human Preference Learning for Large Language Modelscited by 24
  31. 2024f-PO: Generalizing Preference Optimization with f-divergence Minimizationcited by 19
  32. 2023Enabling Language Models to Implicitly Learn Self-Improvementcited by 7
  33. 2021Quantile Filtered Imitation Learningcited by 6

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Datasetcited by 35
  2. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapsecited by 31
  3. 2025MACE: A Hybrid LLM Serving System with Colocated SLO-aware Continuous Retraining Alignmentcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Clear Preferences Leave Traces: Reference Model-Guided Sampling for Preference Learning
  2. 2025Preference Learning with Response Time: Robust Losses and Guarantees
  3. 2023A density estimation perspective on learning from pairwise human preferences
  4. 2024Preference Optimization with Multi-Sample Comparisons
  5. 2025DCRM: A Heuristic to Measure Response Pair Quality in Preference Optimization
  6. 2025Preference Optimization via Contrastive Divergence: Your Reward Model is Secretly an NLL Estimator
  7. 2025Stable Preference Optimization: A Bilevel Approach to Catastrophic Preference Shift
  8. 2025Finding the Sweet Spot: Preference Data Construction for Scaling Preference Optimization