mlsanityData to 7 Oct 2026

What Does Preference Learning Recover from Pairwise Comparison Data?

arXiv 2602.10286 ↗First posted 10 Feb 2026cs.LGcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Pairwise preference learning is central to machine learning, with recent applications in aligning language models with human preferences. A typical dataset consists of triplets (x, y^+, y^-), where response y^+ is preferred over response y^- for context x. The Bradley--Terry (BT) model is the predominant approach, modeling preference probabilities as a function of latent score differences. Standard practice assumes data follows this model and learns the latent scores accordingly. However, real data may violate this assumption, and it remains unclear what BT learning recovers in such cases. Starting from triplet comparison data, we formalize the preference information it encodes through the conditional preference distribution (CPRD). We give precise conditions for when BT is appropriate for modeling the CPRD, and identify factors governing sample efficiency -- namely, margin and connectivity. Together, these results offer a data-centric foundation for understanding what preference learning actually recovers.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 80 references that are in the map. First posted on arXiv 10 Feb 2026.

Builds on

39 of 80 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy2 Jul
  2. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspective19 Mar
  3. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style21 Oct
  4. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignment13 Oct
  5. 2024Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment3 Oct
  6. 2024Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparison15 Sep
  7. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  8. 2024Active Preference Optimization for Sample Efficient RLHF16 Feb
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  10. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  11. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
1 more
  1. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  7. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  8. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  9. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 256
  10. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  11. 2015Contextual Dueling Banditscited by 101
  12. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learningcited by 84
14 more
  1. 2015Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependencecited by 59
  2. 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48
  3. 2015Spectral MLE: Top-K Rank Aggregation from Pairwise Comparisonscited by 34
  4. 2015Sparse Dueling Banditscited by 29
  5. 2021Uncertainty quantification in the Bradley-Terry-Luce modelcited by 28
  6. 2014Minimax-optimal Inference from Partial Rankingscited by 22
  7. 2018Approximate Ranking from Pairwise Comparisonscited by 21
  8. 2022Uncertainty Quantification of MLE for Entity Ranking with Covariatescited by 16
  9. 2016An application of incomplete pairwise comparison matrices for ranking top tennis playerscited by 10
  10. 2021ℓ∞-Bounds of the MLE in the BTL Model under General Comparison Graphscited by 8
  11. 2016Learning Combinatorial Functions from Pairwise Comparisonscited by 7
  12. 2020Preference Modeling with Context-Dependent Salient Featurescited by 7
  13. 2025Towards Understanding Valuable Preference Data for Large Language Model Alignmentcited by 4
  14. 2025On the Role of Preference Variance in Preference Optimizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Reward Learning from Best-of-N Preference Data: Targets, Tradeoffs, and Design Principlescited by 0
  2. 2026Sorting from Counterexamplescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Reward Learning From Preference With Ties
  2. 2025Stable Preference Optimization: A Bilevel Approach to Catastrophic Preference Shift
  3. 2025Generalizing while preserving monotonicity in comparison-based preference learning models
  4. 2025When Less Is More: Binary Feedback Can Outperform Ordinal Comparisons in Ranking Recovery
  5. 2024Reward Modeling with Ordinal Feedback: Wisdom of the Crowd
  6. 2025DCRM: A Heuristic to Measure Response Pair Quality in Preference Optimization
  7. 2025ResponseRank: Data-Efficient Reward Modeling through Preference Strength Learning
  8. 2024Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization