The Limits of Preference Data for Post-Training
On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF
Abstract (arXiv)
Recent progress in strengthening the capabilities of large language models has stemmed from applying reinforcement learning to domains with automatically verifiable outcomes. A key question is whether we can similarly use RL to optimize for outcomes in domains where evaluating outcomes inherently requires human feedback; for example, in tasks like deep research and trip planning, outcome evaluation is qualitative and there are many possible degrees of success. One attractive and scalable modality for collecting human feedback is preference data: ordinal rankings (pairwise or k-wise) that indicate, for k given outcomes, which one is preferred. In this work, we study a critical roadblock: preference data fundamentally and significantly limits outcome-based optimization. Even with idealized preference data (infinite, noiseless, and online), the use of ordinal feedback can prevent obtaining even approximately optimal solutions. We formalize this impossibility using voting theory, drawing an analogy between how a model chooses to answer a query with how voters choose a candidate to elect. This indicates that grounded human scoring and algorithmic innovations are necessary for extending the success of RL post-training to domains demanding human feedback. We also explore why these limitations have disproportionately impacted RLHF when it comes to eliciting reasoning behaviors (e.g., backtracking) versus situations where RLHF has been historically successful (e.g., instruction-tuning and safety training), finding that the limitations of preference data primarily suppress RLHF's ability to elicit robust strategies -- a class that encompasses most reasoning behaviors.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 33 of its 44 references that are in the map. First posted on arXiv 26 May 2025.
Builds on
33 of 44 references.
Closest ancestors · 8
Same direction, newest first
- 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer26 May
- 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF29 Jan
- 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF13 Dec
- 2023Confronting Reward Model Overoptimization with Constrained RLHF6 Oct
- 2023Is RLHF More Difficult than Standard RL?25 Jun
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
- 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
- 2017Deep reinforcement learning from human preferences12 Jun
Other references · 25
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
- 2023Let's Verify Step by Stepcited by 3,598
- 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
- 2022Solving math word problems with process- and outcome-based feedbackcited by 871
- 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 373
- 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
- 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
- 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 265
- 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 239
13 more
- 2024Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learningcited by 238
- 2024Chain of Preference Optimization: Improving Chain-of-Thought Reasoning in LLMscited by 94
- 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
- 2025Trading Inference-Time Compute for Adversarial Robustnesscited by 60
- 2025Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verificationcited by 49
- 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48
- 2025Benchmarking Reasoning Robustness in Large Language Modelscited by 30
- 2025AI-Slop to AI-Polish? Aligning Language Models through Edit-Based Writing Rewards and Test-time Computationcited by 24
- 2025Finding Flawed Fictions: Evaluating Complex Reasoning in Language Models via Plot Hole Detectioncited by 24
- 2025Direct Alignment with Heterogeneous Preferencescited by 19
- 2021Distortion in Social Choice Problems: The First 15 Years and Beyondcited by 16
- 2019Peeking Behind the Ordinal Curtain: Improving Distortion via Cardinal Queriescited by 6
- 2020A Few Queries Go a Long Way: Information-Distortion Tradeoffs in Matchingcited by 3
Built on by
Citing papers per year: 2025: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2024Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparison
- 2024Towards Comprehensive Preference Data Collection for Reward Modeling
- 2024Data-Centric Human Preference with Rationales for Direct Preference Alignment
- 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
- 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
- 2026A Regret Minimization Framework on Preference Learning in Large Language Models
- 2026Data-dependent Exploration for Online Reinforcement Learning from Human Feedback
- 2025Data-adaptive Safety Rules for Training Reward Models