mlsanityData to 7 Oct 2026

Beyond Ordinal Preferences: Why Alignment Needs Cardinal Human Feedback

arXiv 2508.08486 ↗First posted 11 Aug 2025cs.AIcited by 1

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Alignment techniques for LLMs rely on optimizing preference-based objectives -- where these preferences are typically elicited as ordinal, binary choices between responses. Recent work has focused on improving label quality or mitigating particular biases, but we identify a more fundamental limitation: these methods collect the wrong kind of data. We prove an impossibility result: no algorithm relying solely on ordinal comparisons can systematically recover the most preferred model. Intuitively, ordinal data lacks the information needed to resolve tradeoffs -- e.g., fixing a factual error on one prompt versus improving style on another. We show that selecting the optimal model requires recovering preferences over models (rather than just responses), which can only be identified given cardinal feedback about response quality. To address this, we collect and publicly release a dataset of 25,000 cardinal judgments using willingness-to-pay elicitations, a well-established tool from experimental economics. Empirically, we find that incorporating cardinal feedback into preference fine-tuning allows models to prioritize high-impact improvements and outperform ordinal-only methods on downstream benchmarks, such as Arena-Hard.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 17 of its 36 references that are in the map. First posted on arXiv 11 Aug 2025.

Builds on

17 of 36 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Disentangling Length from Quality in Direct Preference Optimization28 Mar
  2. 2024Direct Preference Optimization with an Offset16 Feb
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct

Other references · 14

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  5. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  6. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  7. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  8. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  9. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  10. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66
  11. 2021Learning Reward Functions from Scale Feedbackcited by 24
  12. 2022A Theory of Dynamic Benchmarkscited by 15
2 more
  1. 2024Challenges in Trustworthy Human Evaluation of Chatbotscited by 14
  2. 2025LLM Content Moderation and User Satisfaction: Evidence from Response Refusals in Chatbot Arenacited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment
  2. 2026MixDPO: Modeling Preference Strength for Pluralistic Alignment
  3. 2026How Sampling Shapes LLM Alignment: From One-Shot Optima to Iterative Dynamics
  4. 2026Response Time Enhances Alignment with Heterogeneous Preferences
  5. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
  6. 2025Pluralistic Off-policy Evaluation and Alignment
  7. 2025The Sign Estimator: LLM Alignment in the Face of Choice Heterogeneity
  8. 2024Dissecting Human and LLM Preferences