mlsanityData to 14 Oct 2026

Reward Modeling with Ordinal Feedback: Wisdom of the Crowd

arXiv 2411.12843 ↗First posted 19 Nov 2024cs.LG cs.AI cs.CLcited by 11

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Learning a reward model (RM) from human preferences has been an important component in aligning large language models (LLMs). The canonical setup of learning RMs from pairwise preference data is rooted in the classic Bradley-Terry (BT) model that accepts binary feedback, i.e., the label being either Response 1 is better than Response 2, or the opposite. Such a setup inevitably discards potentially useful samples (such as "tied" between the two responses) and loses more fine-grained information (such as "slightly better"). In this paper, we propose a framework for learning RMs under ordinal feedback which generalizes the case of binary preference feedback to any arbitrary granularity. Specifically, we first identify a marginal unbiasedness condition, which generalizes the assumption of the BT model in the existing binary feedback setting. The condition validates itself via the sociological concept of the wisdom of the crowd. Under the condition, we develop a natural probability model for pairwise preference data under ordinal feedback and analyze its properties. We prove the statistical benefits of ordinal feedback in terms of reducing the Rademacher complexity compared to the case of binary feedback. The proposed learning objective and the theory also extend to hinge loss and direct policy optimization (DPO). In particular, the theoretical analysis may be of independent interest when applying to a seemingly unrelated problem of knowledge distillation to interpret the bias-variance trade-off therein. The framework also sheds light on writing guidance for human annotators. Our numerical experiments validate that fine-grained feedback leads to better reward learning for both in-distribution and out-of-distribution settings. Further experiments show that incorporating a certain proportion of samples with tied preference boosts RM learning.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 47 references that are in the map. First posted on arXiv 19 Nov 2024.

Builds on

34 of 47 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  2. 2024Reward Learning From Preference With Ties5 Oct
  3. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  4. 2024HelpSteer2: Open-source dataset for training top-performing reward models12 Jun
  5. 2024MaxMin-RLHF: Alignment with Diverse Human Preferences14 Feb
  6. 2023A Survey of Reinforcement Learning from Human Feedback22 Dec
  7. 2023AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model3 Oct
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  9. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  10. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  12. 2019Fine-Tuning Language Models from Human Preferences18 Sep
1 more
  1. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  5. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  6. 2019When Does Label Smoothing Help?cited by 988
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  8. 2013Do Deep Nets Really Need to be Deep?cited by 474
  9. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 394
  10. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  11. 2023Preference Ranking Optimization for Human Alignmentcited by 285
  12. 2023AI Alignment: A Comprehensive Surveycited by 263
9 more
  1. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  2. 2021Towards Understanding Knowledge Distillationcited by 158
  3. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 148
  4. 2024Direct Preference Optimization with an Offsetcited by 122
  5. 2021Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspectivecited by 119
  6. 2024LiPO: Listwise Preference Optimization through Learning-to-Rankcited by 96
  7. 2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
  8. 2023On Diversified Preferences of Large Language Model Alignmentcited by 26
  9. 2024On Extending Direct Preference Optimization to Accommodate Tiescited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 7.

11 papers.

  1. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 37
  2. 2025Reward Models in Deep Reinforcement Learning: A Surveycited by 25
  3. 2026Beyond Binary Preferences: A Principled Framework for Reward Modeling with Ordinal Feedbackcited by 3
  4. 2026From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluationcited by 2
  5. 2026JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluationcited by 2
  6. 2026Learning Ordinal Probabilistic Reward from Preferencescited by 1
  7. 2025Minority Reports: Balancing Cost and Quality in Ground Truth Data Annotationcited by 0
  8. 2025Learning Guarantee of Reward Modeling Using Deep Neural Networkscited by 0
  9. 2026Reward Learning through Ranking Mean Squared Errorcited by 0
  10. 2026Variance-aware Reward Modeling with Anchor Guidancecited by 0
  11. 2026From Soft Targets to Reward Signals: How Assignment and Reward Objectives Interactcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023A density estimation perspective on learning from pairwise human preferences
  2. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
  3. 2025Capturing Individual Human Preferences with Reward Features
  4. 2024Beyond the Binary: Capturing Diverse Preferences With Reward Regularization
  5. 2024Transforming and Combining Rewards for Aligning Large Language Models
  6. 2026Preference learning in shades of gray: Interpretable and bias-aware reward modeling for human preferences
  7. 2026VRM: Teaching Reward Models to Understand Authentic Human Preferences
  8. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values