mlsanityData to 14 Oct 2026

Granular feedback merits sophisticated aggregation

arXiv 2507.12041 ↗First posted 16 Jul 2025cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Human feedback is increasingly used across diverse applications like training AI models, developing recommender systems, and measuring public opinion -- with granular feedback often being preferred over binary feedback for its greater informativeness. While it is easy to accurately estimate a population's distribution of feedback given feedback from a large number of individuals, cost constraints typically necessitate using smaller groups. A simple method to approximate the population distribution is regularized averaging: compute the empirical distribution and regularize it toward a prior. Can we do better? As we will discuss, the answer to this question depends on feedback granularity. Suppose one wants to predict a population's distribution of feedback using feedback from a limited number of individuals. We show that, as feedback granularity increases, one can substantially improve upon predictions of regularized averaging by combining individuals' feedback in ways more sophisticated than regularized averaging. Our empirical analysis using questions on social attitudes confirms this pattern. In particular, with binary feedback, sophistication barely reduces the number of individuals required to attain a fixed level of performance. By contrast, with five-point feedback, sophisticated methods match the performance of regularized averaging with about half as many individuals.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 62 references that are in the map. First posted on arXiv 16 Jul 2025.

Builds on

22 of 62 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023A Survey of Reinforcement Learning from Human Feedback22 Dec
  2. 2023From "Thumbs Up" to "10 out of 10": Reconsidering Scalar Feedback in Interactive Reinforcement Learning17 Nov
  3. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  5. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  6. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 16

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2023Whose Opinions Do Language Models Reflect?cited by 773
  9. 2019Rank consistent ordinal regression for neural networks with application to age estimationcited by 133
  10. 2022Jury Learning: Integrating Dissenting Voices into Machine Learning Modelscited by 118
  11. 2020Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotescited by 101
  12. 2021Deep Neural Networks for Rank-Consistent Ordinal Regression Based On Conditional Probabilitiescited by 59
4 more
  1. 2021Learning Reward Functions from Scale Feedbackcited by 24
  2. 2022Knowledge Learning with Crowdsourcing: A Brief Review and Systematic Perspectivecited by 11
  3. 2013Inferring ground truth from multi-annotator ordinal data: a probabilistic approachcited by 4
  4. 2020GPM: A Generic Probabilistic Model to Recover Annotator's Behavior and Ground Truth Labelingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Reward Modeling with Ordinal Feedback: Wisdom of the Crowd
  2. 2026Learning a Ranking from Human Feedback in Log-Concave Random Utility Models
  3. 2026Consensus vs. Dissent: Dynamic LLM Modeling of Subjective Preferences in Group Recommenders
  4. 2024Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation
  5. 2024Naturally Occurring Feedback is Common, Extractable and Useful
  6. 2022Emergent Instabilities in Algorithmic Feedback Loops
  7. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values
  8. 2026Hidden Consensus:Preference-Validity Compression in Human Feedback