mlsanityData to 7 Oct 2026

Verbosity Bias in Preference Labeling by Large Language Models

arXiv 2310.10076 ↗First posted 16 Oct 2023cs.CL cs.AIcited by 89

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

In recent years, Large Language Models (LLMs) have witnessed a remarkable surge in prevalence, altering the landscape of natural language processing and machine learning. One key factor in improving the performance of LLMs is alignment with humans achieved with Reinforcement Learning from Human Feedback (RLHF), as for many LLMs such as GPT-4, Bard, etc. In addition, recent studies are investigating the replacement of human feedback with feedback from other LLMs named Reinforcement Learning from AI Feedback (RLAIF). We examine the biases that come along with evaluating LLMs with other LLMs and take a closer look into verbosity bias -- a bias where LLMs sometimes prefer more verbose answers even if they have similar qualities. We see that in our problem setting, GPT-4 prefers longer answers more than humans. We also propose a metric to measure this bias.

Key facts. Cited by 89 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 10 of its 15 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

10 of 15 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 9

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2016Equality of Opportunity in Supervised Learningcited by 2,000
  7. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  8. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  9. 2023Embrace Divergence for Richer Insights: A Multi-document Summarization Benchmark and a Case Study on Summarizing Diverse Information from News Articlescited by 45

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 18; 2025: 30; 2026: 39.

89 papers.

Sort:

By area:

  1. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  2. 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
  3. 2024Self-Preference Bias in LLM-as-a-Judgecited by 248
  4. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 197
  5. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  6. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  7. 2024What Large Language Models Know and What People Think They Knowcited by 109
  8. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
  9. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 55
  10. 2024Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignmentcited by 46
  11. 2024RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMscited by 42
  12. 2024Optima: Optimizing Effectiveness and Efficiency for LLM-Based Multi-Agent Systemcited by 40
38 more
  1. 2025Do LLM Evaluators Prefer Themselves for a Reason?cited by 38
  2. 2025Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judgecited by 35
  3. 2025DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Modelscited by 32
  4. 2024Stepwise Alignment for Constrained Language Model Policy Optimizationcited by 30
  5. 2025Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluationcited by 23
  6. 2026RubricBench: Aligning Model-Generated Rubrics with Human Standardscited by 21
  7. 2026Large Language Model Reasoning Failurescited by 20
  8. 2024RevisEval: Improving LLM-as-a-Judge via Response-Adapted Referencescited by 19
  9. 2025BLEUBERI: BLEU is a surprisingly effective reward for instruction followingcited by 17
  10. 2025The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judgecited by 13
  11. 2026WS-GRPO: Weakly-Supervised Group-Relative Policy Optimization for Rollout-Efficient Reasoningcited by 12
  12. 2024Automating Thematic Analysis: How LLMs Analyse Controversial Topicscited by 10
  13. 2026CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modelingcited by 10
  14. 2024Modality-Specialized Synergizers for Interleaved Vision-Language Generalistscited by 9
  15. 2025Shaping Shared Languages: Human and Large Language Models' Inductive Biases in Emergent Communicationcited by 9
  16. 2025MindEval: Benchmarking Language Models on Multi-turn Mental Health Supportcited by 9
  17. 2023The Adoption and Efficacy of Large Language Models: Evidence From Consumer Complaints in the Financial Industrycited by 8
  18. 2025Societal Alignment Frameworks Can Improve LLM Alignmentcited by 8
  19. 2025From Prompting to Alignment: A Generative Framework for Query Recommendationcited by 7
  20. 2026Who Endorsed It? Measuring Authority Bias Across Expertise Levels in Language Modelscited by 7
  21. 2026Clarify Before You Draw: Proactive Agents for Robust Text-to-CAD Generationcited by 7
  22. 2026Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelinescited by 7
  23. 2025DEBATE, TRAIN, EVOLVE: Self Evolution of Language Model Reasoningcited by 6
  24. 2026The Impact of AI-Generated Text on the Internetcited by 6
  25. 2024Verbosity neq Veracity: Demystify Verbosity Compensation Behavior of Large Language Modelscited by 5
  26. 2024Mapping out the Space of Human Feedback for Reinforcement Learning: A Conceptual Frameworkcited by 5
  27. 2025From First Draft to Final Insight: A Multi-Agent Approach for Feedback Generationcited by 5
  28. 2026Do Chatbot LLMs Talk Too Much? The YapBench Benchmarkcited by 5
  29. 2025Toward Evaluative Thinking: Meta Policy Optimization with Evolving Reward Modelscited by 4
  30. 2025Zero Reinforcement Learning Towards General Domainscited by 4
  31. 2026Detecting and Suppressing Reward Hacking with Gradient Fingerprintscited by 4
  32. 2024ISR-DPO: Aligning Large Multimodal Models for Videos by Iterative Self-Retrospective DPOcited by 3
  33. 2025From Idea to Implementation: Evaluating the Influence of Large Language Models in Software Development -- An Opinion Papercited by 3
  34. 2025MSQA: Benchmarking LLMs on Graduate-Level Materials Science Reasoning and Knowledgecited by 3
  35. 2025Exploring Modularity of Agentic Systems for Drug Discoverycited by 3
  36. 2025EVALUESTEER: Measuring Reward Model Steerability Towards Values and Preferencescited by 3
  37. 2026Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctnesscited by 3
  38. 2023Shai: A large language model for asset managementcited by 2

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Disentangling Length from Quality in Direct Preference Optimization
  2. 2024From Lists to Emojis: How Format Bias Affects Model Alignment
  3. 2024Dissecting Human and LLM Preferences
  4. 2024Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective
  5. 2025Aligning to What? Limits to RLHF Based Alignment
  6. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models
  7. 2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language
  8. 2026Alignment Tampering: How Reinforcement Learning from Human Feedback Is Exploited to Optimize Misaligned Biases