mlsanityData to 14 Oct 2026

Human Feedback is not Gold Standard

arXiv 2309.16349 ↗First posted 28 Sep 2023cs.CLcited by 83

On the map: LLM alignment / RLHF / preference optimisation → Human values and AI feedback alignment

Abstract (arXiv)

Human feedback has become the de facto standard for evaluating the performance of Large Language Models, and is increasingly being used as a training objective. However, it is not clear which properties of a generated output this single `preference' score captures. We hypothesise that preference scores are subjective and open to undesirable biases. We critically analyse the use of human feedback for both training and evaluation, to verify whether it fully captures a range of crucial error criteria. We find that while preference scores have fairly good coverage, they under-represent important aspects like factuality. We further hypothesise that both preference scores and error annotation may be affected by confounders, and leverage instruction-tuned models to generate outputs that vary along two possible confounding dimensions: assertiveness and complexity. We find that the assertiveness of an output skews the perceived rate of factuality errors, indicating that human annotations are not a fully reliable evaluation metric or training objective. Finally, we offer preliminary evidence that using human feedback as a training objective disproportionately increases the assertiveness of model outputs. We encourage future work to carefully consider whether preference scores are well aligned with the desired objective.

Key facts. Cited by 83 papers in the map (data through 14 Oct 2026). Placed in Human values and AI feedback alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 22 of its 46 references that are in the map. First posted on arXiv 28 Sep 2023.

Builds on

22 of 46 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Discovering Language Model Behaviors with Model-Written Evaluations19 Dec

Other references · 21

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  4. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  5. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  6. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 401
  7. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  8. 2017Why We Need New Evaluation Metrics for NLGcited by 233
  9. 2018WikiHow: A Large Scale Text Summarization Datasetcited by 204
  10. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
  11. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 142
  12. 2023ExpertPrompting: Instructing Large Language Models to be Distinguished Expertscited by 142
9 more
  1. 2022Jury Learning: Integrating Dissenting Voices into Machine Learning Modelscited by 118
  2. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  3. 2022TweetNLP: Cutting-Edge Natural Language Processing for Social Mediacited by 71
  4. 2018RankME: Reliable Human Ratings for Natural Language Generationcited by 69
  5. 2022POTATO: The Portable Text Annotation Toolcited by 64
  6. 2023Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questionscited by 42
  7. 2022Hierarchical Sketch Induction for Paraphrase Generationcited by 22
  8. 2023Collective Human Opinions in Semantic Textual Similaritycited by 8
  9. 2023Attributable and Scalable Opinion Summarizationcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 41; 2025: 17; 2026: 20.

83 papers.

Sort:

By area:

  1. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 244
  2. 2024A Roadmap to Pluralistic Alignmentcited by 184
  3. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 179
  4. 2023Sociotechnical Safety Evaluation of Generative AI Systemscited by 157
  5. 2024The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Modelscited by 144
  6. 2024Language Models Learn to Mislead Humans via RLHFcited by 124
  7. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systemscited by 117
  8. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 87
  9. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  10. 2024Understanding and Mitigating Language Confusion in LLMscited by 75
  11. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 68
  12. 2024Unintended Impacts of LLM Alignment on Global Representationcited by 66
38 more
  1. 2024Reasoning Aware Self-Consistency: Leveraging Reasoning Paths for Efficient LLM Samplingcited by 63
  2. 2023The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Valuescited by 54
  3. 2024(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Textscited by 49
  4. 2024FLAME: Factuality-Aware Alignment for Large Language Modelscited by 47
  5. 2024Improving Reward Models with Synthetic Critiquescited by 44
  6. 2024Inverse Constitutional AI: Compressing Preferences into Principlescited by 41
  7. 2024Instruction Tuning With Loss Over Instructionscited by 40
  8. 2024PARIKSHA: A Large-Scale Investigation of Human-LLM Evaluator Agreement on Multilingual and Multi-Cultural Datacited by 37
  9. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 33
  10. 2025CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning taskscited by 32
  11. 2023Compositional preference models for aligning LMscited by 26
  12. 2024Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignmentcited by 26
  13. 2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Datacited by 26
  14. 2024Aligning Large Language Models by On-Policy Self-Judgmentcited by 20
  15. 2024High-Dimension Human Value Representation in Large Language Modelscited by 20
  16. 2024Beyond Probabilities: Unveiling the Misalignment in Evaluating Large Language Modelscited by 19
  17. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 18
  18. 2024Eureka: Evaluating and Understanding Large Foundation Modelscited by 18
  19. 2024Dissecting Human and LLM Preferencescited by 17
  20. 2026Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learningcited by 16
  21. 2023LIMIT: Less Is More for Instruction Tuning Across Evaluation Paradigmscited by 14
  22. 2025Reward Model Interpretability via Optimal and Pessimal Tokenscited by 10
  23. 2024A Preference-driven Paradigm for Enhanced Translation with Large Language Modelscited by 9
  24. 2025Can External Validation Tools Improve Annotation Quality for LLM-as-a-Judge?cited by 9
  25. 2025Are Checklists Really Useful for Automatic Evaluation of Generative Tasks?cited by 9
  26. 2025Measuring Scalar Constructs in Social Science with LLMscited by 9
  27. 2025LIVS: A Pluralistic Alignment Dataset for Inclusive Public Spacescited by 8
  28. 2024Rethinking the Role of Proxy Rewards in Language Model Alignmentcited by 7
  29. 2024Hierarchical Indexing for Retrieval-Augmented Opinion Summarizationcited by 7
  30. 2024Model-based Preference Optimization in Abstractive Summarization without Human Feedbackcited by 7
  31. 2024Understanding Likelihood Over-optimisation in Direct Alignment Algorithmscited by 7
  32. 2025A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Userscited by 6
  33. 2024Beyond Accuracy Optimization: Computer Vision Losses for Large Language Model Fine-Tuningcited by 5
  34. 2025J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judgecited by 5
  35. 2026Reasoning-Driven Synthetic Data Generation and Evaluationcited by 5
  36. 2024SEAL: Systematic Error Analysis for Value ALignmentcited by 4
  37. 2024Disentangling Preference Representation and Text Generation for Efficient Individual Preference Alignmentcited by 4
  38. 2025Assessing the Reliability of LLMs Annotations in the Context of Demographic Bias and Model Explanationcited by 4

50 most cited of 83 All 83 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models
  2. 2022Training Language Models with Language Feedback
  3. 2025Accelerating Unbiased LLM Evaluation via Synthetic Feedback
  4. 2024ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models
  5. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
  6. 2023DecipherPref: Analyzing Influential Factors in Human Preference Judgments via GPT-4
  7. 2024Aligning language models with human preferences
  8. 2024A Survey on Human Preference Learning for Large Language Models