mlsanityData to 7 Oct 2026

Humans or LLMs as the Judge? A Study on Judgement Biases

arXiv 2402.10669 ↗First posted 16 Feb 2024cs.CLcited by 314

Not on the map. Why?

Cited by 31 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Adopting human and large language models (LLM) as judges (a.k.a human- and LLM-as-a-judge) for evaluating the performance of LLMs has recently gained attention. Nonetheless, this approach concurrently introduces potential biases from human and LLMs, questioning the reliability of the evaluation results. In this paper, we propose a novel framework that is free from referencing groundtruth annotations for investigating Misinformation Oversight Bias, Gender Bias, Authority Bias and Beauty Bias on LLM and human judges. We curate a dataset referring to the revised Bloom's Taxonomy and conduct thousands of evaluations. Results show that human and LLM judges are vulnerable to perturbations to various degrees, and that even the cutting-edge judges possess considerable biases. We further exploit these biases to conduct attacks on LLM judges. We hope that our work can notify the community of the bias and vulnerability of human- and LLM-as-a-judge, as well as the urgency of developing robust evaluation systems.

Key facts. Cited by 314 papers in the map (data through 7 Oct 2026). Builds on 30 of its 105 references that are in the map. First posted on arXiv 16 Feb 2024.

Builds on

30 of 105 references.

References · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  10. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  11. 2023Large Language Modelscited by 774
  12. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
18 more
  1. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  2. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  3. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  4. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  5. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  6. 2019Gradio: Hassle-Free Sharing and Testing of ML Models in the Wildcited by 176
  7. 2023CMB: A Comprehensive Medical Benchmark in Chinesecited by 113
  8. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 88
  9. 2019Better Rewards Yield Better Summaries: Learning to Summarise Without Referencescited by 76
  10. 2023Automated Annotation with Generative AI Requires Validationcited by 75
  11. 2018Can Neural Machine Translation be Improved with User Feedback?cited by 71
  12. 2023Style Over Substance: Evaluation Biases for Large Language Modelscited by 68
  13. 2022DiscoScore: Evaluating Text Generation with BERT and Discourse Coherencecited by 35
  14. 2023Prompt Packer: Deceiving LLMs through Compositional Instruction with Hidden Attackscited by 30
  15. 2020Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Modelscited by 27
  16. 2023Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLPcited by 21
  17. 2023DecipherPref: Analyzing Influential Factors in Human Preference Judgments via GPT-4cited by 16
  18. 2023The influence of user personality and rating scale features on rating behaviour: an empirical studycited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 48; 2025: 120; 2026: 146.

314 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  2. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  3. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  4. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 224
  5. 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
  6. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  7. 2024Lessons from the Trenches on Reproducible Evaluation of Language Modelscited by 172
  8. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  9. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 120
  10. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 119
  11. 2025JudgeLRM: Large Reasoning Models as a Judgecited by 76
  12. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 73
38 more
  1. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 71
  2. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  3. 2025One Token to Fool LLM-as-a-Judgecited by 56
  4. 2025Large Language Models Hallucination: A Comprehensive Surveycited by 45
  5. 2024HelloBench: Evaluating Long Text Generation Capabilities of Large Language Modelscited by 42
  6. 2024Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Ratescited by 42
  7. 2025Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifierscited by 40
  8. 2024Inverse Constitutional AI: Compressing Preferences into Principlescited by 39
  9. 2025Do LLM Evaluators Prefer Themselves for a Reason?cited by 38
  10. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 37
  11. 2024Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarkingcited by 35
  12. 2025Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judgecited by 35
  13. 2025Investigating Non-Transitivity in LLM-as-a-Judgecited by 33
  14. 2025Theoretical Physics Benchmark (TPBench) -- a Dataset and Study of AI Reasoning Capabilities in Theoretical Physicscited by 33
  15. 2024LLM-as-a-Judge & Reward Model: What They Can and Cannot Docited by 32
  16. 2024GameArena: Evaluating LLM Reasoning through Live Computer Gamescited by 32
  17. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 32
  18. 2025CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Taskscited by 32
  19. 2025LLM-based Human Simulations Have Not Yet Been Reliablecited by 30
  20. 2025Search Arena: Analyzing Search-Augmented LLMscited by 30
  21. 2025JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimizationcited by 30
  22. 2024AlphaDPO: Adaptive Reward Margin for Direct Preference Optimizationcited by 27
  23. 2025ExPerT: Effective and Explainable Evaluation of Personalized Long-Form Text Generationcited by 26
  24. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  25. 2025How Benchmark Prediction from Fewer Data Misses the Markcited by 26
  26. 2025Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to Artifactscited by 25
  27. 2025Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directionscited by 23
  28. 2025Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluationcited by 23
  29. 2026Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judgescited by 23
  30. 2024Efficient LLM Comparative Assessment: a Product of Experts Framework for Pairwise Comparisonscited by 22
  31. 2024Evaluating Creative Short Story Generation in Humans and Large Language Modelscited by 21
  32. 2024Reasoning Through Execution: Unifying Process and Outcome Rewards for Code Generationcited by 21
  33. 2025Text Data Augmentation for Large Language Models: A Comprehensive Survey of Methods, Challenges, and Opportunitiescited by 21
  34. 2025Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Resultscited by 21
  35. 2025Shrinking the Generation-Verification Gap with Weak Verifierscited by 21
  36. 2026RubricBench: Aligning Model-Generated Rubrics with Human Standardscited by 21
  37. 2024Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Taskscited by 20
  38. 2024Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detectioncited by 20

50 most cited of 314 All 314 (JSON)