mlsanityData to 7 Oct 2026

Large Language Models are Inconsistent and Biased Evaluators

arXiv 2405.01724 ↗First posted 2 May 2024cs.CL cs.AIcited by 146

Not on the map. Why?

Abstract (arXiv)

The zero-shot capability of Large Language Models (LLMs) has enabled highly flexible, reference-free metrics for various tasks, making LLM evaluators common tools in NLP. However, the robustness of these LLM evaluators remains relatively understudied; existing work mainly pursued optimal performance in terms of correlating LLM scores with human expert scores. In this paper, we conduct a series of analyses using the SummEval dataset and confirm that LLMs are biased evaluators as they: (1) exhibit familiarity bias-a preference for text with lower perplexity, (2) show skewed and biased distributions of ratings, and (3) experience anchoring effects for multi-attribute judgments. We also found that LLMs are inconsistent evaluators, showing low "inter-sample" agreement and sensitivity to prompt differences that are insignificant to human understanding of text quality. Furthermore, we share recipes for configuring LLM evaluators to mitigate these limitations. Experimental results on the RoSE dataset demonstrate improvements over the state-of-the-art LLM evaluators.

Key facts. Cited by 146 papers in the map (data through 7 Oct 2026). Builds on 26 of its 49 references that are in the map. First posted on arXiv 2 May 2024.

Builds on

26 of 49 references.

References · 26

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  4. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  5. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  6. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  7. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  8. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  9. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  10. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  11. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 454
  12. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
14 more
  1. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  2. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  3. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  4. 2020Re-evaluating Evaluation in Text Summarizationcited by 144
  5. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  6. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  7. 2019Answers Unite! Unsupervised Metrics for Reinforced Summarization Modelscited by 92
  8. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 90
  9. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 88
  10. 2020Fill in the BLANC: Human-free quality estimation of document summariescited by 80
  11. 2023ChatGPT as a Factual Inconsistency Evaluator for Text Summarizationcited by 75
  12. 2022On the Limitations of Reference-Free Evaluations of Generated Textcited by 52
  13. 2023Less is More for Long Document Summary Evaluation by LLMscited by 28
  14. 2024Beyond Traditional Benchmarks: Analyzing Behaviors of Open LLMs on Data-to-Text Generationcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 28; 2025: 55; 2026: 63.

146 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  2. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  3. 2024Self-Preference Bias in LLM-as-a-Judgecited by 248
  4. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
  5. 2024Can LLM be a Personalized Judge?cited by 99
  6. 2024On scalable oversight with weak LLMs judging strong LLMscited by 94
  7. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  8. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
  9. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  10. 2025Correlated Errors in Large Language Modelscited by 42
  11. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  12. 2024Inverse Constitutional AI: Compressing Preferences into Principlescited by 39
38 more
  1. 2025Do LLM Evaluators Prefer Themselves for a Reason?cited by 38
  2. 2024Insight Over Sight: Exploring the Vision-Knowledge Conflicts in Multimodal LLMscited by 37
  3. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 36
  4. 2025Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judgecited by 35
  5. 2024Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructionscited by 34
  6. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  7. 2024A Comprehensive Evaluation of Cognitive Biases in LLMscited by 32
  8. 2024LMUnit: Fine-grained Evaluation with Natural Language Unit Testscited by 32
  9. 2025ExPerT: Effective and Explainable Evaluation of Personalized Long-Form Text Generationcited by 26
  10. 2024Evaluating the Consistency of LLM Evaluatorscited by 24
  11. 2025The simulation of judgment in LLMscited by 22
  12. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 22
  13. 2025Multi-Agent Debate for LLM Judges with Adaptive Stability Detectioncited by 20
  14. 2026Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Biascited by 19
  15. 2026Learning to Judge: LLMs Designing and Applying Evaluation Rubricscited by 18
  16. 2024From Single to Multi: How LLMs Hallucinate in Multi-Document Summarizationcited by 17
  17. 2025Your AI, Not Your View: The Bias of LLMs in Investment Analysiscited by 16
  18. 2025Trace Length is a Simple Uncertainty Signal in Reasoning Modelscited by 16
  19. 2024Towards Rationality in Language and Multimodal Agents: A Surveycited by 15
  20. 2025Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluationcited by 15
  21. 2024DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregationcited by 12
  22. 2025Firm or Fickle? Evaluating Large Language Models Consistency in Sequential Interactionscited by 12
  23. 2025Behavioral Analysis of Information Salience in Large Language Modelscited by 11
  24. 2025Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgmentcited by 11
  25. 2026Evaluating LLMs in Finance Requires Explicit Bias Considerationcited by 11
  26. 2024FactLens: Benchmarking Fine-Grained Fact Verificationcited by 10
  27. 2025SOTOPIA-S4: a user-friendly system for flexible, customizable, and large-scale social simulationcited by 10
  28. 2025SPHERE: An Evaluation Card for Human-AI Systemscited by 9
  29. 2025From Superficial Outputs to Superficial Learning: Risks of Large Language Models in Educationcited by 9
  30. 2025Do We Know What LLMs Don't Know? A Study of Consistency in Knowledge Probingcited by 7
  31. 2025Solar Photovoltaic Assessment with Large Language Modelcited by 7
  32. 2026Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelinescited by 7
  33. 2026The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluationcited by 7
  34. 2024From Next-Token to Mathematics: The Learning Dynamics of Mathematical Reasoning in Language Modelscited by 6
  35. 2024SkillAggregation: Reference-free LLM-Dependent Aggregationcited by 6
  36. 2025Skewed Score: A statistical framework to assess autograderscited by 6
  37. 2025Evaluating the Evaluators: Are readability metrics good measures of readability?cited by 6
  38. 2025Deep Associations, High Creativity: A Simple yet Effective Metric for Evaluating Large Language Modelscited by 6

50 most cited of 146 All 146 (JSON)