mlsanityData to 14 Oct 2026

LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scores

arXiv 2311.09766 ↗First posted 16 Nov 2023cs.CLcited by 90

Not on the map. Why?

Abstract (arXiv)

Automatic evaluation of generated textual content presents an ongoing challenge within the field of NLP. Given the impressive capabilities of modern language models (LMs) across diverse NLP tasks, there is a growing trend to employ these models in creating innovative evaluation metrics for automated assessment of generation tasks. This paper investigates a pivotal question: Do language model-driven evaluation metrics inherently exhibit bias favoring texts generated by the same underlying language model? Specifically, we assess whether prominent LM-based evaluation metrics (e.g. BARTScore, T5Score, and GPTScore) demonstrate a favorable bias toward their respective underlying LMs in the context of summarization tasks. Our findings unveil a latent bias, particularly pronounced when such evaluation metrics are used in a reference-free manner without leveraging gold summaries. These results underscore that assessments provided by generative evaluation models can be influenced by factors beyond the inherent text quality, highlighting the necessity of developing more reliable evaluation protocols in the future.

Key facts. Cited by 90 papers in the map (data through 14 Oct 2026). Builds on 33 of its 49 references that are in the map. First posted on arXiv 16 Nov 2023.

Builds on

33 of 49 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  6. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  7. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  8. 2015Teaching Machines to Read and Comprehendcited by 1,049
  9. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  10. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  11. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  12. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
21 more
  1. 2023GPTScore: Evaluate as You Desirecited by 361
  2. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  3. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 244
  4. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  5. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 158
  6. 2020Re-evaluating Evaluation in Text Summarizationcited by 145
  7. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 142
  8. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 132
  9. 2019Better Rewards Yield Better Summaries: Learning to Summarise Without Referencescited by 77
  10. 2020SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarizationcited by 69
  11. 2022On the Limitations of Reference-Free Evaluations of Generated Textcited by 52
  12. 2021Perturbation CheckLists for Evaluating NLG Evaluation Metricscited by 50
  13. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 49
  14. 2022Spurious Correlations in Reference-Free Evaluation of Text Generationcited by 26
  15. 2022How Far are We from Robust Long Abstractive Summarization?cited by 25
  16. 2021A Training-free and Reference-free Summarization Evaluation Metric via Centrality-weighted Relevance and Self-referenced Redundancycited by 21
  17. 2022T5Score: Discriminative Fine-tuning of Generative Evaluation Metricscited by 19
  18. 2022SMART: Sentences as Basic Units for Text Evaluationcited by 18
  19. 2020Unsupervised Reference-Free Summary Quality Evaluation via Contrastive Learningcited by 17
  20. 2022Shortcomings of Question Answering Based Factuality Frameworks for Error Localizationcited by 15
  21. 2022Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?cited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 26; 2025: 38; 2026: 25.

90 papers.

Sort:

By area:

  1. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  2. 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Modelcited by 526
  3. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  4. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  5. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  6. 2024Large Language Models are Inconsistent and Biased Evaluatorscited by 153
  7. 2024Is LLM-as-a-Judge Robust? Investigating Universal Adversarial Attacks on Zero-shot LLM Assessmentcited by 126
  8. 2025Preference Leakage: A Contamination Problem in LLM-as-a-judgecited by 125
  9. 2024Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinementcited by 114
  10. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 98
  11. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  12. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 74
38 more
  1. 2025The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Inputcited by 52
  2. 2024Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the datacited by 51
  3. 2026DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agentscited by 50
  4. 2024Navigating the Metrics Maze: Reconciling Score Magnitudes and Accuraciescited by 40
  5. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 39
  6. 2025Do LLM Evaluators Prefer Themselves for a Reason?cited by 38
  7. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  8. 2025Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judgecited by 36
  9. 2024WildFeedback: Aligning LLMs With In-situ User Interactions And Feedbackcited by 31
  10. 2025Beyond the Surface: Measuring Self-Preference in LLM Judgmentscited by 31
  11. 2024Spontaneous Reward Hacking in Iterative Self-Refinementcited by 26
  12. 2024How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMscited by 23
  13. 2024From Calculation to Adjudication: Examining LLM judges on Mathematical Reasoning Taskscited by 22
  14. 2025TripTailor: A Real-World Benchmark for Personalized Travel Planningcited by 22
  15. 2024Self-Recognition in Language Modelscited by 18
  16. 2025GaRAGe: A Benchmark with Grounding Annotations for RAG Evaluationcited by 18
  17. 2024TREC iKAT 2023: A Test Collection for Evaluating Conversational and Interactive Knowledge Assistantscited by 17
  18. 2024Can Many-Shot In-Context Learning Help LLMs as Evaluators? A Preliminary Empirical Studycited by 15
  19. 2024Writing Style Matters: An Examination of Bias and Fairness in Information Retrieval Systemscited by 15
  20. 2025Rankers, Judges, and Assistants: Towards Understanding the Interplay of LLMs in Information Retrieval Evaluationcited by 15
  21. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  22. 2023LIMIT: Less Is More for Instruction Tuning Across Evaluation Paradigmscited by 14
  23. 2025Reassessing Extractive QA Datasets at Scale: LLM-as-a-Judge and In-Depth Analysescited by 14
  24. 2026Quantifying and Mitigating Self-Preference Bias of LLM Judgescited by 10
  25. 2025Adaptive-VP: A Framework for LLM-Based Virtual Patients that Adapts to Trainees' Dialogue to Facilitate Nurse Communication Trainingcited by 9
  26. 2024MONAL: Model Autophagy Analysis for Modeling Human-AI Interactionscited by 7
  27. 2024AGENT-CQ: Automatic Generation and Evaluation of Clarifying Questions for Conversational Search with LLMscited by 7
  28. 2025Towards Understanding Bias in Synthetic Data for Evaluationcited by 7
  29. 2025Skewed Score: A statistical framework to assess autograderscited by 7
  30. 2025Evaluating the Evaluators: Are readability metrics good measures of readability?cited by 6
  31. 2026MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systemscited by 6
  32. 2026Judging Against the Reference: Uncovering Knowledge-Driven Failures in LLM-Judges on QA Evaluationcited by 5
  33. 2026Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluationscited by 5
  34. 2026FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judgecited by 5
  35. 2025Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Promptscited by 4
  36. 2025Drivel-ology: Challenging LLMs with Interpreting Nonsense with Depthcited by 4
  37. 2026A Statistical Framework for Auditing Behavioral Dependence and Induced Bias in LLM Judgescited by 4
  38. 2025Reverse Engineering Human Preferences with Reinforcement Learningcited by 3

50 most cited of 90 All 90 (JSON)