mlsanityData to 7 Oct 2026

Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations

arXiv 2601.22548 ↗First posted 30 Jan 2026cs.CL cs.AI cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Recent research has shown that large language models (LLMs) favor their own outputs when acting as judges, undermining the integrity of automated post-training and evaluation workflows. However, it is difficult to disentangle which behaviors are explained by narcissism versus experimental confounds. Specifically, LLM evaluators may deliver self-preferring verdicts when comparing responses to questions they fail on; these verdicts may not depend on the identity of the author, but on evaluator quality. We correct this by directly comparing the judge's voting distribution in cases where it evaluates itself versus another model. This evaluator quality baseline reveals that only 51% of examples in previous findings retain statistical significance against this null hypothesis, covering 89.6% of total self-preference probability mass. Finally, we compare the entropy of voting distributions, suggesting uncertainty-driven overlap, and show that our procedure enables more careful documentation against the backdrop of judge-bias research.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 35 of its 42 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

35 of 42 references.

References · 35

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  6. 2024A Survey on LLM-as-a-Judgecited by 1,384
  7. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  8. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  9. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  10. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  11. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  12. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
23 more
  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  2. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
  3. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  4. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 249
  5. 2024Self-Preference Bias in LLM-as-a-Judgecited by 248
  6. 2025Why Language Models Hallucinatecited by 205
  7. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  8. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 147
  9. 2023Taken out of context: On measuring situational awareness in LLMscited by 131
  10. 2024Looking Inward: Language Models Can Learn About Themselves by Introspectioncited by 105
  11. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 88
  12. 2024Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approachcited by 73
  13. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarizationcited by 70
  14. 2025Evaluating Scoring Bias in LLM-as-a-Judgecited by 42
  15. 2025Do LLM Evaluators Prefer Themselves for a Reason?cited by 38
  16. 2025Play Favorites: A Statistical Method to Measure Self-Bias in LLM-as-a-Judgecited by 35
  17. 2024Improving Uncertainty Quantification in Large Language Models via Semantic Embeddingscited by 30
  18. 2025Beyond the Surface: Measuring Self-Preference in LLM Judgmentscited by 30
  19. 2025Token-Level Density-Based Uncertainty Quantification Methods for Eliciting Truthfulness of Large Language Modelscited by 18
  20. 2024Inspection and Control of Self-Generated-Text Recognition Ability in Llama3-8b-Instructcited by 15
  21. 2025Mitigating Self-Preference by Authorship Obfuscationcited by 6
  22. 2025When an LLM is apprehensive about its answers -- and when its uncertainty is justifiedcited by 4
  23. 2025Breaking the Mirror: Activation-Based Mitigation of Self-Preference in LLM Evaluatorscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Self- and Other-Labels Induce Bidirectional Bias in LLM Judgescited by 1
  2. 2026CrossAudit: A Git-Native, Cross-Vendor Audit Loop for Agentic Sciencecited by 0
  3. 2026When Tools Get in the Way: The Effect of Unnecessary Tool Availability on LLM Answeringcited by 0
  4. 2026On the Limits of Metacognitive Monitoring in LLMscited by 0
  5. 2026A Shared Taste for Model-Written Text: The Generator-by-Selector Matrices of "AI-AI Bias" Show No Detectable Own-Model Premiumcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Quantifying and Mitigating Self-Preference Bias of LLM Judges
  2. 2026Self-Preference Bias in Rubric-Based Evaluation of Large Language Models
  3. 2025Are We on the Right Way to Assessing LLM-as-a-Judge?
  4. 2025LLMs Judge Themselves: A Game-Theoretic Framework for Human-Aligned Evaluation
  5. 2026Lower-Resource, Higher Scores: Language Bias in LLM Evaluators
  6. 2025Think Again! The Effect of Test-Time Compute on Preferences, Opinions, and Beliefs of Large Language Models
  7. 2024Evaluating the Evaluator: Measuring LLMs' Adherence to Task Evaluation Instructions
  8. 2025Safer or Luckier? LLMs as Safety Evaluators Are Not Robust to Artifacts