# LLM-Evaluation Tropes: Perspectives on the Validity of LLM-Evaluations

arXiv: https://arxiv.org/abs/2504.19076 · HTML: https://mlsanity.com/paper/2504.19076/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Apr 2025.

## Built on by (most cited first, 15 of 15)

- [The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMs](https://mlsanity.com/paper/2506.11094/) (2025)
- [Evaluating and Mitigating LLM-as-a-judge Bias in Communication Systems](https://mlsanity.com/paper/2510.12462/) (2025)
- [Medical AI Consensus: A Multi-Agent Framework for Radiology Report Generation and Evaluation](https://mlsanity.com/paper/2509.17353/) (2025)
- [Causal Judge Evaluation: Calibrated Surrogate Metrics for LLM Systems](https://mlsanity.com/paper/2512.11150/) (2025)
- [LLM Essay Scoring Under Holistic and Analytic Rubrics: Prompt Effects and Bias](https://mlsanity.com/paper/2604.00259/) (2026)
- [Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets?](https://mlsanity.com/paper/2601.13227/) (2026)
- [How Far Can VLMs Go for Visual Bug Detection? Studying 19,738 Keyframes from 41 Hours of Gameplay Videos](https://mlsanity.com/paper/2603.22706/) (2026)
- [Re-Rankers as Relevance Judges](https://mlsanity.com/paper/2601.04455/) (2026)
- [RMIT-ADM+S at the MMU-RAG NeurIPS 2025 Competition](https://mlsanity.com/paper/2602.20735/) (2026)
- [On Challenges in Evaluation Methodology: A Review](https://mlsanity.com/paper/2604.25923/) (2026)
- [Annotation Quality in Aspect-Based Sentiment Analysis: A Case Study Comparing Experts, Students, Crowdworkers, and Large Language Model](https://mlsanity.com/paper/2605.03624/) (2026)
- [MDIA: A Multi-Agent Diagnostic Intelligence Pipeline on HealthBench Professional](https://mlsanity.com/paper/2605.24699/) (2026)
- [Generalistic or Specific Embeddings, Which is Better? An Empirical Study on Search for Clinical Coding in Non-English Languages](https://mlsanity.com/paper/2605.30529/) (2026)
- [Retrieval, not hallucinations, will be the limiting factor for LLM-based clinical AI tools](https://mlsanity.com/paper/2607.24793/) (2026)
- [LLJ Cards: Best practices for the Use of LLMs as Judges](https://mlsanity.com/paper/2609.24516/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are increasingly used to evaluate information retrieval (IR) systems, generating relevance judgments traditionally made by human assessors. Recent empirical studies suggest that LLM-based evaluations often align with human judgments, leading some to suggest that human judges may no longer be necessary, while others highlight concerns about judgment reliability, validity, and long-term impact. As IR systems begin incorporating LLM-generated signals, evaluation outcomes risk becoming self-reinforcing, potentially leading to misleading conclusions. This paper examines scenarios where LLM-evaluators may falsely indicate success, particularly when LLM-based judgments influence both system development and evaluation. We highlight key risks, including bias reinforcement, reproducibility challenges, and inconsistencies in assessment methodologies. To address these concerns, we propose tests to quantify adverse effects, guardrails, and a collaborative framework for constructing reusable test collections that integrate LLM judgments responsibly. By providing perspectives from academia and industry, this work aims to establish best practices for the principled use of LLMs in IR evaluation.