mlsanityData to 14 Oct 2026

Rating Roulette: Self-Inconsistency in LLM-As-A-Judge Frameworks

arXiv 2510.27106 ↗First posted 31 Oct 2025cs.CLcited by 38

Not on the map. Why?

Abstract (arXiv)

As Natural Language Generation (NLG) continues to be widely adopted, properly assessing it has become quite difficult. Lately, using large language models (LLMs) for evaluating these generations has gained traction, as they tend to align more closely with human preferences than conventional n-gram or embedding-based metrics. In our experiments, we show that LLM judges have low intra-rater reliability in their assigned scores across different runs. This variance makes their ratings inconsistent, almost arbitrary in the worst case, making it difficult to measure how good their judgments actually are. We quantify this inconsistency across different NLG tasks and benchmarks and see if judicious use of LLM judges can still be useful following proper guidelines.

Key facts. Cited by 38 papers in the map (data through 14 Oct 2026). Builds on 29 of its 53 references that are in the map. First posted on arXiv 31 Oct 2025.

Builds on

29 of 53 references.

References · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  6. 2024A Survey on LLM-as-a-Judgecited by 1,423
  7. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  8. 2018Hierarchical Neural Story Generationcited by 1,193
  9. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  10. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  11. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  12. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
17 more
  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  3. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 407
  4. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 326
  5. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  6. 2023State of What Art? A Call for Multi-Prompt LLM Evaluationcited by 310
  7. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 287
  8. 2020Evaluation of Text Generation: A Surveycited by 277
  9. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  10. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 205
  11. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 94
  12. 2025JudgeLRM: Large Reasoning Models as a Judgecited by 78
  13. 2020What Have We Achieved on Text Summarization?cited by 65
  14. 2023Evaluating Evaluation Metrics: A Framework for Analyzing NLG Evaluation Metrics using Measurement Theorycited by 43
  15. 2024From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyondcited by 24
  16. 2023Consistency is Key: Disentangling Label Variation in Natural Language Processing with Intra-Annotator Agreementcited by 23
  17. 2024Black-box Uncertainty Quantification Method for LLM-as-a-Judgecited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 37.

38 papers.

Sort:

By area:

  1. 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Taskscited by 62
  2. 2026Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models Across Agreement, Consistency, and Biascited by 28
  3. 2026CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modelingcited by 12
  4. 2026FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Usecited by 10
  5. 2026Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarkingcited by 10
  6. 2026From Words to Widgets for Controllable LLM Generationcited by 5
  7. 2025Pessimistic Verification for Open Ended Math Questionscited by 3
  8. 2026Label Effects: Shared Heuristic Reliance in Trust Assessment by Humans and LLM-as-a-Judgecited by 3
  9. 2026SIR-Bench: Evaluating Investigation Depth in Security Incident Response Agentscited by 3
  10. 2026LLM-as-a-Judge Scores Are Unreliable Optimization Signals in Closed-Loop Table Recognitioncited by 3
  11. 2026MTVA-Bench: Evaluating the Language Model Inside Cascaded Voice Agentscited by 3
  12. 2026Overtrained, Not Misalignedcited by 2
26 more
  1. 2026When Languages Disagree: Self-Evolving Multilingual LLM Judgescited by 2
  2. 2026Improving Cross-Format Robustness in Language Models with Multi-Format Trainingcited by 2
  3. 2026Interrupting the Loop: Periodic Subject Changes Raise Judged Surprise and Connection in Base Language Modelscited by 2
  4. 2026Evaluative Fingerprints: Stable and Systematic Differences in LLM Evaluator Behaviorcited by 1
  5. 2026The Stability Trap: Evaluating the Reliability of LLM-Based Instruction Adherence Auditingcited by 1
  6. 2026Validating Political Position Predictions of Argumentscited by 1
  7. 2026SEAR: Schema-Based Evaluation and Routing for LLM Gatewayscited by 1
  8. 2026Large Language Model Selection with Limited Annotationscited by 1
  9. 2026LLM Judges Inconsistently Disagree Across Safety Criteria and Harm Categoriescited by 1
  10. 2026CAFE: A Compound-AI Factorial Evaluation Frameworkcited by 1
  11. 2026Suffix-Constrained Greedy Search Algorithms for Causal Language Modelscited by 0
  12. 2026PAC-CF: Calibrating Irreversible Frontier Pruning in LLM-Guided Searchcited by 0
  13. 2026Naturalistic measure of social norms alignmentcited by 0
  14. 2026Multi-Stakeholder LLM Alignment: Decomposing Estimation from Aggregationcited by 0
  15. 2026FraudSMSWalker: Benchmarking Agentic Large Language Models for SMS-to-Webpage Fraud Detectioncited by 0
  16. 2026Towards Spec Learning: Inference-Time Alignment from Preference Pairscited by 0
  17. 2026The Consistency Dilemma in LLMs: Generator-Evaluator Agreement and Vulnerability to Mistakescited by 0
  18. 2026Project Kaleidoscope: Contextual, Human-Aligned Evaluation for Real-World AI Applicationscited by 0
  19. 2026PlanFlip: Attacking Multi-Agent LLM Systems via Planning-Phase Prompt Injectioncited by 0
  20. 2026Evaluating and Guarding Citation Faithfulness in Agentic Scientific Synthesiscited by 0
  21. 2026LLM-Derived Preference Judgments Are Not Self-Consistentcited by 0
  22. 2026Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpointscited by 0
  23. 2026Style Over Substance: Content-Invariant Wrappers Flip LLM Safety-Judge Verdictscited by 0
  24. 2026RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?cited by 0
  25. 2026Better Deck or Different Judge? Evaluating Agentic Harness Gains in Corporate and Investment Bankingcited by 0
  26. 2026SoK: Grading Open-Text Legal Answers, from Law School Exams to Trustworthy Rubric Judgescited by 0