# Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judge

arXiv: https://arxiv.org/abs/2603.11027 · HTML: https://mlsanity.com/paper/2603.11027/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 38 references that are in the map.
- First posted on arXiv 11 Mar 2026.

## Builds on (most cited first, 27 of 27)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [Scaling Laws for Reward Model Overoptimization](https://mlsanity.com/paper/2210.10760/) (2022)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Self-Rewarding Language Models](https://mlsanity.com/paper/2401.10020/) (2024)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://mlsanity.com/paper/2405.01535/) (2024)
- [JudgeBench: A Benchmark for Evaluating LLM-based Judges](https://mlsanity.com/paper/2410.12784/) (2024)
- [Self-Preference Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2410.21819/) (2024)
- [FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets](https://mlsanity.com/paper/2307.10928/) (2023)
- [Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms](https://mlsanity.com/paper/2406.02900/) (2024)
- [The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation](https://mlsanity.com/paper/2211.02570/) (2022)
- [OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment](https://mlsanity.com/paper/2510.07743/) (2025)
- [WritingBench: A Comprehensive Benchmark for Generative Writing](https://mlsanity.com/paper/2503.05244/) (2025)
- [CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists](https://mlsanity.com/paper/2403.18771/) (2024)
- [Online Rubrics Elicitation from Pairwise Comparisons](https://mlsanity.com/paper/2510.07284/) (2025)
- [LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing](https://mlsanity.com/paper/2507.00769/) (2025)
- [SedarEval: Automated Evaluation using Self-Adaptive Rubrics](https://mlsanity.com/paper/2501.15595/) (2025)
- [CM2: Reinforcement Learning with Checklist Rewards for Multi-Turn and Multi-Step Agentic Tool Use](https://mlsanity.com/paper/2602.12268/) (2026)
- [Curse of Knowledge: When Complex Evaluation Context Benefits yet Biases LLM Judges](https://mlsanity.com/paper/2509.03419/) (2025)
- [MENTOR: A Metacognition-Driven Self-Evolution Framework for Uncovering and Mitigating Implicit Domain Risks in LLMs](https://mlsanity.com/paper/2511.07107/) (2025)

## Built on by (most cited first, 4 of 4)

- [The Deliberative Deficit: An Empirical Critique of LLMs in Democratic Discourse](https://mlsanity.com/paper/2608.10186/) (2026)
- [Agreement Without Alignment: Label-Free Rubric Clarification Increases LLM-Judge Consensus but Not Agreement with Humans](https://mlsanity.com/paper/2610.13303/) (2026)
- [GrowLoop: Self-Evolving Conversation Evaluation Seeded by Human](https://mlsanity.com/paper/2605.28882/) (2026)
- [From Judgment Quality to Downstream Utility: Rethinking LLM-as-a-Judge for Open-Ended Tasks](https://mlsanity.com/paper/2609.37145/) (2026)

## Abstract (arXiv)

The paradigm of LLM-as-a-judge relies on a critical assumption, namely that high inter-evaluator agreement indicates reliable and objective evaluation. We present two complementary findings that challenge this assumption. First, we demonstrate that this consensus is frequently illusory. We identify and formalize Evaluation Illusion, a phenomenon where LLM judges generate sophisticated critiques yet anchor scores on shared surface heuristics rather than substantive quality. Through a large-scale study of 105,600 evaluation instances (32 LLMs × 3 frontier judges × 100 tasks × 11 temperatures), we show that model-level agreement (Spearman ρ = 0.99) masks fragile sample-level agreement (Pearson barr = 0.72; absolute agreement ICC = 0.67), that merely sharing rubric structure restores 62% of total agreement, and that high-quality outputs paradoxically receive the textitleast consistent evaluations. Second, we demonstrate that dynamically generating evaluation rubrics grounded in domain knowledge produces more meaningful assessment. We introduce MERG (Metacognitive Enhanced Rubric Generation), a knowledge-driven rubric generation framework whose domain-selective effects confirm this. Agreement textitincreases in codified domains (Education +22%, Academic +27%) where knowledge anchors evaluators on shared standards, while it decreases in subjective domains where genuine evaluative pluralism emerges. These findings suggest that evaluation rubrics should be dynamically enriched with expert knowledge rather than relying on generic criteria, with implications for reward modeling in RLAIF.