Judge Circuits Explain Format-Induced Inconsistency in LLM-as-a-Judge
Not on the map. Why?
Abstract (arXiv)
LLM-as-a-judge has become the dominant paradigm for grading model outputs at scale, yet the same model assigns systematically different scores when its output format changes (e.g., a 1-5 rating vs. a True/False label). Existing diagnoses of these format-induced inconsistencies stop at the input-output level. Using Position-aware Edge Attribution Patching (PEAP), we causally investigate the internal mechanism in five open-weight instruction-tuned models (Gemma-3, Qwen2.5, Llama-3.1) across five judgment tasks. We find that judgments across structured understanding and open-ended preference tasks share a sparse Latent Evaluator sub-graph in the mid-to-late layers; zero-ablating it collapses judgment while damaging knowledge probes substantially less than a random ablation of equal size in architecturally modular models. By structurally decoupling abstract judging from output formatting, we provide a mechanistic account of format-induced inconsistency on the open-weight models we study: a continuous judgment signal computed in the shared trunk is mapped through fragile, format-specific terminal branches. The judgment itself can therefore be read out independently of the requested output format. Our findings imply that benchmark comparisons of judge reliability across formats partly measure the fragile formatting stage, and can understate the quality of the underlying evaluation.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 40 of its 55 references that are in the map. First posted on arXiv 15 May 2026.
Builds on
40 of 55 references.
References · 40
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2024Qwen2.5 Technical Reportcited by 4,942
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
- 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
- 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
- 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
- 2018Neural Network Acceptability Judgmentscited by 958
- 2023Steering Language Models With Activation Engineeringcited by 922
- 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
28 more
- 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
- 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
- 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
- 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
- 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 209
- 2024Transcoders Find Interpretable LLM Feature Circuitscited by 200
- 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 149
- 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 103
- 2023Language Models Implement Simple Word2Vec-style Vector Arithmeticcited by 98
- 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 90
- 2021CREAK: A Dataset for Commonsense Reasoning over Entity Knowledgecited by 88
- 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 74
- 2024LLM-based NLG Evaluation: Current Status and Challengescited by 73
- 2024LLM Circuit Analyses Are Consistent Across Training and Scalecited by 63
- 2025MIB: A Mechanistic Interpretability Benchmarkcited by 56
- 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 39
- 2025Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?cited by 34
- 2024On the Similarity of Circuits across Languages: a Case Study on the Subject-verb Agreement Taskcited by 24
- 2024Evaluating the Consistency of LLM Evaluatorscited by 24
- 2023Towards Interpretable Sequence Continuation: Analyzing Shared Circuits in Large Language Modelscited by 15
- 2025Position-aware Automatic Circuit Discoverycited by 14
- 2025FADE: Why Bad Descriptions Happen to Good Featurescited by 13
- 2023Uncovering Intermediate Variables in Transformers using Circuit Probingcited by 12
- 2025Are formal and functional linguistic mechanisms dissociated in language models?cited by 11
- 2025Short-circuiting Shortcuts: Mechanistic Investigation of Shortcuts in Text Classificationcited by 5
- 2025Circuit Insights: Towards Interpretability Beyond Activationscited by 5
- 2025Reference-Free Rating of LLM Responses via Latent Informationcited by 4
- 2026Circuit Fingerprints: How Answer Tokens Encode Their Geometrical Pathcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias
- 2025When Judgment Becomes Noise: How Design Failures in LLM Judge Benchmarks Silently Undermine Validity
- 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
- 2026Same Input, Different Scores: A Multi Model Study on the Inconsistency of LLM Judge
- 2026Context Over Content: Exposing Evaluation Faking in Automated Judges
- 2026Anchoring Bias in LLM-as-a-Judge Systems: Prior Scores Compromise Evaluation Independence
- 2026From Judgment Quality to Downstream Utility: Rethinking LLM-as-a-Judge for Open-Ended Tasks
- 2026FairJudge: An Adaptive, Debiased, and Consistent LLM-as-a-Judge