The Formalism Trap: Are LLM-as-a-Judge Evaluators Blinded by Consensus Mimicry under Social Load?
On the map: LLM agents and tool use → Social simulation and multi-agent debate
Abstract (arXiv)
We introduce the textitAgentic Formalism Trap and the Evaluative Dissonance Index (DE), quantifying how LLM-as-a-Judge systems conflate structural proceduralism with semantic truth under adversarial load. Analyzing 22,500 trajectories across 3 domains (GAIA, SWE-bench, Multi-Challenge), we extract a semantic taxonomy of hallucination maneuvers, validated via deterministic lexical grounding (p < 10-120). A logistic meta-evaluator isolates the exact syntactic triggers of this evaluator capture (ROC-AUC 0.8779), while a zero-shot Leave-One-Domain-Out transfer proves the vulnerability is universally domain-agnostic (mean ROC-AUC 0.7482). Architectural profiling reveals that distinct simulated swarm topologies induce mathematically disparate semantic blind spots, proving that unanchored closed-loop evaluation is unstable, systemically divergent and necessitates architecture-specific vigilance filters.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 20 of its 28 references that are in the map. First posted on arXiv 20 May 2026.
Builds on
20 of 28 references.
Closest ancestors · 2
Same direction, newest first
- 2026The Bystander Effect in Multi-Agent Reasoning: Quantifying Cognitive Loafing in Collaborative Interactions11 May
- 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate30 May
Other references · 18
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
- 2024A Survey on LLM-as-a-Judgecited by 1,384
- 2023GAIA: a benchmark for General AI Assistantscited by 1,084
- 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmarkcited by 362
- 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
- 2023How is ChatGPT's behavior changing over time?cited by 265
- 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
- 2025MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMscited by 171
- 2025MultiAgentBench: Evaluating the Collaboration and Competition of LLM agentscited by 169
- 2017Breaking the Softmax Bottleneck: A High-Rank RNN Language Modelcited by 147
6 more
- 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
- 2023Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communicationcited by 78
- 2024Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the datacited by 46
- 2025SwarmAgentic: Towards Fully Automated Agentic System Generation via Swarm Intelligencecited by 17
- 2025Evaluating Generalization Capabilities of LLM-Based Agents in Mixed-Motive Scenarios Using Concordiacited by 11
- 2025CoGate-LSTM: Prototype-Guided Feature-Space Gating for Mitigating Gradient Dilution in Imbalanced Toxic Comment Classificationcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation
- 2026Agent-as-a-Judge
- 2026The Consensus Trap: Rescuing Multi-Agent LLMs from Adversarial Majorities via Token-Level Collaboration
- 2026The Inverse-Wisdom Law: Architectural Tribalism and the Consensus Paradox in Agentic Swarms
- 2026FormalJudge: A Neuro-Symbolic Paradigm for Agentic Oversight
- 2026Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judge
- 2026Context Over Content: Exposing Evaluation Faking in Automated Judges
- 2026LLM Agents Make Collective Belief Dynamics Programmable: Challenges and Research Directions