Logarithmic Scores, Power-Law Discoveries: Disentangling Measurement from Coverage in Agent-Based Evaluation
On the map: LLM agents and tool use → Social simulation and multi-agent debate
Abstract (arXiv)
LLM-based agent judges are an emerging approach to evaluating conversational AI, yet a fundamental uncertainty remains: can we trust their assessments, and if so, how many are needed? Through 960 sessions with two model pairs across 15 tasks, we show that persona-based agent judges produce evaluations indistinguishable from human raters in a Turing-style validation. We then identify a score-coverage dissociation: quality scores improve logarithmically with panel size, while unique issue discoveries follow a sublinear power law-both exhibit diminishing returns, but scores saturate roughly twice as fast as discoveries. We hypothesize this reflects a power law distribution of the finding space: critical issues are discovered first by small panels, while corner cases require progressively larger panels, analogous to species accumulation curves in ecology. The mechanism traces to ensemble diversity-Big Five personality conditioning makes agents probe different quality dimensions, with expert judges acting as adversarial probes that push discovery into the tail of the finding distribution. A controlled ablation confirms that structured persona conditioning, not simple prompting, is required to produce these scaling properties.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 16 of its 32 references that are in the map. First posted on arXiv 1 Apr 2026.
Builds on
16 of 32 references.
References · 16
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
- 2023Large Language Models are not Fair Evaluatorscited by 1,050
- 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
- 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
- 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
- 2024Agent-as-a-Judge: Evaluate Agents with Agentscited by 223
- 2023Personality Traits in Large Language Modelscited by 166
- 2023A Unified Theory of Diversity in Ensemble Learningcited by 71
- 2025Assessing Judging Bias in Large Reasoning Models: An Empirical Studycited by 30
- 2025IntellAgent: A Multi-Agent Framework for Evaluating Conversational AI Systemscited by 25
- 2024Can LLM "Self-report"?: Evaluating the Validity of Self-report Scales in Measuring Personality Design in LLM-based Chatbotscited by 19
4 more
- 2025When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMscited by 16
- 2026Agent-as-a-Judgecited by 15
- 2025UXAgent: A System for Simulating Usability Testing of Web Design with LLM Agentscited by 10
- 2025Evaluating Large Language Models as Expert Annotatorscited by 9
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025The Social Laboratory: A Psychometric Framework for Multi-Agent LLM Evaluation
- 2026GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents
- 2025Agent-Testing Agent: A Meta-Agent for Automated Testing and Evaluation of Conversational AI Agents
- 2026Beyond Symmetric Agents: Cognitive Diversity and Multi-Agent Debate in Small Language Models
- 2026AgentWorld: Personality-Aware Reliability Evaluation for Agentic Information Retrieval
- 2026Inducing Reward-Free Judging Rubrics that Reduce Over-Crediting in Agent Evaluation
- 2026The Confident Liar: Diagnosing Multi-Agent Debate with Log-Probabilities and LLM-as-Judge
- 2026Beyond Consensus: Downward Bias and Role Asymmetry in Multi-Agent LLM Judges for Subjective Evaluation