mlsanityData to 14 Oct 2026

Designing Reliable LLM-as-a-Judge Measurement Systems for Multi-Turn Business Agents

arXiv 2609.33955 ↗First posted 27 Sep 2026cs.AIcited by 0

On the map: LLM agents and tool use → Social simulation and multi-agent debate also fits: Software engineering coding agents

Abstract (arXiv)

Many LLM-as-a-judge evaluations score fixed outputs under a fixed task definition. Production multi-turn business agents instead require a maintained measurement system: correctness depends on business-specific facts and procedures, outcomes emerge across turns, and failures must be attributed to either agent capability or missing business knowledge before they are actionable. We present an integrated methodology spanning evaluation specification, modular LLM judges, intent-preserving user simulation, and human-in-the-loop governance. The specification defines conversation-level end states and actionable failure ownership. Atomic judges share versioned evidence and feed an explicit aggregation graph. The simulator is released only after task-preservation and stability checks. Independent human audits estimate measurement fidelity, renew tiered reference sets, and route disagreements to label correction, guideline revision, or judge improvement. Production studies show that system-level fidelity improved across repeated audits, that human reviewers and automated judges improved together under the shared feedback loop, and that their combined workflow had the strongest descriptive performance in both reported task-completion settings. Because the studies are observational and the human reference itself required revision, these findings demonstrate operational usefulness rather than causal or universal superiority. The contribution is a practical framework for making multi-turn agent measurement reliable, actionable, and maintainable as the evaluated system and its evidence evolve.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 39 of its 45 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

39 of 45 references.

References · 39

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  2. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  3. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  4. 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
  5. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 575
  6. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  7. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 395
  8. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 356
  9. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  10. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
  11. 2024MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialoguescited by 250
  12. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 205
27 more
  1. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 201
  2. 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 144
  3. 2020Unsupervised Evaluation of Interactive Dialog with DialoGPTcited by 139
  4. 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 139
  5. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 128
  6. 2024Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answeringcited by 96
  7. 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 94
  8. 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 87
  9. 2024MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Modelscited by 81
  10. 2024CRMArena: Understanding the Capacity of LLM Agents to Perform Professional CRM Tasks in Realistic Environmentscited by 71
  11. 2024TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generationcited by 68
  12. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 57
  13. 2022Don't Blame the Annotator: Bias Already Starts in the Annotation Instructionscited by 47
  14. 2025RocketEval: Efficient Automated LLM Evaluation via Grading Checklistcited by 39
  15. 2024HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decompositioncited by 33
  16. 2026Autorubric: A Unifying Framework for Rubric-Based LLM Evaluation on Non-Verifiable Taskscited by 26
  17. 2024Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performancecited by 20
  18. 2024Automated test generation to evaluate tool-augmented LLMs as conversational AI agentscited by 17
  19. 2022Detecting Label Errors by using Pre-Trained Language Modelscited by 16
  20. 2024DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregationcited by 12
  21. 2024Evaluation and Continual Improvement for an Enterprise AI Assistantcited by 7
  22. 2025Humans Hallucinate Too: Language Models Identify and Correct Subjective Annotation Errors With Label-in-a-Haystack Promptscited by 5
  23. 2025EvalAssist: A Human-Centered Tool for LLM-as-a-Judgecited by 5
  24. 2026Ask, Don't Judge: Binary Questions for Interpretable LLM Evaluation and Self-Improvementcited by 5
  25. 2026Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agentscited by 4
  26. 2026Building Customer Support AI Agents at 100M-User Scale: An Evaluation-Driven Frameworkcited by 3
  27. 2026Shopping Reasoning Bench: An Expert-Authored Benchmark for Multi-Turn Conversational Shopping Assistantscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026When Agent Evaluation Becomes Unreliable: An Empirical Study of Oracle Failures and Configuration-Dependent Measurement in an LLM Agent Benchmark
  2. 2026GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents
  3. 2025When AIs Judge AIs: The Rise of Agent-as-a-Judge Evaluation for LLMs
  4. 2026The Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the Mean
  5. 2026Beyond Task Completion: Revealing Corrupt Success in LLM Agents through Procedure-Aware Evaluation
  6. 2026Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
  7. 2026Loop-Back Authority in LLM Agent Teams: A Paired Experiment on Flat and Hierarchical Coordination
  8. 2026AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling