Diagnostics of cognitive failures in multi-agent expert systems using dynamic evaluation protocols and subsequent mutation of the processing context
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
Abstract (arXiv)
The rapid evolution of neural architectures - from multilayer perceptrons to large-scale Transformer-based models - has enabled language models (LLMs) to exhibit emergent agentic behaviours when equipped with memory, planning, and external tool use. However, their inherent stochasticity and multi-step decision processes render classical evaluation methods inadequate for diagnosing agentic performance. This work introduces a diagnostic framework for expert systems that not only evaluates but also facilitates the transfer of expert behaviour into LLM-powered agents. The framework integrates (i) curated golden datasets of expert annotations, (ii) silver datasets generated through controlled behavioural mutation, and (iii) an LLM-based Agent Judge that scores and prescribes targeted improvements. These prescriptions are embedded into a vectorized recommendation map, allowing expert interventions to propagate as reusable improvement trajectories across multiple system instances. We demonstrate the framework on a multi-agent recruiter-assistant system, showing that it uncovers latent cognitive failures - such as biased phrasing, extraction drift, and tool misrouting - while simultaneously steering agents toward expert-level reasoning and style. The results establish a foundation for standardized, reproducible expert behaviour transfer in stochastic, tool-augmented LLM agents, moving beyond static evaluation to active expert system refinement.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 41 of its 56 references that are in the map. First posted on arXiv 18 Sep 2025.
Builds on
41 of 56 references.
Closest ancestors · 4
Same direction, newest first
- 2025AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios22 May
- 2025Survey on Evaluation of LLM-based Agents20 Mar
- 2024Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecture21 Nov
- 2024Agent-as-a-Judge: Evaluate Agents with Agents14 Oct
Other references · 37
- 2017Attention Is All You Needcited by 60,361
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
25 more
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2020Longformer: The Long-Document Transformercited by 3,413
- 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
- 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
- 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
- 2020GLU Variants Improve Transformercited by 1,804
- 2024A Survey on LLM-as-a-Judgecited by 1,423
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
- 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
- 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
- 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
- 2020On Layer Normalization in the Transformer Architecturecited by 922
- 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
- 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
- 2023RestGPT: Connecting Large Language Models with Real-World RESTful APIscited by 135
- 2022Holistic Evaluation of Language Modelscited by 131
- 2024Automated Unit Test Improvement using Large Language Models at Metacited by 75
- 2025ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenariocited by 55
- 2023Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformerscited by 30
- 2025VRoPE: Rotary Position Embedding for Video Large Language Modelscited by 18
- 2025Episodic Memories Generation and Evaluation Benchmark for Large Language Modelscited by 14
- 2024Reflection-Bench: Evaluating Epistemic Agency in Large Language Modelscited by 7
- 2021Position Masking for Improved Layout-Aware Document Understandingcited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Towards Self-Improving Error Diagnosis in Multi-Agent Systems
- 2025AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?
- 2026Agent-as-a-Judge
- 2025Where LLM Agents Fail and How They can Learn From Failures
- 2025Towards Cognitive Synergy in LLM-Based Multi-Agent Systems: Integrating Theory of Mind and Critical Evaluation
- 2026Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents
- 2026The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break
- 2025Agentic Systems: A Guide to Transforming Industries with Vertical AI Agents