mlsanityData to 14 Oct 2026

Diagnostics of cognitive failures in multi-agent expert systems using dynamic evaluation protocols and subsequent mutation of the processing context

arXiv 2509.15366 ↗First posted 18 Sep 2025cs.AIcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

The rapid evolution of neural architectures - from multilayer perceptrons to large-scale Transformer-based models - has enabled language models (LLMs) to exhibit emergent agentic behaviours when equipped with memory, planning, and external tool use. However, their inherent stochasticity and multi-step decision processes render classical evaluation methods inadequate for diagnosing agentic performance. This work introduces a diagnostic framework for expert systems that not only evaluates but also facilitates the transfer of expert behaviour into LLM-powered agents. The framework integrates (i) curated golden datasets of expert annotations, (ii) silver datasets generated through controlled behavioural mutation, and (iii) an LLM-based Agent Judge that scores and prescribes targeted improvements. These prescriptions are embedded into a vectorized recommendation map, allowing expert interventions to propagate as reusable improvement trajectories across multiple system instances. We demonstrate the framework on a multi-agent recruiter-assistant system, showing that it uncovers latent cognitive failures - such as biased phrasing, extraction drift, and tool misrouting - while simultaneously steering agents toward expert-level reasoning and style. The results establish a foundation for standardized, reproducible expert behaviour transfer in stochastic, tool-augmented LLM agents, moving beyond static evaluation to active expert system refinement.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 41 of its 56 references that are in the map. First posted on arXiv 18 Sep 2025.

Builds on

41 of 56 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios22 May
  2. 2025Survey on Evaluation of LLM-based Agents20 Mar
  3. 2024Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecture21 Nov
  4. 2024Agent-as-a-Judge: Evaluate Agents with Agents14 Oct

Other references · 37

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  10. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  12. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
25 more
  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  7. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  8. 2020GLU Variants Improve Transformercited by 1,804
  9. 2024A Survey on LLM-as-a-Judgecited by 1,423
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  11. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  12. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  13. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  14. 2020On Layer Normalization in the Transformer Architecturecited by 922
  15. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  16. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
  17. 2023RestGPT: Connecting Large Language Models with Real-World RESTful APIscited by 135
  18. 2022Holistic Evaluation of Language Modelscited by 131
  19. 2024Automated Unit Test Improvement using Large Language Models at Metacited by 75
  20. 2025ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenariocited by 55
  21. 2023Pre-RMSNorm and Pre-CRMSNorm Transformers: Equivalent and Efficient Pre-LN Transformerscited by 30
  22. 2025VRoPE: Rotary Position Embedding for Video Large Language Modelscited by 18
  23. 2025Episodic Memories Generation and Evaluation Benchmark for Large Language Modelscited by 14
  24. 2024Reflection-Bench: Evaluating Epistemic Agency in Large Language Modelscited by 7
  25. 2021Position Masking for Improved Layout-Aware Document Understandingcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Self-Improving Error Diagnosis in Multi-Agent Systems
  2. 2025AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?
  3. 2026Agent-as-a-Judge
  4. 2025Where LLM Agents Fail and How They can Learn From Failures
  5. 2025Towards Cognitive Synergy in LLM-Based Multi-Agent Systems: Integrating Theory of Mind and Critical Evaluation
  6. 2026Beyond the Leaderboard: A Synthesis of Tool-Use, Planning, and Reasoning Failures in Large Language Model Agents
  7. 2026The Long-Horizon Task Mirage? Diagnosing Where and Why Agentic Systems Break
  8. 2025Agentic Systems: A Guide to Transforming Industries with Vertical AI Agents