mlsanityData to 7 Oct 2026

AgentCheck: A Reproduce-Intervene-Mitigate Workbench for LLM Agents over MCP

arXiv 2607.11098 ↗First posted 13 Jul 2026cs.SE cs.AI cs.CLcited by 4

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Tool-using LLM agents are mostly evaluated assuming all tools work. When a tool times out, returns a week-stale value, or has its description poisoned in deployment, the developer needs a controlled way to reproduce the failure, test a fix, and confirm the fix worked before deployment. We present AgentCheck, an open-source web workbench that turns an MCP server into an intervention surface. AgentCheck runs an agent against its real tools and records every tool response, then re-runs the agent with the response perturbed by a fault (12 types) injector. Matching tool calls are replayed from cache, and later tool calls go live after the agent diverges. This yields a reproduce-intervene-confirm loop: the developer toggles a mitigation, re-runs against the identical fault, and sees if the failure goes away. Scoring has two parts: deterministic pass/fail rules, plus an LLM judge for interpretive labels, validated against human annotations. Across five agents, the best passes 105/120 scenarios and the weakest only 77. The failures are usually silent, confident use of incorrect tool outputs rather than crashes. On the weakest agent, a retry mitigation raises success on timeout error faults from as few as 30% of cases to 100%, whereas stale-data faults remain near 3-4 of 10 regardless of the mitigation. AgentCheck makes these failure modes reproducible, comparable, and verifiable before deployment.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 18 of its 29 references that are in the map. First posted on arXiv 13 Jul 2026.

Builds on

18 of 29 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026From Confident Closing to Silent Failure: Characterizing False Success in LLM Agents1 Jun
  2. 2025Why Do Multi-Agent LLM Systems Fail?17 Mar
  3. 2025Interactive Debugging and Steering of Multi-Agent AI Systems3 Mar

Other references · 15

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  5. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,280
  6. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  7. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  8. 2025MCPSecBench: A Systematic Security Benchmark and Playground for Testing Model Context Protocolscited by 47
  9. 2026Breaking the Protocol: Security Analysis of the Model Context Protocol Specification and Prompt Injection Vulnerabilities in Tool-Integrated LLM Agentscited by 18
  10. 2026When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agentscited by 15
  11. 2026PlanBench-XL: Evaluating Long-Horizon Planning of LLM Tool-Use Agents in Large-Scale Tool Ecosystemscited by 10
  12. 2026Auditing Automated Evaluation, Error Propagation, and Runtime Mitigation in Tool-Using Language Agentscited by 9
3 more
  1. 2025How Do LLMs Fail In Agentic Scenarios? A Qualitative Analysis of Success and Failure Scenarios of Various LLMs in Agentic Simulationscited by 5
  2. 2026TRUSTDESC: Preventing Tool Poisoning in LLM Applications via Trusted Description Generationcited by 3
  3. 2026An Evaluation of Data Leakage Risks in Tool-Using LLM Agents in Realistic Scenarioscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failurescited by 6
  2. 2026Testing and Evaluation of Agentic AI Systems In Military Command and Controlcited by 0
  3. 2026What Stops a Small Language Model From Driving a Database Agentcited by 0
  4. 2026Where Does Exactly-Once Live? Model, Harness, and Tool-Contract Effects on Duplicate Side Effects in LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AgentTether: Graph-Guided Diagnosis and Runtime Intervention for Reliable LLM Agent Operation
  2. 2026Probe-and-Refine Tuning of Repository Guidance for Coding Agents
  3. 2025Agentic Program Repair from Test Failures at Scale: A Neuro-symbolic approach with static analysis and test execution feedback
  4. 2026Verified Tool Calls Improve LLM Agent Reliability Under Non-Atomic Failures
  5. 2026AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents
  6. 2026ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress Conditions
  7. 2026Automated structural testing of LLM-based agents: methods, framework, and case studies
  8. 2026AgentFixer: From Failure Detection to Fix Recommendations in LLM Agentic Systems