mlsanityData to 7 Oct 2026

Real-Time Detection and Repair of LLM Agent Failures

arXiv 2608.02464 ↗First posted 3 Aug 2026cs.AI cs.LG cs.SEcited by 2

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

LLM agents fail mid-episode -- they loop, cascade tool errors, drift off goal, fabricate results, or silently absorb corrupted content -- and the standard remedy, judging every step with a second LLM, costs more than the agent itself. We ask how much detection is achievable from observable step telemetry alone, using monitors costing microseconds per step and trained only on healthy runs. On 2,823 committed agent episodes across three frameworks, three local models (qwen2.5 7b/3b, llama3.1 8b) and a commercial API (gemini-2.5-flash), a one-class echo-state-network ensemble with CUSUM alarms detects 0.71 of failures at a 5% false-alarm budget (AUROC 0.872). Its advantage over a memoryless baseline is a monotone function of post-onset horizon (+0.09 at <=3 steps, +0.40 at >=9), predicting its own failure region out of sample on AFTraj-2K. Ranking transfers with no retraining to two corpora from other groups (AFTraj-2K 0.745, ATBench 0.779). Monitors carry two burdens: a per-deployment healthy null (they do not transfer -- AUROC 0.527 cold against 0.885 recalibrated) and a residual false-alarm rate. We add a layer carrying neither: deterministic verification, which recomputes a run's stated total from the tool results it actually received and confirms every required call was made. Head-to-head it catches 60% of failures (96% with the coverage check) at 0 of 63 false positives against the monitor's 54% at 17%, transfers unchanged to llama3.1:8b (110 of 110 at 0 of 10), and trips on 0 of 1825 healthy episodes. Detection is then closed into repair: each flagged run is rolled back and re-run live, recovering 45% of failures against a 16% resampling control (p=0.0005) and lifting task success from 52% to 73% for about one extra model call per run. The system runs at ~200 microseconds per step, three orders of magnitude below a judge call. Code, traces and results are released.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 20 of its 33 references that are in the map. First posted on arXiv 3 Aug 2026.

Builds on

20 of 33 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026AgentTether: Graph-Guided Diagnosis and Runtime Intervention for Reliable LLM Agent Operation7 Jul
  2. 2026When Evidence is Sparse: Weakly Supervised Early Failure Alerting in Dialogs and LLM-Agent Trajectories3 Jun
  3. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems30 Apr
  4. 2025Why Do Multi-Agent LLM Systems Fail?17 Mar
  5. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation16 Aug

Other references · 15

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  5. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  6. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  7. 2024A Survey on LLM-as-a-Judgecited by 1,384
  8. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  9. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  10. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
  11. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
  12. 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agentscited by 196
3 more
  1. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  2. 2026Trajectory Guard -- A Lightweight, Sequence-Aware Model for Real-Time Anomaly Detection in Agentic AIcited by 10
  3. 2026Doomed from the Start: Early Abort of LLM Agent Episodes via a Recall-Controlled Probe Cascadecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Reliability Engineering for AI Systems: Challenges, Methods, and Directionscited by 0
  2. 2026What Can Component-Replacement Evidence Establish? A Critical Scoping Review of Local Decisions in LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AgentFixer: From Failure Detection to Fix Recommendations in LLM Agentic Systems
  2. 2026TelemetrySuffBench: Is Agent Telemetry Sufficient for Failure-Origin Diagnosis?
  3. 2026When Errors Become Narratives: A Longitudinal Taxonomy of Silent Failures in a Production LLM Agent Runtime
  4. 2026AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents
  5. 2026MemRiskBench: Trace-Aware Risk-Preserving Evaluation for Long-Horizon LLM Agents
  6. 2026Repair or Resample? Rethinking Failure Debugging in LLM Multi-Agent Systems
  7. 2026Early Diagnosis of Wasted Computation in Multi-Agent LLM Systems via Failure-Aware Observability
  8. 2026When Agentic Executions Fail: Detecting and Localizing Runtime Faults from Telemetry