mlsanityData to 14 Oct 2026

Knowledge-Based Zero-Replay Debugging of Multi-Agent LLM Traces

arXiv 2606.14805 ↗First posted 11 Jun 2026cs.SE cs.AIcited by 2

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

Reliable operation of multi-agent large language model (LLM) systems depends on debugging long execution traces, where the few causally decisive events are buried in unstructured logs of messages, routes, memory writes, and tool calls. The standard tool is counterfactual replay (rewind, edit, and re-run the trajectory to measure each event's effect), but its cost grows linearly with the number of candidate events, making exhaustive replay infeasible at scale. We frame trace debugging as a knowledge-based decision-support problem. Each trace is compiled into a structured event knowledge graph over routing, memory, tool-use, uncertainty, and latent evidence, and a calibrated predictor decides where a scarce replay budget should be spent. We do not propose a new replay oracle; we propose a method to predict its results without paying the replay cost. We formulate zero-replay counterfactual-effect prediction: given a trace under a fixed budget, predict which events the oracle would mark high-effect before any replay is performed. BranchPoint-Latent is a lightweight predictor over observable, structural, uncertainty, and latent features of the knowledge graph. Calibrated against a deterministic replay oracle across 37 trace families, a single learning-to-rank gradient-boosted predictor raises per-trace localization (Branch Recall@5) from 0.73 to 0.93 on held-out families at zero oracle-replay cost. Rather than claiming universal dominance, we characterize when cheap graph centrality suffices and when learned evidence is necessary. The result is an auditable, cost-efficient decision-support system for AI-reliability debugging, positioned explicitly on the cost-accuracy frontier with reproducible artifacts.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 23 of its 29 references that are in the map. First posted on arXiv 11 Jun 2026.

Builds on

23 of 29 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026MASPrism: Lightweight Failure Attribution for Multi-Agent Systems Using Prefill-Stage Signals8 May
  2. 2026Seeing the Whole Elephant: A Benchmark for Failure Attribution in LLM-based Multi-Agent Systems24 Apr
  3. 2025AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?3 Sep
  4. 2025TRAIL: Trace Reasoning and Agentic Issue Localization13 May
  5. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems30 Apr
  6. 2025Interactive Debugging and Steering of Multi-Agent AI Systems3 Mar

Other references · 17

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  8. 2021Program Synthesis with Large Language Modelscited by 3,637
  9. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  11. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  12. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
5 more
  1. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  2. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 743
  3. 2026Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnetcited by 191
  4. 2026Reasoning Models Struggle to Control their Chains of Thoughtcited by 25
  5. 2026WhatIf: Interactive Exploration of LLM-Powered Social Simulations for Policy Reasoningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems
  2. 2026Repair or Resample? Rethinking Failure Debugging in LLM Multi-Agent Systems
  3. 2026PrefixGuard: From LLM-Agent Traces to Online Failure-Warning Monitors
  4. 2026Early Diagnosis of Wasted Computation in Multi-Agent LLM Systems via Failure-Aware Observability
  5. 2026Automata from Agent Traces: Failure and Next-Step Prediction
  6. 2026Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents
  7. 2026Doomed from the Start: Early Abort of LLM Agent Episodes via a Recall-Controlled Probe Cascade
  8. 2026TraceCompiler: Skill-Guided Mining and Compilation of LLM Agent Traces into Mostly Deterministic Workflows