mlsanityData to 14 Oct 2026

Conformal Agent Error Attribution

arXiv 2605.06788 ↗First posted 7 May 2026cs.LG cs.MAcited by 4

On the map: LLM agents and tool use → Software engineering coding agents

Abstract (arXiv)

When multi-agent systems (MAS) fail, identifying where the decisive error occurred is the first step for automated recovery to an earlier state. Error attribution remains a fundamental challenge due to the long interaction traces that large language model-based MAS generate. This paper presents a framework for error attribution based on conformal prediction (CP) which provides finite-sample, distribution-free coverage guarantees. We introduce new algorithms for filtration-based CP designed for sequential data such as agent trajectories. Unlike existing CP algorithms, our approach predicts sets that are contiguous sequences to enable efficient recovery and debugging. We verify our theoretical guarantees on a variety of agents and datasets, show that errors can be precisely isolated, then use prediction sets to rollback MAS to correct their own errors. Our overall approach is model-agnostic, and offers a principled uncertainty layer for MAS error attribution. We release code at https://github.com/layer6ai-labs/conformal-agent-error-attribution.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. Builds on 27 of its 33 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

27 of 33 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Where Did It All Go Wrong? A Hierarchical Look into Multi-Agent Error Attribution6 Oct
  2. 2025CORRECT: COndensed eRror RECognition via knowledge Transfer in multi-agent systems28 Sep
  3. 2025Aegis: Automated Error Generation and Attribution for Multi-Agent Systems17 Sep
  4. 2025Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems30 Apr
  5. 2025Why Do Multi-Agent LLM Systems Fail?17 Mar

Other references · 22

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  3. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,808
  4. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
  5. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 769
  6. 2020Classification with Valid and Adaptive Coveragecited by 396
  7. 2021Adaptive Conformal Inference Under Distribution Shiftcited by 375
  8. 2020Uncertainty Sets for Image Classifiers using Conformal Predictioncited by 366
  9. 2023A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaborationcited by 290
  10. 2024Scaling Large Language Model-based Multi-Agent Collaborationcited by 241
  11. 2024FinCon: A Synthesized LLM Multi-Agent System with Conceptual Verbal Reinforcement for Enhanced Financial Decision Makingcited by 113
  12. 2019Nested conformal prediction and quantile out-of-bag ensemble methodscited by 89
10 more
  1. 2023Large Language Models are Diverse Role-Players for Summarization Evaluationcited by 62
  2. 2023Conformal Prediction for Deep Classifier via Label Rankingcited by 53
  3. 2024Conformal Prediction Sets Improve Human Decision Makingcited by 35
  4. 2025RAFFLES: Reasoning-based Attribution of Faults for LLM Systemscited by 21
  5. 2024Conformal Prediction Sets Can Cause Disparate Impactcited by 14
  6. 2025Classifying and Addressing the Diversity of Errors in Retrieval-Augmented Generation Systemscited by 10
  7. 2025Textual Bayes: Quantifying Prompt Uncertainty in LLM-Based Systemscited by 8
  8. 2015Temporal Convolutional Neural Networks for Diagnosis from Lab Testscited by 6
  9. 2025Document Summarization with Conformal Importance Guaranteescited by 6
  10. 2025Hierarchical Conformal Classificationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Conf-Gen: Conformal Uncertainty Quantification for Generative Modelscited by 1
  2. 2026Agentic Monte Carlo: Simulating Reinforcement Learning for Black-Box Agentscited by 1
  3. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Studycited by 0
  4. 2026G-CARB: Graph-Localized Conformal Agent Risk Budget for Compositional Harmcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systems
  2. 2026Error-Propagation Modeling for Failure Attribution in LLM-Based Multi-Agent Systems
  3. 2026Adaptive Influence Graphs for Failure Attribution in Multi-Agent Systems
  4. 2026Seeing the Whole Elephant: A Benchmark for Failure Attribution in LLM-based Multi-Agent Systems
  5. 2026Rethinking Failure Attribution in Multi-Agent Systems: A Multi-Perspective Benchmark and Evaluation
  6. 2025Who is Introducing the Failure? Automatically Attributing Failures of Multi-Agent Systems via Spectrum Analysis
  7. 2026MASPrism: Lightweight Failure Attribution for Multi-Agent Systems Using Prefill-Stage Signals
  8. 2026Towards Self-Improving Error Diagnosis in Multi-Agent Systems