# AgentHallu: Benchmarking Automated Hallucination Attribution of LLM-based Agents

arXiv: https://arxiv.org/abs/2601.06818 · HTML: https://mlsanity.com/paper/2601.06818/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use.
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 11 Jan 2026.

## Placed in

- LLM agents and tool use / [Medical and scientific domain agents](https://mlsanity.com/agents/medical-and-scientific-domain-agents/)

## Builds on (most cited first, 26 of 26)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate](https://mlsanity.com/paper/2305.19118/) (2023)
- [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024)
- [The Flan Collection: Designing Data and Methods for Effective Instruction Tuning](https://mlsanity.com/paper/2301.13688/) (2023)
- [Executable Code Actions Elicit Better LLM Agents](https://mlsanity.com/paper/2402.01030/) (2024)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models](https://mlsanity.com/paper/2401.00396/) (2023)
- [Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems](https://mlsanity.com/paper/2505.00212/) (2025)
- [Fine-grained Hallucination Detection and Editing for Language Models](https://mlsanity.com/paper/2401.06855/) (2024)
- [OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automation](https://mlsanity.com/paper/2505.23885/) (2025)
- [SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistency](https://mlsanity.com/paper/2311.01740/) (2023)
- [Open Deep Search: Democratizing Search with Open-source Reasoning Agents](https://mlsanity.com/paper/2503.20201/) (2025)
- [Large Language Model-based Human-Agent Collaboration for Complex Task Solving](https://mlsanity.com/paper/2402.12914/) (2024)
- [GUARDIAN: Safeguarding LLM Multi-Agent Collaborations with Temporal Graph Modeling](https://mlsanity.com/paper/2505.19234/) (2025)
- [HALoGEN: Fantastic LLM Hallucinations and Where to Find Them](https://mlsanity.com/paper/2501.08292/) (2025)
- [ICR Probe: Tracking Hidden State Dynamics for Reliable Hallucination Detection in LLMs](https://mlsanity.com/paper/2507.16488/) (2025)
- [HD-NDEs: Neural Differential Equations for Hallucination Detection in LLMs](https://mlsanity.com/paper/2506.00088/) (2025)

## Built on by (most cited first, 16 of 16)

- [Delayed Verification Destabilizes Multi-Agent LLM Belief: Instability Thresholds and Optimal Corrector Placement](https://mlsanity.com/paper/2606.27409/) (2026)
- [The Hallucination Snowball: Modeling Error Propagation as State Transitions in Multi-Agent LLM Pipelines](https://mlsanity.com/paper/2608.14588/) (2026)
- [HalluWorld: A Controlled Benchmark for Hallucination via Reference World Models](https://mlsanity.com/paper/2605.19341/) (2026)
- [Hallucination as Context Drift: Synchronization Protocols for Multi-Agent LLM Systems](https://mlsanity.com/paper/2606.21666/) (2026)
- [The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Project](https://mlsanity.com/paper/2603.21354/) (2026)
- [Bounded Autonomy for Enterprise AI: Typed Action Contracts and Consumer-Side Execution](https://mlsanity.com/paper/2604.14723/) (2026)
- [Benchmarking Source-Sensitive Reasoning in Turkish: Humans and LLMs under Evidential Trust Manipulation](https://mlsanity.com/paper/2604.24665/) (2026)
- [Guardrails as Scapegoats: Auditing Unfaithful Safety Refusals in Tool-Augmented LLM Agents](https://mlsanity.com/paper/2607.19449/) (2026)
- [When Do Hallucinations Arise? A Graph Perspective on the Evolution of Path Reuse and Path Compression](https://mlsanity.com/paper/2604.03557/) (2026)
- [Cascading Hallucination in Agentic RAG: The CHARM Framework for Detection and Mitigation](https://mlsanity.com/paper/2606.04435/) (2026)
- [Rethinking Scientific Discovery in the Agentic Era](https://mlsanity.com/paper/2607.03863/) (2026)
- [Tracing the Cascade: A Topology-Aware Evaluation Framework for Scientific Agent Hallucinations](https://mlsanity.com/paper/2608.00711/) (2026)
- [Search-G1: Grounded Search Agents via Representation-Based Intrinsic Rewards](https://mlsanity.com/paper/2608.07531/) (2026)
- [Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Study](https://mlsanity.com/paper/2609.07395/) (2026)
- [When Should a Human Take Back Control? Optimal Delegation under Turbulent AI Risk](https://mlsanity.com/paper/2609.32083/) (2026)
- [Understanding and Mitigating Hallucination Escape in Tool-Using LLM Agents](https://mlsanity.com/paper/2610.04409/) (2026)

## Abstract (arXiv)

As LLM-based agents operate over sequential multi-step reasoning, hallucinations arising at intermediate steps risk propagating along the trajectory, thus degrading overall reliability. Unlike hallucination detection in single-turn responses, diagnosing hallucinations in multi-step workflows requires identifying which step causes the initial divergence. To fill this gap, we propose a new research task, automated hallucination attribution of LLM-based agents, aiming to identify the step responsible for the hallucination and explain why. To support this task, we introduce AgentHallu, a comprehensive benchmark with: (1) 693 high-quality trajectories spanning 7 agent frameworks and 5 domains, (2) a hallucination taxonomy organized into 5 categories (Planning, Retrieval, Reasoning, Human-Interaction, and Tool-Use) and 14 sub-categories, and (3) multi-level annotations curated by humans, covering binary labels, hallucination-responsible steps, and causal explanations. We evaluate 13 leading models, and results show the task is challenging even for top-tier models (like GPT-5, Gemini-2.5-Pro). The best-performing model achieves only 41.1% step localization accuracy, where tool-use hallucinations are the most challenging at just 11.6%. We believe AgentHallu will catalyze future research into developing robust, transparent, and reliable agentic systems.