# LexAgentHallu: A Hierarchical Benchmark for Profiling Hallucinations in Legal Agents

arXiv: https://arxiv.org/abs/2609.09754 · HTML: https://mlsanity.com/paper/2609.09754/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use.
- Builds on 17 of its 154 references that are in the map.
- First posted on arXiv 9 Sep 2026.

## Placed in

- LLM agents and tool use / [Medical and scientific domain agents](https://mlsanity.com/agents/medical-and-scientific-domain-agents/)

## Builds on (most cited first, 17 of 17)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks](https://mlsanity.com/paper/2412.14161/) (2024)
- [DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Services](https://mlsanity.com/paper/2309.11325/) (2023)
- [LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directions](https://mlsanity.com/paper/2509.18970/) (2025)
- [Fine-tuning Large Language Models for Improving Factuality in Legal Question Answering](https://mlsanity.com/paper/2501.06521/) (2025)
- [Small Agent Can Also Rock! Empowering Small Language Models as Hallucination Detector](https://mlsanity.com/paper/2406.11277/) (2024)
- [CitaLaw: Enhancing LLM with Citations in Legal Domain](https://mlsanity.com/paper/2412.14556/) (2024)
- [Ready Jurist One: Benchmarking Language Agents for Legal Intelligence in Dynamic Environments](https://mlsanity.com/paper/2507.04037/) (2025)
- [Unilaw-R1: A Large Language Model for Legal Reasoning with Reinforcement Learning and Iterative Inference](https://mlsanity.com/paper/2510.10072/) (2025)
- [LRAS: Advanced Legal Reasoning with Agentic Search](https://mlsanity.com/paper/2601.07296/) (2026)
- [Challenges for Generative AI in Legal Reasoning](https://mlsanity.com/paper/2508.18880/) (2025)
- [GLARE: Agentic Reasoning for Legal Judgment Prediction](https://mlsanity.com/paper/2508.16383/) (2025)
- [SkillProx: Self-Evolving Agent Skills via Proximal Textual Gradient Descent](https://mlsanity.com/paper/2608.07449/) (2026)
- [LawThinker: A Deep Research Legal Agent in Dynamic Environments](https://mlsanity.com/paper/2602.12056/) (2026)

## Abstract (arXiv)

As large language models are increasingly deployed as tool-augmented legal agents, they introduce agentic hallucinations where tool-call and reasoning errors cascade into fabricated holdings and miscited authority. However, existing legal benchmarks evaluate only single-turn QA with outcome-level metrics, while agentic hallucination benchmarks lack legal-specific diagnostic capability. Neither answers to what extent and how a legal agent hallucinates along its trajectory. To address these limitations, we introduce LexAgentHallu, a legal agentic hallucination benchmark designed to evaluate to what extent and how legal agents fail along multi-step trajectories. Built through a four-stage expert-in-the-loop pipeline, LexAgentHallu contains 3414 instances across 17 legal categories and 6 task types. Each instance is annotated under a dual-layer hallucination taxonomy of 7 high-level categories and 27 fine-grained subclasses, covering both substantive errors and agent-procedural failures. We further design fine-grained metrics that quantify to what extent and localize how each failure occurs along an agent's execution path. Our evaluation across 18 proprietary and open-source agents uncovers a Right-Answer-Wrong-Reason effect and reveals that hallucination subclasses cluster rather than scatter, forming distinct agentic framework, legal task, and category profiles. These findings, invisible to outcome-level evaluation, validate the diagnostic power of LexAgentHallu for evaluating agentic hallucination in law.