mlsanityData to 7 Oct 2026

LexAgentHallu: A Hierarchical Benchmark for Profiling Hallucinations in Legal Agents

arXiv 2609.09754 ↗First posted 9 Sep 2026cs.AIcited by 0

On the map: LLM agents and tool use → Medical and scientific domain agents

Abstract (arXiv)

As large language models are increasingly deployed as tool-augmented legal agents, they introduce agentic hallucinations where tool-call and reasoning errors cascade into fabricated holdings and miscited authority. However, existing legal benchmarks evaluate only single-turn QA with outcome-level metrics, while agentic hallucination benchmarks lack legal-specific diagnostic capability. Neither answers to what extent and how a legal agent hallucinates along its trajectory. To address these limitations, we introduce LexAgentHallu, a legal agentic hallucination benchmark designed to evaluate to what extent and how legal agents fail along multi-step trajectories. Built through a four-stage expert-in-the-loop pipeline, LexAgentHallu contains 3414 instances across 17 legal categories and 6 task types. Each instance is annotated under a dual-layer hallucination taxonomy of 7 high-level categories and 27 fine-grained subclasses, covering both substantive errors and agent-procedural failures. We further design fine-grained metrics that quantify to what extent and localize how each failure occurs along an agent's execution path. Our evaluation across 18 proprietary and open-source agents uncovers a Right-Answer-Wrong-Reason effect and reveals that hallucination subclasses cluster rather than scatter, forming distinct agentic framework, legal task, and category profiles. These findings, invisible to outcome-level evaluation, validate the diagnostic power of LexAgentHallu for evaluating agentic hallucination in law.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use. Builds on 17 of its 154 references that are in the map. First posted on arXiv 9 Sep 2026.

Builds on

17 of 154 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directions23 Sep
  2. 2024Small Agent Can Also Rock! Empowering Small Language Models as Hallucination Detector17 Jun

Other references · 15

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  4. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  5. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 306
  6. 2023DISC-LawLLM: Fine-tuning Large Language Models for Intelligent Legal Servicescited by 117
  7. 2025Fine-tuning Large Language Models for Improving Factuality in Legal Question Answeringcited by 28
  8. 2024CitaLaw: Enhancing LLM with Citations in Legal Domaincited by 16
  9. 2025Ready Jurist One: Benchmarking Language Agents for Legal Intelligence in Dynamic Environmentscited by 13
  10. 2025Unilaw-R1: A Large Language Model for Legal Reasoning with Reinforcement Learning and Iterative Inferencecited by 10
  11. 2026LRAS: Advanced Legal Reasoning with Agentic Searchcited by 8
  12. 2025Challenges for Generative AI in Legal Reasoningcited by 7
3 more
  1. 2025GLARE: Agentic Reasoning for Legal Judgment Predictioncited by 6
  2. 2026LawThinker: A Deep Research Legal Agent in Dynamic Environmentscited by 3
  3. 2026SkillProx: Self-Evolving Agent Skills via Proximal Textual Gradient Descentcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026LegalHalluLens: Typed Hallucination Auditing and Calibrated Multi-Agent Debate for Trustworthy Legal AI
  2. 2025MIRAGE-Bench: LLM Agent is Hallucinating and Where to Find Them
  3. 2026Beyond Final Answers: Auditing Trajectory-Level Hallucinations in Multi-Agent Industrial Workflows
  4. 2026AgentHallu: Benchmarking Automated Hallucination Attribution of LLM-based Agents
  5. 2026Who Checks the Citations? Benchmarking Legal Hallucination Detection
  6. 2026Internal Representations as Indicators of Hallucinations in Agent Tool Selection
  7. 2026HalluClear: Diagnosing, Evaluating and Mitigating Hallucinations in GUI Agents
  8. 2026Tracing the Cascade: A Topology-Aware Evaluation Framework for Scientific Agent Hallucinations