LegalHalluLens: Typed Hallucination Auditing and Calibrated Multi-Agent Debate for Trustworthy Legal AI
On the map: LLM agents and tool use → Medical and scientific domain agents
Abstract (arXiv)
AI systems deployed in legal workflows hallucinate at rates that aggregate metrics report at ~52%, but this average conceals where errors concentrate and in which direction they run, leaving compliance officers without an actionable signal for trustworthy deployment. We present LegalHalluLens, an auditing framework with three components: typed hallucination profiles across four legally-motivated claim categories (numeric, temporal, obligation/entitlement, factual) over CUAD (Hendrycks et al., 2021); a Risk Direction Index (RDI) that reduces omission-versus-invention bias to a single deployment-comparable scalar; and a typed debate pipeline calibrated to both magnitudes and directions. Across 510 contracts and 249,252 clause-level instances we measure a within-model gap of approximately 38-40 pp between obligation/numeric and temporal claims that aggregate reporting hides, and show that two systems with matched 52% rates can carry opposite RDIs. The debate pipeline reduces fabricated detections by 45% with per-category gains tracking the diagnosis, matching commercial APIs with a substantially smaller backbone (4B active parameters). Typed profiles and RDI surface failure modes that aggregate metrics hide; we further show these diagnostics serve as calibration inputs for multi-agent debate pipelines, where Skeptic challenges and asymmetric gates targeted at measured failure modes outperform generically-tuned debate. The framework supports direction-aware procurement, accountability, and agent design for legal AI deployed in the wild.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use. Builds on 14 of its 24 references that are in the map. First posted on arXiv 16 Jun 2026.
Builds on
14 of 24 references.
References · 14
- 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
- 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
- 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
- 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
- 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 328
- 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 232
- 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 170
- 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 144
- 2024Lynx: An Open Source Hallucination Evaluation Modelcited by 55
- 2025Removal of Hallucination on Hallucination: Debate-Augmented RAGcited by 29
- 2024Counterfactual Debating with Preset Stances for Hallucination Elimination of LLMscited by 17
- 2025ContractEval: Benchmarking LLMs for Clause-Level Legal Risk Identification in Commercial Contractscited by 9
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025HalluGraph: Auditable Hallucination Detection for Legal RAG Systems via Knowledge Graph Alignment
- 2026Who Checks the Citations? Benchmarking Legal Hallucination Detection
- 2026LexAgentHallu: A Hierarchical Benchmark for Profiling Hallucinations in Legal Agents
- 2026When Citations Mislead? A Claim-Level Benchmark for Legal Hallucination Detection
- 2026CLAIR-Fin: An Adversarial Multi-Agent Framework for Claim-Level Verification and Adaptive Debate in Cross-Modal Financial QA
- 2024Towards Detecting LLMs Hallucination via Markov Chain-based Multi-agent Debate Framework
- 2026Tool Receipts, Not Zero-Knowledge Proofs: Practical Hallucination Detection for AI Agents
- 2025Fine-tuning Large Language Models for Improving Factuality in Legal Question Answering