mlsanityData to 7 Oct 2026

LegalHalluLens: Typed Hallucination Auditing and Calibrated Multi-Agent Debate for Trustworthy Legal AI

arXiv 2606.18021 ↗First posted 16 Jun 2026cs.AI cs.CL cs.LGcited by 1

On the map: LLM agents and tool use → Medical and scientific domain agents

Abstract (arXiv)

AI systems deployed in legal workflows hallucinate at rates that aggregate metrics report at ~52%, but this average conceals where errors concentrate and in which direction they run, leaving compliance officers without an actionable signal for trustworthy deployment. We present LegalHalluLens, an auditing framework with three components: typed hallucination profiles across four legally-motivated claim categories (numeric, temporal, obligation/entitlement, factual) over CUAD (Hendrycks et al., 2021); a Risk Direction Index (RDI) that reduces omission-versus-invention bias to a single deployment-comparable scalar; and a typed debate pipeline calibrated to both magnitudes and directions. Across 510 contracts and 249,252 clause-level instances we measure a within-model gap of approximately 38-40 pp between obligation/numeric and temporal claims that aggregate reporting hides, and show that two systems with matched 52% rates can carry opposite RDIs. The debate pipeline reduces fabricated detections by 45% with per-category gains tracking the diagnosis, matching commercial APIs with a substantially smaller backbone (4B active parameters). Typed profiles and RDI surface failure modes that aggregate metrics hide; we further show these diagnostics serve as calibration inputs for multi-agent debate pipelines, where Skeptic challenges and asymmetric gates targeted at measured failure modes outperform generically-tuned debate. The framework supports direction-aware procurement, accountability, and agent design for legal AI deployed in the wild.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use. Builds on 14 of its 24 references that are in the map. First posted on arXiv 16 Jun 2026.

Builds on

14 of 24 references.

References · 14

  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  2. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  3. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  4. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  5. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 328
  6. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 232
  7. 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 170
  8. 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 144
  9. 2024Lynx: An Open Source Hallucination Evaluation Modelcited by 55
  10. 2025Removal of Hallucination on Hallucination: Debate-Augmented RAGcited by 29
  11. 2024Counterfactual Debating with Preset Stances for Hallucination Elimination of LLMscited by 17
  12. 2025ContractEval: Benchmarking LLMs for Clause-Level Legal Risk Identification in Commercial Contractscited by 9
2 more
  1. 2025LeMAJ (Legal LLM-as-a-Judge): Bridging Legal Reasoning and LLM Evaluationcited by 9
  2. 2026Validate Your Authority: Benchmarking LLMs on Multi-Label Precedent Treatment Classificationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2025HalluGraph: Auditable Hallucination Detection for Legal RAG Systems via Knowledge Graph Alignment
  2. 2026Who Checks the Citations? Benchmarking Legal Hallucination Detection
  3. 2026LexAgentHallu: A Hierarchical Benchmark for Profiling Hallucinations in Legal Agents
  4. 2026When Citations Mislead? A Claim-Level Benchmark for Legal Hallucination Detection
  5. 2026CLAIR-Fin: An Adversarial Multi-Agent Framework for Claim-Level Verification and Adaptive Debate in Cross-Modal Financial QA
  6. 2024Towards Detecting LLMs Hallucination via Markov Chain-based Multi-agent Debate Framework
  7. 2026Tool Receipts, Not Zero-Knowledge Proofs: Practical Hallucination Detection for AI Agents
  8. 2025Fine-tuning Large Language Models for Improving Factuality in Legal Question Answering