mlsanityData to 14 Oct 2026

Legal LLM Hallucination Should Be Evaluated as Failure of Legal Warrant

arXiv 2609.17546 ↗First posted 15 Jul 2026cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

In this position paper, we argue that legal LLMs' hallucinations should be evaluated as a failure of legal warrant rather than as factual inaccuracy or citation failure. We define claim-authority warrant as the context-sensitive relation between a consequential legal claim and authority that exists, applies to the relevant jurisdiction, is current for the date of analysis, has the legal status represented by the system, and supports the proposition asserted. Warranted legal generation is the broader system behavior that answers, narrows, asks, warns, corrects a false premise, or abstains according to that relation. The falsifiable prediction is that warrant metrics reveal material failures that answer accuracy, citation existence, generic attribution, LegalHalBench-style statute relevance, and CitaLaw-style sentence-citation alignment can miss. We sharpen this claim with a side-by-side comparison item and a small, reproducible pilot over public-rule tests. We then specify benchmark records, claim boundaries, support labels, mixed response-policy scoring, risk weights, annotation reliability reporting, and jurisdiction-specific authority ontologies. The result is a concrete research agenda for evaluating legal AI systems by whether their consequential claims are licensed by law.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 15 of its 24 references that are in the map. First posted on arXiv 15 Jul 2026.

Builds on

15 of 24 references.

References · 15

  1. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  2. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  3. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  4. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 310
  5. 2021Measuring Attribution in Natural Language Generation Modelscited by 256
  6. 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 176
  7. 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 150
  8. 2024From Lists to Emojis: How Format Bias Affects Model Alignmentcited by 49
  9. 2025A Reasoning-Focused Legal Retrieval Benchmarkcited by 37
  10. 2025Fine-tuning Large Language Models for Improving Factuality in Legal Question Answeringcited by 28
  11. 2024CitaLaw: Enhancing LLM with Citations in Legal Domaincited by 16
  12. 2024Verifiable Generation with Subsentence-Level Fine-Grained Citationscited by 15
3 more
  1. 2024FactLens: Benchmarking Fine-Grained Fact Verificationcited by 10
  2. 2025Place Matters: Comparing LLM Hallucination Rates for Place-Based Legal Queriescited by 3
  3. 2026InfoGatherer: Principled Information Seeking via Evidence Retrieval and Strategic Questioningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.