mlsanityData to 14 Oct 2026

How Much Do Legal RAG Systems Still Hallucinate?

arXiv 2608.14210 ↗First posted 14 Aug 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Hallucination is a major challenge for retrieval-augmented generation (RAG) systems in the legal domain, where ungrounded answers can lead to serious consequences. To better understand this problem, we conduct a fine-grained analysis of hallucination behavior in eight legal RAG systems across two legal corpora, the GDPR (in English) and a national civil law (in French). Using claim-level and answer-level evaluation, we report on hallucination density and severity, analyze performance across question categories and user personas, and validate our findings on an independent set of 142 legal-expert-authored questions. Our results show that hallucinations remain pervasive, ranging from less than 10% of responses for the best-performing systems to nearly half in the worst case. We further find that false-premise questions, containing incorrect assumptions that must be rejected, produce high hallucination rates on the manually-drafted questions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 22 of its 32 references that are in the map. First posted on arXiv 14 Aug 2026.

Builds on

22 of 32 references.

References · 22

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  2. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  3. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  4. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 310
  5. 2023Improving Text Embeddings with Large Language Modelscited by 303
  6. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 244
  7. 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 176
  8. 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 150
  9. 2024LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domaincited by 118
  10. 2025HalluLens: LLM Hallucination Benchmarkcited by 92
  11. 2024LLMs Will Always Hallucinate, and We Need to Live With Thiscited by 78
  12. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
10 more
  1. 2024RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generationcited by 61
  2. 2023Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Modelscited by 35
  3. 2024FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extractioncited by 28
  4. 2025Fine-tuning Large Language Models for Improving Factuality in Legal Question Answeringcited by 28
  5. 2024RefChecker: Reference-based Fine-grained Hallucination Checker and Benchmark for Large Language Modelscited by 25
  6. 2024Reference-free Hallucination Detection for Large Vision-Language Modelscited by 23
  7. 2025LUMINA: Detecting Hallucinations in RAG System with Context-Knowledge Signalscited by 9
  8. 2024Persona-SQ: A Personalized Suggested Question Generation Framework For Real-world Documentscited by 3
  9. 2026Detecting Hallucinations in Retrieval-Augmented Generation via Semantic-level Internal Reasoning Graphcited by 2
  10. 2026Fine-grained Claim-level RAG Benchmark for Lawcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Removal of Hallucination on Hallucination: Debate-Augmented RAG
  2. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models
  3. 2025Real-Time Evaluation Models for RAG: Who Detects Hallucinations Best?
  4. 2025Bi'an: A Bilingual Benchmark and Model for Hallucination Detection in Retrieval-Augmented Generation
  5. 2026Beyond Document Grounding: Span-Level Hallucination Detection over Code, Tool Output, and Documents
  6. 2026Mitigating Hallucination on Hallucination in RAG via Ensemble Voting
  7. 2025LettuceDetect: A Hallucination Detection Framework for RAG Applications
  8. 2025MetaRAG: Metamorphic Testing for Hallucination Detection in RAG Systems