# Who Checks the Citations? Benchmarking Legal Hallucination Detection

arXiv: https://arxiv.org/abs/2606.21155 · HTML: https://mlsanity.com/paper/2606.21155/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Medical and scientific domain agents, one of 16 directions in LLM agents and tool use.
- Builds on 19 of its 49 references that are in the map.
- First posted on arXiv 19 Jun 2026.

## Placed in

- LLM agents and tool use / [Medical and scientific domain agents](https://mlsanity.com/agents/medical-and-scientific-domain-agents/)

## Builds on (most cited first, 19 of 19)

- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools](https://mlsanity.com/paper/2405.20362/) (2024)
- [Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models](https://mlsanity.com/paper/2401.01301/) (2024)
- [olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models](https://mlsanity.com/paper/2502.18443/) (2025)
- [Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation](https://mlsanity.com/paper/2305.15852/) (2023)
- [Towards a Science of AI Agent Reliability](https://mlsanity.com/paper/2602.16666/) (2026)
- [PURR: Efficiently Editing Language Model Hallucinations by Denoising Language Model Corruptions](https://mlsanity.com/paper/2305.14908/) (2023)
- [Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models](https://mlsanity.com/paper/2407.04121/) (2024)
- [How Ready are Pre-trained Abstractive Models and LLMs for Legal Case Judgement Summarization?](https://mlsanity.com/paper/2306.01248/) (2023)
- [Explaining Legal Concepts with Augmented Large Language Models (GPT-4)](https://mlsanity.com/paper/2306.09525/) (2023)
- [ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks](https://mlsanity.com/paper/2310.12516/) (2023)
- [Fine-tuning Large Language Models for Improving Factuality in Legal Question Answering](https://mlsanity.com/paper/2501.06521/) (2025)
- [InterrogateLLM: Zero-Resource Hallucination Detection in LLM-Generated Answers](https://mlsanity.com/paper/2403.02889/) (2024)
- [Hallucination Detection in Foundation Models for Decision-Making: A Flexible Definition and Review of the State of the Art](https://mlsanity.com/paper/2403.16527/) (2024)
- [Agentic Forecasting using Sequential Bayesian Updating of Linguistic Beliefs](https://mlsanity.com/paper/2604.18576/) (2026)
- [HalluHard: A Hard Multi-Turn Hallucination Benchmark](https://mlsanity.com/paper/2602.01031/) (2026)
- [ABBEL: Learning Natural-Language Belief States for Memory-Efficient Interaction](https://mlsanity.com/paper/2512.20111/) (2025)
- [Bye-bye, Bluebook? Automating Legal Drudgery With AI-Augmented Rule Following](https://mlsanity.com/paper/2505.02763/) (2025)

## Built on by (most cited first, 2 of 2)

- [Is this Citation on Point?](https://mlsanity.com/paper/2608.12571/) (2026)
- [Cited but Not Consulted: A Counterfactual Audit of Legal Chain-of-Thought Faithfulness](https://mlsanity.com/paper/2610.12361/) (2026)

## Abstract (arXiv)

Attorneys, judges, and pro se filers increasingly use AI to draft legal documents, yet these tools frequently fabricate citations. Despite predictions that newer models would hallucinate less or that court sanctions would deter negligent filers, we found over 1,000 filings containing fabricated citations---with this number growing year-over-year. This study evaluates whether AI-based systems can mitigate these errors by automatically detecting hallucinations. We propose a taxonomy of legal citation hallucinations grounded in actual court filings and introduce a dataset of 1,300 brief excerpts containing injected errors. Benchmarking five models in agentic and non-agentic settings as well as Claude Code reveals that while the latest iterations perform better---GPT-5 achieves 84.4% recall and a 55.0% F1 score in an agentic framework---all models struggle with subtle error categories. Agentic verification remains resource-intensive, with GPT-5 averaging 15.3 steps per excerpt. Furthermore, restricted information access limits the efficacy of even the best agents. This gap creates policy concerns, as it disadvantages both AI systems and litigants who lack subscriptions to commercial legal databases. Together, our dataset, tools, and policy recommendations provide a foundation for building and auditing reliable legal citation checking tools.