# Citation Grounding Measures the Oracle: Graph Coverage Determines Reported LLM Hallucination Rates in Law

arXiv: https://arxiv.org/abs/2606.00898 · HTML: https://mlsanity.com/paper/2606.00898/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 30 May 2026.

## Builds on (most cited first, 15 of 15)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://mlsanity.com/paper/2308.11462/) (2023)
- [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021)
- [Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools](https://mlsanity.com/paper/2405.20362/) (2024)
- [Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models](https://mlsanity.com/paper/2401.01301/) (2024)
- [SaulLM-7B: A pioneering Large Language Model for Law](https://mlsanity.com/paper/2403.03883/) (2024)
- [LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain](https://mlsanity.com/paper/2301.13126/) (2023)
- [CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era](https://mlsanity.com/paper/2602.23452/) (2026)
- [Emergence of Hierarchy in Networked Endorsement Dynamics](https://mlsanity.com/paper/2007.04448/) (2020)
- [Falkor-IRAC: Graph-Constrained Generation for Verified Legal Reasoning in Indian Judicial AI](https://mlsanity.com/paper/2605.14665/) (2026)
- [Automatic Construction of a Legal Citation Graph from 100 Million Ukrainian Court Decisions: Large-Scale Extraction, Topological Analysis, and Ontology-Driven Clustering](https://mlsanity.com/paper/2605.15362/) (2026)

## Built on by (most cited first, 1 of 1)

- [Cited but Not Consulted: A Counterfactual Audit of Legal Chain-of-Thought Faithfulness](https://mlsanity.com/paper/2610.12361/) (2026)

## Abstract (arXiv)

Verifying LLM-generated legal citations against a graph of citations extracted from real court decisions is an appealing way to measure hallucination at scale: no annotators, no reference answers. We show that what such a metric reports is governed by the coverage of the graph it queries rather than by the model it evaluates, and that at the coverage where its verdicts become trustworthy it stops distinguishing models at all. We score 400 responses (100 Ukrainian legal queries x four commercial LLMs) against two snapshots of the same national citation graph, holding responses, extractor and metric fixed. Against a sparse snapshot (4.7e5 records) citation grounding ranges 0.791-0.855, apparently showing 15-21% of citations hallucinated. Against a dense snapshot of the same registry re-derived ten weeks later (3.3e8 records, 5.8e7 decisions) the identical responses score 0.989-0.999. Subsampling shows the cause is coverage: harvesting modelled as uniform record sampling, calibrated on nothing but the record count, reproduces the sparse scores to within 0.018 while knowing nothing about the models. Bootstrapping over the queries shows the other half, which no version of this work reported: no pair of systems is separable at 95% at any oracle size tested. The metric is caught between two failures. Sparse oracles discriminate, but what they discriminate is harvesting coverage; dense oracles are trustworthy and separate nothing. Densifying the graph, the obvious remedy for the first, produces the second. An independent legislation registry adjudicates: all 54 citations flagged by the sparse oracle name real articles, a 100% false-positive rate; of the four flagged by the dense oracle, two are fabrications, one a coverage gap, and one we cannot classify - it looks like a repealed provision, which is the argument in miniature.