Citation Grounding Measures the Oracle: Graph Coverage Determines Reported LLM Hallucination Rates in Law
Not on the map. Why?
Abstract (arXiv)
Verifying LLM-generated legal citations against a graph of citations extracted from real court decisions is an appealing way to measure hallucination at scale: no annotators, no reference answers. We show that what such a metric reports is governed by the coverage of the graph it queries rather than by the model it evaluates, and that at the coverage where its verdicts become trustworthy it stops distinguishing models at all. We score 400 responses (100 Ukrainian legal queries x four commercial LLMs) against two snapshots of the same national citation graph, holding responses, extractor and metric fixed. Against a sparse snapshot (4.7e5 records) citation grounding ranges 0.791-0.855, apparently showing 15-21% of citations hallucinated. Against a dense snapshot of the same registry re-derived ten weeks later (3.3e8 records, 5.8e7 decisions) the identical responses score 0.989-0.999. Subsampling shows the cause is coverage: harvesting modelled as uniform record sampling, calibrated on nothing but the record count, reproduces the sparse scores to within 0.018 while knowing nothing about the models. Bootstrapping over the queries shows the other half, which no version of this work reported: no pair of systems is separable at 95% at any oracle size tested. The metric is caught between two failures. Sparse oracles discriminate, but what they discriminate is harvesting coverage; dense oracles are trustworthy and separate nothing. Densifying the graph, the obvious remedy for the first, produces the second. An independent legislation registry adjudicates: all 54 citations flagged by the sparse oracle name real articles, a 100% false-positive rate; of the four flagged by the dense oracle, two are fabrications, one a coverage gap, and one we cannot classify - it looks like a repealed provision, which is the argument in miniature.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 15 of its 26 references that are in the map. First posted on arXiv 30 May 2026.
Builds on
15 of 26 references.
References · 15
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
- 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
- 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 328
- 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 302
- 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 170
- 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 144
- 2024SaulLM-7B: A pioneering Large Language Model for Lawcited by 136
- 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
- 2026CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Eracited by 14
3 more
- 2020Emergence of Hierarchy in Networked Endorsement Dynamicscited by 2
- 2026Falkor-IRAC: Graph-Constrained Generation for Verified Legal Reasoning in Indian Judicial AIcited by 2
- 2026Automatic Construction of a Legal Citation Graph from 100 Million Ukrainian Court Decisions: Large-Scale Extraction, Topological Analysis, and Ontology-Driven Clusteringcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.