Is this Citation on Point?
Not on the map. Why?
Abstract (arXiv)
In 2023, a New York judge sanctioned two attorneys in Mata v. Avianca for filing a brief with hallucinated citations generated by ChatGPT. Such failures are largely caught by database lookups; the harder problem is detecting citations that point to real cases but do not support the propositions for which they are offered -- a failure mode that existing evaluations of LLMs for legal use cases largely overlook. In this paper, we study proposition-level citation support verification through controlled perturbations of real legal citations obtained from two legal corpora, either replacing the cited case or changing only the pinpoint page within the same case. We evaluate fourteen model configurations on the resulting examples. Models catch 93-100% of wrong-case corruptions. They catch only 37-61% of wrong-pinpoint corruptions on court opinions and 52-83% on legal briefs. When models fail to catch wrong-pinpoint corruptions, they accept the citation based on topical overlap rather than page-level support. Scale and extended reasoning narrow the gap but do not close it: GPT-5.4 with high reasoning effort still misses 40% of pinpoint mismatches on court opinions and 18% on briefs. Prompting the model to verify support at the cited page improves recall, but it also raises the false positive rate. Recognizing the right legal topic and verifying support for the cited proposition are distinct capabilities, and current models conflate them.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 55 references that are in the map. First posted on arXiv 12 Aug 2026.
Builds on
24 of 55 references.
References · 24
- 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
- 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
- 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
- 2023Enabling Large Language Models to Generate Text with Citationscited by 595
- 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
- 2020Fact or Fiction: Verifying Scientific Claimscited by 554
- 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 150
- 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 83
- 2024CiteME: Can Language Models Accurately Cite Scientific Claims?cited by 48
- 2023Language Models Hallucinate, but May Excel at Fact Verificationcited by 37
- 2023FactCHD: Benchmarking Fact-Conflicting Hallucination Detectioncited by 29
- 2025From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaningcited by 25
12 more
- 2025CiteEval: Principle-Driven Citation Evaluation for Source Attributioncited by 19
- 2026HalluCitation Matters: Revealing the Impact of Hallucinated References with 300 Hallucinated Papers in ACL Conferencescited by 19
- 2024CLERC: A Dataset for Legal Case Retrieval and Retrieval-Augmented Analysis Generationcited by 18
- 2026CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Eracited by 15
- 2026GhostCite: A Large-Scale Analysis of Citation Validity in the Age of Large Language Modelscited by 11
- 2025CiteGuard: Faithful Citation Attribution for LLMs via Retrieval-Augmented Validationcited by 9
- 2024Caveat Lector: Large Language Models in Legal Practicecited by 6
- 2026Detecting and Correcting Reference Hallucinations in Commercial LLMs and Deep Research Agentscited by 6
- 2025BriefMe: A Legal NLP Benchmark for Assisting with Legal Briefscited by 4
- 2025Bye-bye, Bluebook? Automating Legal Drudgery With AI-Augmented Rule Followingcited by 3
- 2025LLM-as-a-Judge: Rapid Evaluation of Legal Document Recommendation for Retrieval-Augmented Generationcited by 3
- 2026Who Checks the Citations? Benchmarking Legal Hallucination Detectioncited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026When Citations Mislead? A Claim-Level Benchmark for Legal Hallucination Detection
- 2024Evaluating LLM-based Approaches to Legal Citation Prediction: Domain-specific Pre-training, Fine-tuning, or RAG? A Benchmark and an Australian Law Case Study
- 2025Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics
- 2026LegalCiteTrust: Benchmarking Citation Trustworthiness in Chinese Long-Form Legal Research Reports
- 2026ContraVis: Evidence-Grounded Visual Analytics for Contradiction Review in Legal Contracts
- 2026Grounded Revision vs. Prior Injection: Probing Retrieval-Augmented Patent Claim Amendment
- 2025Generation-Time vs. Post-hoc Citation: A Holistic Evaluation of LLM Attribution
- 2025GreekBarBench: A Challenging Benchmark for Free-Text Legal Reasoning and Citations