When Citations Mislead? A Claim-Level Benchmark for Legal Hallucination Detection
Not on the map. Why?
Abstract (arXiv)
Large language models are increasingly used in legal research and drafting, but they can still produce claims that sound convincing without being supported by the cited source. We introduce PARCEL, a benchmark for checking whether a legal claim is supported by the underlying authority. Using recent New York State Court of Appeals decisions, we build a dataset of 3,396 parenthetical-style claims labeled as Supported, Refuted, or Not Found. We cast this task as a three-way natural language inference problem and evaluate several state-of-the-art LLMs in a zero-shot setting. Although the strongest models reach up to 0.97 accuracy, the results also show an important weakness: models still incorrectly mark unsupported claims as supported, even when the full opinion text is provided. Across models, missing support is harder to detect than direct contradiction, and fabricated but plausible citations cause the largest drop in performance. Overall, PARCEL provides a practical benchmark for testing claim-level groundedness in legal RAG systems.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 6 of its 22 references that are in the map. First posted on arXiv 7 Oct 2026.
Builds on
6 of 22 references.
References · 6
- 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
- 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 170
- 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 111
- 2025Towards Effective Extraction and Evaluation of Factual Claimscited by 18
- 2025Bye-bye, Bluebook? Automating Legal Drudgery With AI-Augmented Rule Followingcited by 3
- 2025Enhancing Health Fact-Checking with LLM-Generated Synthetic Datacited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Who Checks the Citations? Benchmarking Legal Hallucination Detection
- 2026LegalHalluLens: Typed Hallucination Auditing and Calibrated Multi-Agent Debate for Trustworthy Legal AI
- 2025HalluGraph: Auditable Hallucination Detection for Legal RAG Systems via Knowledge Graph Alignment
- 2025Fine-tuning Large Language Models for Improving Factuality in Legal Question Answering
- 2025SynClaimEval: A Framework for Evaluating the Utility of Synthetic Data in Long-Context Claim Verification
- 2026Is this Citation on Point?
- 2026Source or It Didn't Happen: A Multi-Agent Framework for Citation Hallucination Detection
- 2025HalluGuard: Evidence-Grounded Small Reasoning Models to Mitigate Hallucinations in Retrieval-Augmented Generation