# Fine-grained Claim-level RAG Benchmark for Law

arXiv: https://arxiv.org/abs/2605.21071 · HTML: https://mlsanity.com/paper/2605.21071/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Legal and financial RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 33 of its 46 references that are in the map.
- First posted on arXiv 20 May 2026.

## Placed in

- Retrieval-augmented generation / [Legal and financial RAG](https://mlsanity.com/rag/legal-and-financial-rag/)

## Builds on (most cited first, 30 of 33)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Mixtral of Experts](https://mlsanity.com/paper/2401.04088/) (2024)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [Ragas: Automated Evaluation of Retrieval Augmented Generation](https://mlsanity.com/paper/2309.15217/) (2023)
- [LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models](https://mlsanity.com/paper/2308.11462/) (2023)
- [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021)
- [Improving Text Embeddings with Large Language Models](https://mlsanity.com/paper/2401.00368/) (2023)
- [Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools](https://mlsanity.com/paper/2405.20362/) (2024)
- [The Effect of Sampling Temperature on Problem Solving in Large Language Models](https://mlsanity.com/paper/2402.05201/) (2024)
- [Long-form factuality in large language models](https://mlsanity.com/paper/2403.18802/) (2024)
- [Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models](https://mlsanity.com/paper/2401.01301/) (2024)
- [LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain](https://mlsanity.com/paper/2408.10343/) (2024)
- [Calibrated Language Models Must Hallucinate](https://mlsanity.com/paper/2311.14648/) (2023)
- [CBR-RAG: Case-Based Reasoning for Retrieval Augmented Generation in LLMs for Legal Question Answering](https://mlsanity.com/paper/2404.04302/) (2024)
- [LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain](https://mlsanity.com/paper/2301.13126/) (2023)
- [RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation](https://mlsanity.com/paper/2408.08067/) (2024)
- [The Curse of Dense Low-Dimensional Information Retrieval for Large Index Sizes](https://mlsanity.com/paper/2012.14210/) (2020)
- [Exploring the Impact of Temperature on Large Language Models:Hot or Cold?](https://mlsanity.com/paper/2506.07295/) (2025)
- [Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2309.17050/) (2023)
- [FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction](https://mlsanity.com/paper/2403.02270/) (2024)
- [RefChecker: Reference-based Fine-grained Hallucination Checker and Benchmark for Large Language Models](https://mlsanity.com/paper/2405.14486/) (2024)
- [Towards Effective Extraction and Evaluation of Factual Claims](https://mlsanity.com/paper/2502.10855/) (2025)
- [OpenFactCheck: Building, Benchmarking Customized Fact-Checking Systems and Evaluating the Factuality of Claims and LLMs](https://mlsanity.com/paper/2405.05583/) (2024)
- [KoBLEX: Open Legal Question Answering with Multi-hop Reasoning](https://mlsanity.com/paper/2509.01324/) (2025)
- [LRAGE: Legal Retrieval Augmented Generation Evaluation Tool](https://mlsanity.com/paper/2504.01840/) (2025)

## Built on by (most cited first, 2 of 2)

- [Temporal Misgrounding in Legal RAG: A Versioned-Corpus Benchmark for French Tax Law](https://mlsanity.com/paper/2608.09393/) (2026)
- [How Much Do Legal RAG Systems Still Hallucinate?](https://mlsanity.com/paper/2608.14210/) (2026)

## Abstract (arXiv)

The rapid progress of large language models (LLMs) is shifting semantic search toward a question-answering paradigm, where users ask questions and LLMs generate responses. In high-stake domains such as law, retrieval-augmented generation (RAG) is commonly used to mitigate hallucinations in generated responses. Nonetheless, prior work shows that RAG systems, whether general-purpose or legal-specific, still hallucinate at varying rates, making fine-grained evaluation essential. Despite the need, existing evaluation frameworks for legal RAG systems lack the granularity required to provide detailed analysis of retrieval and generation performance separately. Moreover, current benchmarks are largely English-only and centered on legal expert queries, overlooking non-expert needs. We introduce ClaimRAG-LAW, a comprehensive dataset for legal RAG that supports French and English, targets both experts and non-experts, and includes diverse question types reflecting realistic scenarios. We further apply a fine-grained evaluation framework of state-of-the-art legal RAG systems, revealing limitations in retrieval, generation, and claim-level analysis in the legal domain.