# LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain

arXiv: https://arxiv.org/abs/2408.10343 · HTML: https://mlsanity.com/paper/2408.10343/

## Key facts

- Cited by 118 papers in the map (data through 14 Oct 2026).
- Placed in Legal and financial RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 12 of its 16 references that are in the map.
- First posted on arXiv 19 Aug 2024.

## Placed in

- Retrieval-augmented generation / [Legal and financial RAG](https://mlsanity.com/rag/legal-and-financial-rag/)

## Builds on (most cited first, 12 of 12)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021)
- [RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models](https://mlsanity.com/paper/2401.00396/) (2023)
- [MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries](https://mlsanity.com/paper/2401.15391/) (2024)
- [CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review](https://mlsanity.com/paper/2103.06268/) (2021)
- [ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts](https://mlsanity.com/paper/2110.01799/) (2021)
- [Evaluation of Retrieval-Augmented Generation: A Survey](https://mlsanity.com/paper/2405.07437/) (2024)
- [Question Answering for Privacy Policies: Combining Computational and Legal Perspectives](https://mlsanity.com/paper/1911.00841/) (2019)
- [Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularities](https://mlsanity.com/paper/2206.10883/) (2022)
- [Legal-Tech Open Diaries: Lesson learned on how to develop and deploy light-weight models in the era of humongous Language Models](https://mlsanity.com/paper/2210.13086/) (2022)

## Built on by (most cited first, 30 of 118)

- [A Survey on Knowledge-Oriented Retrieval-Augmented Generation](https://mlsanity.com/paper/2503.10677/) (2025)
- [Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2502.06872/) (2025)
- [Towards Knowledge Checking in Retrieval-augmented Generation: A Representation Perspective](https://mlsanity.com/paper/2411.14572/) (2024)
- [Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.14891/) (2025)
- [CPA-RAG:Covert Poisoning Attacks on Retrieval-Augmented Generation in Large Language Models](https://mlsanity.com/paper/2505.19864/) (2025)
- [Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression](https://mlsanity.com/paper/2505.19433/) (2025)
- [PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice](https://mlsanity.com/paper/2601.16669/) (2026)
- [TP-RAG: Benchmarking Retrieval-Augmented Large Language Model Agents for Spatiotemporal-Aware Travel Planning](https://mlsanity.com/paper/2504.08694/) (2025)
- [Towards Reliable Retrieval in RAG Systems for Large Legal Datasets](https://mlsanity.com/paper/2510.06999/) (2025)
- [Out of Style: RAG's Fragility to Linguistic Variation](https://mlsanity.com/paper/2504.08231/) (2025)
- [Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaning](https://mlsanity.com/paper/2506.08354/) (2025)
- [Bridging Legal Knowledge and AI: Retrieval-Augmented Generation with Vector Stores, Knowledge Graphs, and Hierarchical Non-negative Matrix Factorization](https://mlsanity.com/paper/2502.20364/) (2025)
- [Retrieval-Augmented Generation in Industry: An Interview Study on Use Cases, Requirements, Challenges, and Evaluation](https://mlsanity.com/paper/2508.14066/) (2025)
- [Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasets](https://mlsanity.com/paper/2504.20119/) (2025)
- [Challenges for Generative AI in Legal Reasoning](https://mlsanity.com/paper/2508.18880/) (2025)
- [Beyond Guilt: Legal Judgment Prediction with Trichotomous Reasoning](https://mlsanity.com/paper/2412.14588/) (2024)
- [The Lottery LLM Hypothesis, Rethinking What Abilities Should LLM Compression Preserve?](https://mlsanity.com/paper/2502.17535/) (2025)
- [NyayaRAG: Realistic Legal Judgment Prediction with RAG under the Indian Common Law System](https://mlsanity.com/paper/2508.00709/) (2025)
- [AIReg-Bench: Benchmarking Language Models That Assess AI Regulation Compliance](https://mlsanity.com/paper/2510.01474/) (2025)
- [The Massive Legal Embedding Benchmark (MLEB)](https://mlsanity.com/paper/2510.19365/) (2025)
- [Do LLMs Truly Understand When a Precedent Is Overruled?](https://mlsanity.com/paper/2510.20941/) (2025)
- [LRAGE: Legal Retrieval Augmented Generation Evaluation Tool](https://mlsanity.com/paper/2504.01840/) (2025)
- [LegalRAG: A Hybrid RAG System for Multilingual Legal Information Retrieval](https://mlsanity.com/paper/2504.16121/) (2025)
- [The Hidden Structure -- Improving Legal Document Understanding Through Explicit Text Formatting](https://mlsanity.com/paper/2505.12837/) (2025)
- [Ranking Free RAG: Replacing Re-ranking with Selection in RAG for Sensitive Domains](https://mlsanity.com/paper/2505.16014/) (2025)
- [On Path to Multimodal Historical Reasoning: HistBench and HistAgent](https://mlsanity.com/paper/2505.20246/) (2025)
- [Hierarchical Retrieval with Evidence Curation for Open-Domain Financial Question Answering on Standardized Documents](https://mlsanity.com/paper/2505.20368/) (2025)
- [All for law and law for all: Adaptive RAG Pipeline for Legal Research](https://mlsanity.com/paper/2508.13107/) (2025)
- [MizanQA: Benchmarking Large Language Models on Moroccan Legal Question Answering](https://mlsanity.com/paper/2508.16357/) (2025)
- [LegalGraphRAG: Multi-Agent Graph Retrieval-Augmented Generation for Reliable Legal Reasoning](https://mlsanity.com/paper/2605.28120/) (2026)

## Abstract (arXiv)

Retrieval-Augmented Generation (RAG) systems are showing promising potential, and are becoming increasingly relevant in AI-powered legal applications. Existing benchmarks, such as LegalBench, assess the generative capabilities of Large Language Models (LLMs) in the legal domain, but there is a critical gap in evaluating the retrieval component of RAG systems. To address this, we introduce LegalBench-RAG, the first benchmark specifically designed to evaluate the retrieval step of RAG pipelines within the legal space. LegalBench-RAG emphasizes precise retrieval by focusing on extracting minimal, highly relevant text segments from legal documents. These highly relevant snippets are preferred over retrieving document IDs, or large sequences of imprecise chunks, both of which can exceed context window limitations. Long context windows cost more to process, induce higher latency, and lead LLMs to forget or hallucinate information. Additionally, precise results allow LLMs to generate citations for the end user. The LegalBench-RAG benchmark is constructed by retracing the context used in LegalBench queries back to their original locations within the legal corpus, resulting in a dataset of 6,858 query-answer pairs over a corpus of over 79M characters, entirely human-annotated by legal experts. We also introduce LegalBench-RAG-mini, a lightweight version for rapid iteration and experimentation. By providing a dedicated benchmark for legal retrieval, LegalBench-RAG serves as a critical tool for companies and researchers focused on enhancing the accuracy and performance of RAG systems in the legal domain. The LegalBench-RAG dataset is publicly available at https://github.com/zeroentropy-cc/legalbenchrag.