mlsanityData to 7 Oct 2026

LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain

arXiv 2408.10343 ↗First posted 19 Aug 2024cs.AIcited by 113

On the map: Retrieval-augmented generation → Legal and financial RAG

Abstract (arXiv)

Retrieval-Augmented Generation (RAG) systems are showing promising potential, and are becoming increasingly relevant in AI-powered legal applications. Existing benchmarks, such as LegalBench, assess the generative capabilities of Large Language Models (LLMs) in the legal domain, but there is a critical gap in evaluating the retrieval component of RAG systems. To address this, we introduce LegalBench-RAG, the first benchmark specifically designed to evaluate the retrieval step of RAG pipelines within the legal space. LegalBench-RAG emphasizes precise retrieval by focusing on extracting minimal, highly relevant text segments from legal documents. These highly relevant snippets are preferred over retrieving document IDs, or large sequences of imprecise chunks, both of which can exceed context window limitations. Long context windows cost more to process, induce higher latency, and lead LLMs to forget or hallucinate information. Additionally, precise results allow LLMs to generate citations for the end user. The LegalBench-RAG benchmark is constructed by retracing the context used in LegalBench queries back to their original locations within the legal corpus, resulting in a dataset of 6,858 query-answer pairs over a corpus of over 79M characters, entirely human-annotated by legal experts. We also introduce LegalBench-RAG-mini, a lightweight version for rapid iteration and experimentation. By providing a dedicated benchmark for legal retrieval, LegalBench-RAG serves as a critical tool for companies and researchers focused on enhancing the accuracy and performance of RAG systems in the legal domain. The LegalBench-RAG dataset is publicly available at https://github.com/zeroentropy-cc/legalbenchrag.

Key facts. Cited by 113 papers in the map (data through 7 Oct 2026). Placed in Legal and financial RAG, one of 15 directions in Retrieval-augmented generation. Builds on 12 of its 16 references that are in the map. First posted on arXiv 19 Aug 2024.

Builds on

12 of 16 references.

References · 12

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  4. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 302
  5. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 272
  6. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 253
  7. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 232
  8. 2024Evaluation of Retrieval-Augmented Generation: A Surveycited by 146
  9. 2021ContractNLI: A Dataset for Document-level Natural Language Inference for Contractscited by 145
  10. 2019Question Answering for Privacy Policies: Combining Computational and Legal Perspectivescited by 65
  11. 2022Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularitiescited by 47
  12. 2022Legal-Tech Open Diaries: Lesson learned on how to develop and deploy light-weight models in the era of humongous Language Modelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 49; 2026: 59.

113 papers.

Sort:

By area:

  1. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generationcited by 34
  2. 2025Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Surveycited by 29
  3. 2024Towards Knowledge Checking in Retrieval-augmented Generation: A Representation Perspectivecited by 20
  4. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Surveycited by 20
  5. 2025CPA-RAG:Covert Poisoning Attacks on Retrieval-Augmented Generation in Large Language Modelscited by 17
  6. 2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 15
  7. 2026PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practicecited by 13
  8. 2025TP-RAG: Benchmarking Retrieval-Augmented Large Language Model Agents for Spatiotemporal-Aware Travel Planningcited by 12
  9. 2025Towards Reliable Retrieval in RAG Systems for Large Legal Datasetscited by 12
  10. 2025Out of Style: RAG's Fragility to Linguistic Variationcited by 10
  11. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  12. 2025Bridging Legal Knowledge and AI: Retrieval-Augmented Generation with Vector Stores, Knowledge Graphs, and Hierarchical Non-negative Matrix Factorizationcited by 8
38 more
  1. 2025Retrieval-Augmented Generation in Industry: An Interview Study on Use Cases, Requirements, Challenges, and Evaluationcited by 8
  2. 2025Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasetscited by 7
  3. 2025Challenges for Generative AI in Legal Reasoningcited by 7
  4. 2025The Lottery LLM Hypothesis, Rethinking What Abilities Should LLM Compression Preserve?cited by 6
  5. 2025The Massive Legal Embedding Benchmark (MLEB)cited by 6
  6. 2025Do LLMs Truly Understand When a Precedent Is Overruled?cited by 6
  7. 2024Beyond Guilt: Legal Judgment Prediction with Trichotomous Reasoningcited by 5
  8. 2025LRAGE: Legal Retrieval Augmented Generation Evaluation Toolcited by 5
  9. 2025LegalRAG: A Hybrid RAG System for Multilingual Legal Information Retrievalcited by 5
  10. 2025The Hidden Structure -- Improving Legal Document Understanding Through Explicit Text Formattingcited by 5
  11. 2025Ranking Free RAG: Replacing Re-ranking with Selection in RAG for Sensitive Domainscited by 5
  12. 2025On Path to Multimodal Historical Reasoning: HistBench and HistAgentcited by 5
  13. 2025NyayaRAG: Realistic Legal Judgment Prediction with RAG under the Indian Common Law Systemcited by 5
  14. 2025All for law and law for all: Adaptive RAG Pipeline for Legal Researchcited by 5
  15. 2025MizanQA: Benchmarking Large Language Models on Moroccan Legal Question Answeringcited by 5
  16. 2025AIReg-Bench: Benchmarking Language Models That Assess AI Regulation Compliancecited by 5
  17. 2026LegalGraphRAG: Multi-Agent Graph Retrieval-Augmented Generation for Reliable Legal Reasoningcited by 5
  18. 2025Hierarchical Retrieval with Evidence Curation for Open-Domain Financial Question Answering on Standardized Documentscited by 4
  19. 2025SAMVAD: A Multi-Agent System for Simulating Judicial Deliberation Dynamics in Indiacited by 4
  20. 2025LLM-guided Hierarchical Search for End-to-end Reasoning Intensive Retrievalcited by 4
  21. 2025A Preliminary Study of RAG for Taiwanese Historical Archivescited by 4
  22. 2025An Ontology-Driven Graph RAG for Legal Norms: A Structural, Temporal, and Deterministic Approachcited by 3
  23. 2025Hypercube-Based Retrieval-Augmented Generation for Scientific Question-Answeringcited by 3
  24. 2025ASP2LJ : An Adversarial Self-Play Laywer Augmented Legal Judgment Frameworkcited by 3
  25. 2025OpinioRAG: Towards Generating User-Centric Opinion Highlights from Large-scale Online Reviewscited by 3
  26. 2026Long-Document QA with Chain-of-Structured-Thought and Fine-Tuned SLMscited by 3
  27. 2024Quebec Automobile Insurance Question-Answering With Retrieval-Augmented Generationcited by 2
  28. 2025Computational Law: Datasets, Benchmarks, and Ontologiescited by 2
  29. 2025JurisTCU: A Brazilian Portuguese Information Retrieval Dataset with Query Relevance Judgmentscited by 2
  30. 2025Investigating Retrieval-Augmented Generation in Quranic Studies: A Study of 13 Open-Source Large Language Modelscited by 2
  31. 2025Adapting General-Purpose Embedding Models to Private Datasets Using Keyword-based Retrievalcited by 2
  32. 2025GAIus: Combining Genai with Legal Clauses Retrieval for Knowledge-based Assistantcited by 2
  33. 2025From Query to Explanation: Uni-RAG for Multi-Modal Retrieval-Augmented Learning in STEMcited by 2
  34. 2025Deterministic Legal Agents: A Canonical Primitive API for Auditable Reasoning over Temporal Knowledge Graphscited by 2
  35. 2026Chunking, Retrieval, and Re-ranking: An Empirical Evaluation of RAG Architectures for Policy Document Question Answeringcited by 2
  36. 2026Knowledge Is Not Static: Order-Aware Hypergraph RAG for Language Modelscited by 2
  37. 2026ComplianceNLP: Knowledge-Graph-Augmented RAG for Multi-Framework Regulatory Gap Detectioncited by 2
  38. 2026Fine-grained Claim-level RAG Benchmark for Lawcited by 2

50 most cited of 113 All 113 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Reasoning-Focused Legal Retrieval Benchmark
  2. 2026Enhancing Legal LLMs through Metadata-Enriched RAG Pipelines and Direct Preference Optimization
  3. 2025LexRAG: Benchmarking Retrieval-Augmented Generation in Multi-Turn Legal Consultation Conversation
  4. 2026Legal-DC: Benchmarking Retrieval-Augmented Generation for Legal Documents
  5. 2025ASVRI-Legal: Fine-Tuning LLMs with Retrieval Augmented Generation for Enhanced Legal Regulation
  6. 2026NepKANUN: A RAG-Based Nepali Legal Assistant
  7. 2024CBR-RAG: Case-Based Reasoning for Retrieval Augmented Generation in LLMs for Legal Question Answering
  8. 2026Chunking German Legal Code