mlsanityData to 14 Oct 2026

A Reasoning-Focused Legal Retrieval Benchmark

arXiv 2505.03970 ↗First posted 6 May 2025cs.CLcited by 37

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

As the legal community increasingly examines the use of large language models (LLMs) for various legal applications, legal AI developers have turned to retrieval-augmented LLMs ("RAG" systems) to improve system performance and robustness. An obstacle to the development of specialized RAG systems is the lack of realistic legal RAG benchmarks which capture the complexity of both legal retrieval and downstream legal question-answering. To address this, we introduce two novel legal RAG benchmarks: Bar Exam QA and Housing Statute QA. Our tasks correspond to real-world legal research tasks, and were produced through annotation processes which resemble legal research. We describe the construction of these benchmarks and the performance of existing retriever pipelines. Our results suggest that legal RAG remains a challenging application, thus motivating future research.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 21 of its 43 references that are in the map. First posted on arXiv 6 May 2025.

Builds on

21 of 43 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  4. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  6. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
  7. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  8. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  9. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 230
  10. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 201
  11. 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 176
  12. 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 150
9 more
  1. 2019JEC-QA: A Legal-Domain Question Answering Datasetcited by 116
  2. 2020A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answeringcited by 62
  3. 2021A Statutory Article Retrieval Dataset in Frenchcited by 40
  4. 2023Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Modelscited by 35
  5. 2021Yes, BM25 is a Strong Baseline for Legal Case Retrievalcited by 24
  6. 2023MILL: Mutual Verification with Large Language Models for Zero-Shot Query Expansioncited by 23
  7. 2024CLERC: A Dataset for Legal Case Retrieval and Retrieval-Augmented Analysis Generationcited by 18
  8. 2024BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectivescited by 15
  9. 2023LePaRD: A Large-Scale Dataset of Judges Citing Precedentscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 23.

37 papers.

Sort:

By area:

  1. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 86
  2. 2025L-MARS: Legal Multi-Agent System with Agentic Search and Citation-Faithfulness Auditcited by 11
  3. 2025ContractEval: Benchmarking LLMs for Clause-Level Legal Risk Identification in Commercial Contractscited by 9
  4. 2025GRADE: Generating multi-hop QA and fine-gRAined Difficulty matrix for RAG Evaluationcited by 8
  5. 2026OBLIQ-Bench: Exposing Overlooked Bottlenecks in Modern Retrievers with Latent and Implicit Queriescited by 8
  6. 2025Benchmarking Information Retrieval Models on Complex Retrieval Taskscited by 6
  7. 2025Generation-Time vs. Post-hoc Citation: A Holistic Evaluation of LLM Attributioncited by 6
  8. 2025AIReg-Bench: Benchmarking Language Models That Assess AI Regulation Compliancecited by 6
  9. 2025The Massive Legal Embedding Benchmark (MLEB)cited by 6
  10. 2025Ranking Free RAG: Replacing Re-ranking with Selection in RAG for Sensitive Domainscited by 5
  11. 2026EvidenceRL: Reinforcing Evidence Consistency for Trustworthy Language Modelscited by 5
  12. 2025A Preliminary Study of RAG for Taiwanese Historical Archivescited by 4
25 more
  1. 2025LLM-as-a-Judge is Bad, Based on AI Attempting the Exam Qualifying for the Member of the Polish National Board of Appealcited by 3
  2. 2026InfoGatherer: Principled Information Seeking via Evidence Retrieval and Strategic Questioningcited by 3
  3. 2025Deterministic Legal Agents: A Canonical Primitive API for Auditable Reasoning over Temporal Knowledge Graphscited by 2
  4. 2026Scaling Laws for Embedding Dimension in Information Retrievalcited by 2
  5. 2026CanLegalRAGBench: Evaluating Retrieval-Augmented Generation on Canadian Case Lawcited by 2
  6. 2026HARVE: Hacking-Aware Reward-Head Vector Editing for Robust Reward Modelscited by 2
  7. 2026Demystifying Numerical Instability in LLM Inference: Achieving Reproducible Inference for Mission-Critical Tasks with HEALcited by 2
  8. 2026Legal LLM Hallucination Should Be Evaluated as Failure of Legal Warrantcited by 2
  9. 2025Embedding Domain Knowledge for Large Language Models via Reinforcement Learning from Augmented Generationcited by 1
  10. 2025From Deferral to Learning: Online In-Context Knowledge Distillation for LLM Cascadescited by 1
  11. 2026Legal RAG Bench: an end-to-end benchmark for legal RAGcited by 1
  12. 2026AI-Assisted Moot Courts: Simulating Justice-Specific Questioning in Oral Argumentscited by 1
  13. 2026Beyond Case Law: Evaluating Structure-Aware Retrieval and Safety in Statute-Centric Legal QAcited by 1
  14. 2026Kernel Affine Hull Machines as Compute-Efficient Encoders for Frozen Semantic Spacescited by 1
  15. 2026Asking For An Old Friend: Diagnosing and Mitigating Temporal Failure Modes in LLM-based Statutory Question Answeringcited by 1
  16. 2026GreekBarRetrieval: A Benchmark for Greek Statutory Retrievalcited by 1
  17. 2026Building Legal Reward Models for Grounding and Abstentioncited by 1
  18. 2026Generation-Augmented Generation: A Plug-and-Play Framework for Private Knowledge Injection in Large Language Modelscited by 0
  19. 2026Benchmarking Legal RAG: The Promise and Limits of AI Statutory Surveyscited by 0
  20. 2026Justified or Just Convincing? Error Verifiability as a Dimension of LLM Qualitycited by 0
  21. 2026When Cases Get Rare: A Retrieval Benchmark for Off-Guideline Clinical Question Answeringcited by 0
  22. 2026Decompose-and-Refine: Structured Legal Question Answering with Parametric Retrievalcited by 0
  23. 2026Beyond Probabilistic Similarity: Structural, Temporal, and Causal Limitations of Retrieval-Augmented Generation in the Legal Domaincited by 0
  24. 2026Annotated Surrogate Retrieval for Polish Statutory Lawcited by 0
  25. 2026ARGUS: Role-Aware Event Knowledge Graphs for U.S. Employment-Discrimination Complaintscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain
  2. 2026Fine-grained Claim-level RAG Benchmark for Law
  3. 2025Hybrid Retrieval-Augmented Generation Agent for Trustworthy Legal Question Answering in Judicial Forensics
  4. 2025LexRAG: Benchmarking Retrieval-Augmented Generation in Multi-Turn Legal Consultation Conversation
  5. 2024Experimenting with Legal AI Solutions: The Case of Question-Answering for Access to Justice
  6. 2026AILQA: Evaluating AI-Driven Legal Question Answering Systems for the Indian Legal System
  7. 2025Elevating Legal LLM Responses: Harnessing Trainable Logical Structures and Semantic Knowledge with Legal Reasoning
  8. 2026Legal-DC: Benchmarking Retrieval-Augmented Generation for Legal Documents