mlsanityData to 7 Oct 2026

CLERC: A Dataset for Legal Case Retrieval and Retrieval-Augmented Analysis Generation

arXiv 2406.17186 ↗First posted 24 Jun 2024cs.CL cs.CYcited by 18

On the map: Retrieval-augmented generation → Legal and financial RAG

Abstract (arXiv)

Legal professionals need to write analyses that rely on citations to relevant precedents, i.e., previous case decisions. Intelligent systems assisting legal professionals in writing such documents provide great benefits but are challenging to design. Such systems need to help locate, summarize, and reason over salient precedents in order to be useful. To enable systems for such tasks, we work with legal professionals to transform a large open-source legal corpus into a dataset supporting two important backbone tasks: information retrieval (IR) and retrieval-augmented generation (RAG). This dataset CLERC (Case Law Evaluation Retrieval Corpus), is constructed for training and evaluating models on their ability to (1) find corresponding citations for a given piece of legal analysis and to (2) compile the text of these citations (as well as previous context) into a cogent analysis that supports a reasoning goal. We benchmark state-of-the-art models on CLERC, showing that current approaches still struggle: GPT-4o generates analyses with the highest ROUGE F-scores but hallucinates the most, while zero-shot IR models only achieve 48.3% recall@1000.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Legal and financial RAG, one of 15 directions in Retrieval-augmented generation. Builds on 43 of its 71 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

43 of 71 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  11. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  12. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
31 more
  1. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  2. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  3. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  4. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  5. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  6. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  7. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  8. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 464
  9. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 373
  10. 2023LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Modelscited by 328
  11. 2023Evaluating Verifiability in Generative Search Enginescited by 297
  12. 2023Fine-Tuning LLaMA for Multi-Stage Text Retrievalcited by 271
  13. 2021RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-rankingcited by 175
  14. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  15. 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 170
  16. 2024FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructionscited by 67
  17. 2023Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecturecited by 62
  18. 2020A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answeringcited by 61
  19. 2022Tevatron: An Efficient and Flexible Toolkit for Dense Retrievalcited by 47
  20. 2022Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularitiescited by 47
  21. 2023LeCaRDv2: A Large-Scale Chinese Legal Case Retrieval Datasetcited by 37
  22. 2023Jina Embeddings: A Novel Set of High-Performance Sentence Embedding Modelscited by 35
  23. 2023Dodo: Dynamic Contextual Compression for Decoder-only LMscited by 23
  24. 2024Leveraging Large Language Models for Learning Complex Legal Concepts through Storytellingcited by 23
  25. 2022The NLP Task Effectiveness of Long-Range Transformerscited by 22
  26. 2024Translate-Distill: Learning Cross-Language Dense Retrieval by Translation and Distillationcited by 22
  27. 2024Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERTcited by 21
  28. 2021AutoLAW: Augmented Legal Reasoning through Legal Precedent Predictioncited by 14
  29. 2023Explicitly Integrating Judgment Prediction with Legal Document Retrieval: A Law-Guided Generative Approachcited by 14
  30. 2024LLaMandement: Large Language Models for Summarization of French Legislative Proposalscited by 9
  31. 2024PILOT: Legal Case Outcome Prediction with Case Lawcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 15; 2026: 2.

18 papers.

Sort:

By area:

  1. 2025A Reasoning-Focused Legal Retrieval Benchmarkcited by 37
  2. 2024LLMs are Also Effective Embedding Models: An In-depth Overviewcited by 32
  3. 2025CPA-RAG:Covert Poisoning Attacks on Retrieval-Augmented Generation in Large Language Modelscited by 17
  4. 2025Towards Reliable Retrieval in RAG Systems for Large Legal Datasetscited by 12
  5. 2025ACORD: An Expert-Annotated Retrieval Dataset for Legal Contract Draftingcited by 11
  6. 2025Large Language Models Meet Legal Artificial Intelligence: A Surveycited by 9
  7. 2025GuRE:Generative Query REwriter for Legal Passage Retrievalcited by 6
  8. 2025CoCoLex: Confidence-guided Copy-based Decoding for Grounded Legal Text Generationcited by 6
  9. 2025LRAGE: Legal Retrieval Augmented Generation Evaluation Toolcited by 5
  10. 2025NyayaRAG: Realistic Legal Judgment Prediction with RAG under the Indian Common Law Systemcited by 5
  11. 2025REIS: A High-Performance and Energy-Efficient Retrieval System with In-Storage Processingcited by 4
  12. 2025Assessing the Performance Gap Between Lexical and Semantic Models for Information Retrieval With Formulaic Legal Languagecited by 3
6 more
  1. 2025Computational Law: Datasets, Benchmarks, and Ontologiescited by 2
  2. 2025Build AI Assistants using Large Language Models and Agents to Enhance the Engineering Education of Biomechanicscited by 1
  3. 2026DALDALL: Data Augmentation for Lexical and Semantic Diverse in Legal Domain by leveraging LLM-Personacited by 1
  4. 2025Structured Legal Document Generation in India: A Model-Agnostic Wrapper Approach with VidhikDastaavejcited by 0
  5. 2025Evaluating the Robustness of Dense Retrievers in Interdisciplinary Domainscited by 0
  6. 2026Is this Citation on Point?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Augmented Question-guided Retrieval (AQgR) of Indian Case Law with LLM, RAG, and Structured Summaries ↗
  2. 2024LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domain
  3. 2024Evaluating LLM-based Approaches to Legal Citation Prediction: Domain-specific Pre-training, Fine-tuning, or RAG? A Benchmark and an Australian Law Case Study
  4. 2025All for law and law for all: Adaptive RAG Pipeline for Legal Research
  5. 2025A Data Science Approach to Calcutta High Court Judgments: An Efficient LLM and RAG-powered Framework for Summarization and Similar Cases Retrieval
  6. 2026Fine-grained Claim-level RAG Benchmark for Law
  7. 2026CanLegalRAGBench: Evaluating Retrieval-Augmented Generation on Canadian Case Law
  8. 2026Legal-DC: Benchmarking Retrieval-Augmented Generation for Legal Documents