mlsanityData to 7 Oct 2026

ACORD: An Expert-Annotated Retrieval Dataset for Legal Contract Drafting

arXiv 2501.06582 ↗First posted 11 Jan 2025cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Information retrieval, specifically contract clause retrieval, is foundational to contract drafting because lawyers rarely draft contracts from scratch; instead, they locate and revise the most relevant precedent. We introduce the Atticus Clause Retrieval Dataset (ACORD), the first retrieval benchmark for contract drafting fully annotated by experts. ACORD focuses on complex contract clauses such as Limitation of Liability, Indemnification, Change of Control, and Most Favored Nation. It includes 114 queries and over 126,000 query-clause pairs, each ranked on a scale from 1 to 5 stars. The task is to find the most relevant precedent clauses to a query. The bi-encoder retriever paired with pointwise LLMs re-rankers shows promising results. However, substantial improvements are still needed to effectively manage the complex legal work typically undertaken by lawyers. As the first retrieval benchmark for contract drafting annotated by experts, ACORD can serve as a valuable IR benchmark for the NLP community.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 18 of its 39 references that are in the map. First posted on arXiv 11 Jan 2025.

Builds on

18 of 39 references.

References · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  4. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  5. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  6. 2023Large Language Models are Effective Text Rankers with Pairwise Ranking Promptingcited by 302
  7. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 283
  8. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 272
  9. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 232
  10. 2024Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Toolscited by 170
  11. 2021ContractNLI: A Dataset for Document-level Natural Language Inference for Contractscited by 145
  12. 2023Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labelscited by 86
6 more
  1. 2021Lepard: Learning partial point cloud matching in rigid and deformable scenescited by 85
  2. 2020SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrievalcited by 50
  3. 2020A Benchmark for Lease Contract Reviewcited by 26
  4. 2024Better Call GPT, Comparing Large Language Models Against Lawyerscited by 25
  5. 2024CLERC: A Dataset for Legal Case Retrieval and Retrieval-Augmented Analysis Generationcited by 18
  6. 2021CLAUSEREC: A Clause Recommendation Framework for AI-aided Contract Authoringcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 3.

11 papers.

  1. 2025AIReg-Bench: Benchmarking Language Models That Assess AI Regulation Compliancecited by 5
  2. 2025Evolution of meta's llama models and parameter-efficient fine-tuning of large language models: a surveycited by 4
  3. 2025Legal Rule Induction: Towards Generalizable Principle Discovery from Analogous Judicial Precedentscited by 3
  4. 2025MASLegalBench: Benchmarking Multi-Agent Systems in Deductive Legal Reasoningcited by 3
  5. 2025The Medium Is Not the Message: Deconfounding Document Embeddings via Linear Concept Erasurecited by 1
  6. 2025Hybrid Topic-Semantic Labeling and Graph Embeddings for Unsupervised Legal Document Clusteringcited by 1
  7. 2026Contract2Plan: Verified Contract-Grounded Retrieval-Augmented Optimization for BOM-Aware Procurement and Multi-Echelon Inventory Planningcited by 1
  8. 2026Aggregation Queries over Unstructured Text: Benchmark and Agentic Methodcited by 1
  9. 2025The Material Contracts Corpuscited by 0
  10. 2025Optimizing Legal Document Retrieval in Vietnamese with Semi-Hard Negative Miningcited by 0
  11. 2026Toward Human Rights Benchmarking for LLMs: A Pilot Methodologycited by 0