mlsanityData to 7 Oct 2026

ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts

arXiv 2110.01799 ↗First posted 5 Oct 2021cs.CL cs.AI cs.LGcited by 145

Not on the map. Why?

Abstract (arXiv)

Reviewing contracts is a time-consuming procedure that incurs large expenses to companies and social inequality to those who cannot afford it. In this work, we propose "document-level natural language inference (NLI) for contracts", a novel, real-world application of NLI that addresses such problems. In this task, a system is given a set of hypotheses (such as "Some obligations of Agreement may survive termination.") and a contract, and it is asked to classify whether each hypothesis is "entailed by", "contradicting to" or "not mentioned by" (neutral to) the contract as well as identifying "evidence" for the decision as spans in the contract. We annotated and release the largest corpus to date consisting of 607 annotated contracts. We then show that existing models fail badly on our task and introduce a strong baseline, which (1) models evidence identification as multi-label classification over spans instead of trying to predict start and end tokens, and (2) employs more sophisticated context segmentation for dealing with long documents. We also show that linguistic characteristics of contracts, such as negations by exceptions, are contributing to the difficulty of this task and that there is much room for improvement.

Key facts. Cited by 145 papers in the map (data through 7 Oct 2026). Builds on 12 of its 24 references that are in the map. First posted on arXiv 5 Oct 2021.

Builds on

12 of 24 references.

References · 12

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  3. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  4. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  5. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
  6. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 232
  7. 2020HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verificationcited by 199
  8. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 199
  9. 2018CLAUDETTE: an Automated Detector of Potentially Unfair Clauses in Online Terms of Servicecited by 72
  10. 2020A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answeringcited by 61
  11. 2020A Benchmark for Lease Contract Reviewcited by 26
  12. 2021Capturing Logical Structure of Visually Structured Documents with Multimodal Transition Parsercited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 15; 2023: 24; 2024: 23; 2025: 30; 2026: 53.

145 papers.

Sort:

By area:

  1. 2022UL2: Unifying Language Learning Paradigmscited by 292
  2. 2023FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Setscited by 198
  3. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  4. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  5. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  6. 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 118
  7. 2023AutoMix: Automatically Mixing Language Modelscited by 116
  8. 2024LegalBench-RAG: A Benchmark for Retrieval-Augmented Generation in the Legal Domaincited by 113
  9. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 96
  10. 2023CoLT5: Faster Long-Range Transformers with Conditional Computationcited by 84
  11. 2022Efficient Long-Text Understanding with Short-Text Modelscited by 71
  12. 2023LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Developmentcited by 54
38 more
  1. 2022Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusterscited by 44
  2. 2022A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Predictioncited by 44
  3. 2022An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classificationcited by 38
  4. 2022LEVEN: A Large-Scale Chinese Legal Event Detection Datasetcited by 36
  5. 2025KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Modelcited by 35
  6. 2023MAUD: An Expert-Annotated Legal NLP Dataset for Merger Agreement Understandingcited by 32
  7. 2024Learning to Generate Instruction Tuning Datasets for Zero-Shot Task Adaptationcited by 27
  8. 2024Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLPcited by 27
  9. 2024Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolationcited by 23
  10. 2023Fast and Accurate Factual Inconsistency Detection Over Long Documentscited by 22
  11. 2024FinDVer: Explainable Claim Verification over Long and Hybrid-Content Financial Documentscited by 19
  12. 2022ClassActionPrediction: A Challenging Benchmark for Legal Judgment Prediction of Class Action Cases in the UScited by 17
  13. 2025Direct Reasoning Optimization: Token-Level Reasoning Reflectivity Meets Rubric Gates for Unverifiable Taskscited by 16
  14. 2023ChatGPT's One-year Anniversary: Are Open-Source Large Language Models Catching up?cited by 15
  15. 2023Bringing order into the realm of Transformer-based language models for artificial intelligence and lawcited by 13
  16. 2023GLoRE: Evaluating Logical Reasoning of Large Language Modelscited by 13
  17. 2023More Samples or More Prompts? Exploring Effective In-Context Sampling for LLM Few-Shot Prompt Engineeringcited by 13
  18. 2023Human Still Wins over LLM: An Empirical Study of Active Learning on Domain-Specific Annotation Taskscited by 13
  19. 2024LegalLens: Leveraging LLMs for Legal Violation Identification in Unstructured Textcited by 12
  20. 2025Towards Reliable Retrieval in RAG Systems for Large Legal Datasetscited by 12
  21. 2024Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolboxcited by 11
  22. 2024CEBench: A Benchmarking Toolkit for the Cost-Effectiveness of LLM Pipelinescited by 11
  23. 2025ACORD: An Expert-Annotated Retrieval Dataset for Legal Contract Draftingcited by 11
  24. 2024LawInstruct: A Resource for Studying Language Model Adaptation to the Legal Domaincited by 10
  25. 2024Attribute or Abstain: Large Language Models as Long Document Assistantscited by 10
  26. 2025HoPE: Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Modelscited by 10
  27. 2024On Evaluating Explanation Utility for Human-AI Decision Making in NLPcited by 9
  28. 2025ContractEval: Benchmarking LLMs for Clause-Level Legal Risk Identification in Commercial Contractscited by 9
  29. 2024A Novel Cartography-Based Curriculum Learning Method Applied on RoNLI: The First Romanian Natural Language Inference Corpuscited by 8
  30. 2026Same Scores, Different Decisions: Evaluating JEV and Language Models for Legal Document Understandingcited by 8
  31. 2023Neural Natural Language Processing for Long Texts: A Survey on Classification and Summarizationcited by 7
  32. 2023LAIT: Efficient Multi-Segment Encoding in Transformers with Layer-Adjustable Interactioncited by 7
  33. 2024Lawma: The Power of Specialization for Legal Annotationcited by 7
  34. 2024LexSumm and LexT5: Benchmarking and Modeling Legal Summarization Tasks in Englishcited by 7
  35. 2025LegalWiz: A Multi-Agent Generation Framework for Contradiction Detection in Legal Documentscited by 7
  36. 2026Why Supervised Fine-Tuning Fails to Learn: A Systematic Study of Incomplete Learning in Large Language Modelscited by 7
  37. 2022LawngNLI: A Long-Premise Benchmark for In-Domain Generalization from Short to Long Contexts and for Implication-Based Retrievalcited by 6
  38. 2023Vcc: Scaling Transformers to 128K Tokens or More by Prioritizing Important Tokenscited by 6

50 most cited of 145 All 145 (JSON)