# A Benchmark for Lease Contract Review

arXiv: https://arxiv.org/abs/2010.10386 · HTML: https://mlsanity.com/paper/2010.10386/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Oct 2020.

## Built on by (most cited first, 26 of 26)

- [CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review](https://mlsanity.com/paper/2103.06268/) (2021)
- [ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts](https://mlsanity.com/paper/2110.01799/) (2021)
- [AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing](https://mlsanity.com/paper/2108.05542/) (2021)
- [Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Dataset](https://mlsanity.com/paper/2207.00220/) (2022)
- [Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligence](https://mlsanity.com/paper/2209.02970/) (2022)
- [MAUD: An Expert-Annotated Legal NLP Dataset for Merger Agreement Understanding](https://mlsanity.com/paper/2301.00876/) (2023)
- [Unlocking Practical Applications in Legal Domain: Evaluation of GPT for Zero-Shot Semantic Annotation of Legal Texts](https://mlsanity.com/paper/2305.04417/) (2023)
- [ACORD: An Expert-Annotated Retrieval Dataset for Legal Contract Drafting](https://mlsanity.com/paper/2501.06582/) (2025)
- [The Law and NLP: Bridging Disciplinary Disconnects](https://mlsanity.com/paper/2310.14346/) (2023)
- [KidLM: Advancing Language Models for Children -- Early Insights and Future Directions](https://mlsanity.com/paper/2410.03884/) (2024)
- [Agent-Specific Deontic Modality Detection in Legal Language](https://mlsanity.com/paper/2211.12752/) (2022)
- [AIReg-Bench: Benchmarking Language Models That Assess AI Regulation Compliance](https://mlsanity.com/paper/2510.01474/) (2025)
- [ConReader: Exploring Implicit Relations in Contracts for Contract Clause Extraction](https://mlsanity.com/paper/2210.08697/) (2022)
- [Automating IRAC Analysis in Malaysian Contract Law using a Semi-Structured Knowledge Base](https://mlsanity.com/paper/2406.13217/) (2024)
- [RealKIE: Five Novel Datasets for Enterprise Key Information Extraction](https://mlsanity.com/paper/2403.20101/) (2024)
- [Real-Time Trustworthiness Scoring for LLM Structured Outputs and Data Extraction](https://mlsanity.com/paper/2603.18014/) (2026)
- [Extracting Norms from Contracts Via ChatGPT: Opportunities and Challenges](https://mlsanity.com/paper/2404.02269/) (2024)
- [Green AI in Action: Strategic Model Selection for Ensembles in Production](https://mlsanity.com/paper/2405.17451/) (2024)
- [ContractScrub: A benchmark for final review of legal contracts](https://mlsanity.com/paper/2608.20204/) (2026)
- [To Tune or Not To Tune? Zero-shot Models for Legal Case Entailment](https://mlsanity.com/paper/2202.03120/) (2022)
- [Investigating Strategies for Clause Recommendation](https://mlsanity.com/paper/2301.10716/) (2023)
- [LLMs for Legal Subsumption in German Employment Contracts](https://mlsanity.com/paper/2507.01734/) (2025)
- [LLMs for LLMs: A Structured Prompting Methodology for Long Legal Documents](https://mlsanity.com/paper/2509.02241/) (2025)
- [Streamlining Industrial Contract Management with Retrieval-Augmented LLMs](https://mlsanity.com/paper/2511.14671/) (2025)
- [A Two-Stage Architecture for NDA Analysis: LLM-based Segmentation and Transformer-based Clause Classification](https://mlsanity.com/paper/2603.09990/) (2026)
- [Accurate Legal Reasoning at Scale: Neuro-Symbolic Offloading and Structural Auditability for Robust Legal Adjudication](https://mlsanity.com/paper/2605.02472/) (2026)

## Abstract (arXiv)

Extracting entities and other useful information from legal contracts is an important task whose automation can help legal professionals perform contract reviews more efficiently and reduce relevant risks. In this paper, we tackle the problem of detecting two different types of elements that play an important role in a contract review, namely entities and red flags. The latter are terms or sentences that indicate that there is some danger or other potentially problematic situation for one or more of the signing parties. We focus on supporting the review of lease agreements, a contract type that has received little attention in the legal information extraction literature, and we define the types of entities and red flags needed for that task. We release a new benchmark dataset of 179 lease agreement documents that we have manually annotated with the entities and red flags they contain, and which can be used to train and test relevant extraction algorithms. Finally, we release a new language model, called ALeaseBERT, pre-trained on this dataset and fine-tuned for the detection of the aforementioned elements, providing a baseline for further research