# ContractEval: Benchmarking LLMs for Clause-Level Legal Risk Identification in Commercial Contracts

arXiv: https://arxiv.org/abs/2508.03080 · HTML: https://mlsanity.com/paper/2508.03080/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 26 references that are in the map.
- First posted on arXiv 5 Aug 2025.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review](https://mlsanity.com/paper/2103.06268/) (2021)
- [CAIL2018: A Large-Scale Legal Dataset for Judgment Prediction](https://mlsanity.com/paper/1807.02478/) (2018)
- [ContractNLI: A Dataset for Document-level Natural Language Inference for Contracts](https://mlsanity.com/paper/2110.01799/) (2021)
- [SaulLM-7B: A pioneering Large Language Model for Law](https://mlsanity.com/paper/2403.03883/) (2024)
- [LawBench: Benchmarking Legal Knowledge of Large Language Models](https://mlsanity.com/paper/2309.16289/) (2023)
- [A Reasoning-Focused Legal Retrieval Benchmark](https://mlsanity.com/paper/2505.03970/) (2025)
- [Lawma: The Power of Specialization for Legal Annotation](https://mlsanity.com/paper/2407.16615/) (2024)
- [AGB-DE: A Corpus for the Automated Legal Assessment of Clauses in German Consumer Contracts](https://mlsanity.com/paper/2406.06809/) (2024)
- [CHANCERY: Evaluating Corporate Governance Reasoning Capabilities in Language Models](https://mlsanity.com/paper/2506.04636/) (2025)

## Built on by (most cited first, 9 of 9)

- [LLM Agents in Law: Taxonomy, Applications, and Challenges](https://mlsanity.com/paper/2601.06216/) (2026)
- [Scaling Policy Compliance Assessment in Language Models with Policy Reasoning Traces](https://mlsanity.com/paper/2509.23291/) (2025)
- [HARVE: Hacking-Aware Reward-Head Vector Editing for Robust Reward Models](https://mlsanity.com/paper/2606.03131/) (2026)
- [AI-Assisted Moot Courts: Simulating Justice-Specific Questioning in Oral Arguments](https://mlsanity.com/paper/2603.04718/) (2026)
- [Compiled Memory: Not More Information, but More Precise Instructions for Language Agents](https://mlsanity.com/paper/2603.15666/) (2026)
- [LegalHalluLens: Typed Hallucination Auditing and Calibrated Multi-Agent Debate for Trustworthy Legal AI](https://mlsanity.com/paper/2606.18021/) (2026)
- [ContractScrub: A benchmark for final review of legal contracts](https://mlsanity.com/paper/2608.20204/) (2026)
- [What the Window Does Not Contain: Auditing Provenance in a Document-Grounded Instability Benchmark](https://mlsanity.com/paper/2609.06147/) (2026)
- [ContraVis: Evidence-Grounded Visual Analytics for Contradiction Review in Legal Contracts](https://mlsanity.com/paper/2609.27014/) (2026)

## Abstract (arXiv)

The potential of large language models (LLMs) in specialized domains such as legal risk analysis remains underexplored. In response to growing interest in locally deploying open-source LLMs for legal tasks while preserving data confidentiality, this paper introduces ContractEval, the first benchmark to thoroughly evaluate whether open-source LLMs could match proprietary LLMs in identifying clause-level legal risks in commercial contracts. Using the Contract Understanding Atticus Dataset (CUAD), we assess 4 proprietary and 15 open-source LLMs. Our results highlight five key findings: (1) Proprietary models outperform open-source models in both correctness and output effectiveness, though some open-source models are competitive in certain specific dimensions. (2) Larger open-source models generally perform better, though the improvement slows down as models get bigger. (3) Reasoning ("thinking") mode improves output effectiveness but reduces correctness, likely due to over-complicating simpler tasks. (4) Open-source models generate "no related clause" responses more frequently even when relevant clauses are present. This suggests "laziness" in thinking or low confidence in extracting relevant content. (5) Model quantization speeds up inference but at the cost of performance drop, showing the tradeoff between efficiency and accuracy. These findings suggest that while most LLMs perform at a level comparable to junior legal assistants, open-source models require targeted fine-tuning to ensure correctness and effectiveness in high-stakes legal settings. ContractEval offers a solid benchmark to guide future development of legal-domain LLMs.