# BERT Goes to Law School: Quantifying the Competitive Advantage of Access to Large Legal Corpora in Contract Understanding

arXiv: https://arxiv.org/abs/1911.00473 · HTML: https://mlsanity.com/paper/1911.00473/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Nov 2019.

## Built on by (most cited first, 16 of 16)

- [When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset](https://mlsanity.com/paper/2104.08671/) (2021)
- [Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents](https://mlsanity.com/paper/2105.03887/) (2021)
- [Factoring Statutory Reasoning as Language Understanding Challenges](https://mlsanity.com/paper/2105.07903/) (2021)
- [JuriBERT: A Masked-Language Model Adaptation for French Legal Text](https://mlsanity.com/paper/2110.01485/) (2021)
- [Privacy-Preserving Models for Legal Natural Language Processing](https://mlsanity.com/paper/2211.02956/) (2022)
- [Customizing Contextualized Language Models forLegal Document Reviews](https://mlsanity.com/paper/2102.05757/) (2021)
- [A Comparison of SVM against Pre-trained Language Models (PLMs) for Text Classification Tasks](https://mlsanity.com/paper/2211.02563/) (2022)
- [The Law of Large Documents: Understanding the Structure of Legal Contracts Using Visual Cues](https://mlsanity.com/paper/2107.08128/) (2021)
- [Better Transcription of UK Supreme Court Hearings](https://mlsanity.com/paper/2211.17094/) (2022)
- [Enhancing Legal Argument Mining with Domain Pre-training and Neural Networks](https://mlsanity.com/paper/2202.13457/) (2022)
- [DeeperDive: The Unreasonable Effectiveness of Weak Supervision in Document Understanding A Case Study in Collaboration with UiPath Inc](https://mlsanity.com/paper/2208.08000/) (2022)
- [Classification of US Supreme Court Cases using BERT-Based Techniques](https://mlsanity.com/paper/2304.08649/) (2023)
- [Annotation and Classification of Relevant Clauses in Terms-and-Conditions Contracts](https://mlsanity.com/paper/2402.14457/) (2024)
- [A Small Claims Court for the NLP: Judging Legal Text Classification Strategies With Small Datasets](https://mlsanity.com/paper/2409.05972/) (2024)
- [Contrast Is All You Need](https://mlsanity.com/paper/2307.02882/) (2023)
- [Health Insurance Coverage Rule Interpretation Corpus: Law, Policy, and Medical Guidance for Health Insurance Coverage Understanding](https://mlsanity.com/paper/2508.03718/) (2025)

## Abstract (arXiv)

Fine-tuning language models, such as BERT, on domain specific corpora has proven to be valuable in domains like scientific papers and biomedical text. In this paper, we show that fine-tuning BERT on legal documents similarly provides valuable improvements on NLP tasks in the legal domain. Demonstrating this outcome is significant for analyzing commercial agreements, because obtaining large legal corpora is challenging due to their confidential nature. As such, we show that having access to large legal corpora is a competitive advantage for commercial applications, and academic research on analyzing contracts.