# Legal Transformer Models May Not Always Help

arXiv: https://arxiv.org/abs/2109.06862 · HTML: https://mlsanity.com/paper/2109.06862/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 16 references that are in the map.
- First posted on arXiv 14 Sep 2021.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset](https://mlsanity.com/paper/2104.08671/) (2021)
- [LEGAL-BERT: The Muppets straight out of Law School](https://mlsanity.com/paper/2010.02559/) (2020)
- [Large-Scale Multi-Label Text Classification on EU Legislation](https://mlsanity.com/paper/1906.02192/) (2019)
- [On the importance of pre-training data volume for compact language models](https://mlsanity.com/paper/2010.03813/) (2020)
- [To Pretrain or Not to Pretrain: Examining the Benefits of Pretraining on Resource Rich Tasks](https://mlsanity.com/paper/2006.08671/) (2020)

## Built on by (most cited first, 6 of 6)

- [A Predictive Factor Analysis of Social Biases and Task-Performance in Pretrained Masked Language Models](https://mlsanity.com/paper/2310.12936/) (2023)
- [Can Large Language Models Predict the Outcome of Judicial Decisions?](https://mlsanity.com/paper/2501.09768/) (2025)
- [Evaluating Customized vs. Generalist Transformer-based Models for Legal Contract Classification](https://mlsanity.com/paper/2508.07849/) (2025)
- [FastDoc: Domain-Specific Fast Continual Pre-training Technique using Document-Level Metadata and Taxonomy](https://mlsanity.com/paper/2306.06190/) (2023)
- [Towards Unsupervised Question Answering System with Multi-level Summarization for Legal Text](https://mlsanity.com/paper/2403.13107/) (2024)
- [LLM-Assisted Causal Structure Disambiguation and Factor Extraction for Legal Judgment Prediction](https://mlsanity.com/paper/2603.11446/) (2026)

## Abstract (arXiv)

Deep learning-based Natural Language Processing methods, especially transformers, have achieved impressive performance in the last few years. Applying those state-of-the-art NLP methods to legal activities to automate or simplify some simple work is of great value. This work investigates the value of domain adaptive pre-training and language adapters in legal NLP tasks. By comparing the performance of language models with domain adaptive pre-training on different tasks and different dataset splits, we show that domain adaptive pre-training is only helpful with low-resource downstream tasks, thus far from being a panacea. We also benchmark the performance of adapters in a typical legal NLP task and show that they can yield similar performance to full model tuning with much smaller training costs. As an additional result, we release LegalRoBERTa, a RoBERTa model further pre-trained on legal corpora.