# Parameter-Efficient Legal Domain Adaptation

arXiv: https://arxiv.org/abs/2210.13712 · HTML: https://mlsanity.com/paper/2210.13712/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 25 Oct 2022.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [True Few-Shot Learning with Language Models](https://mlsanity.com/paper/2105.11447/) (2021)
- [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021)
- [The Pushshift Reddit Dataset](https://mlsanity.com/paper/2001.08435/) (2020)
- [SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer](https://mlsanity.com/paper/2110.07904/) (2021)
- [PPT: Pre-trained Prompt Tuning for Few-shot Learning](https://mlsanity.com/paper/2109.04332/) (2021)
- [Neural Legal Judgment Prediction in English](https://mlsanity.com/paper/1906.02059/) (2019)
- [When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset](https://mlsanity.com/paper/2104.08671/) (2021)
- [Scruples: A Corpus of Community Ethical Judgments on 32,000 Real-Life Anecdotes](https://mlsanity.com/paper/2008.09094/) (2020)
- [Natural Language Understanding with the Quora Question Pairs Dataset](https://mlsanity.com/paper/1907.01041/) (2019)
- [COMETA: A Corpus for Medical Entity Linking in the Social Media](https://mlsanity.com/paper/2010.03295/) (2020)
- [Large Scale Legal Text Classification Using Transformer Models](https://mlsanity.com/paper/2010.12871/) (2020)
- [Unsupervised Domain Adaptation with Adapter](https://mlsanity.com/paper/2111.00667/) (2021)
- [Legal Area Classification: A Comparative Study of Text Classifiers on Singapore Supreme Court Judgments](https://mlsanity.com/paper/1904.06470/) (2019)
- [Dank or Not? -- Analyzing and Predicting the Popularity of Memes on Reddit](https://mlsanity.com/paper/2011.14326/) (2020)
- [Interpretable Low-Resource Legal Decision Making](https://mlsanity.com/paper/2201.01164/) (2022)

## Built on by (most cited first, 12 of 12)

- [Constructing Domain-Specific Evaluation Sets for LLM-as-a-judge](https://mlsanity.com/paper/2408.08808/) (2024)
- [DecipherPref: Analyzing Influential Factors in Human Preference Judgments via GPT-4](https://mlsanity.com/paper/2305.14702/) (2023)
- [Walking a Tightrope -- Evaluating Large Language Models in High-Risk Domains](https://mlsanity.com/paper/2311.14966/) (2023)
- [LongSafety: Enhance Safety for Long-Context LLMs](https://mlsanity.com/paper/2411.06899/) (2024)
- [Domaino1s: Guiding LLM Reasoning for Explainable Answers in High-Stakes Domains](https://mlsanity.com/paper/2501.14431/) (2025)
- [LLM-as-a-Judge for Privacy Evaluation? Exploring the Alignment of Human and LLM Perceptions of Privacy in Textual Data](https://mlsanity.com/paper/2508.12158/) (2025)
- [Experimenting with Legal AI Solutions: The Case of Question-Answering for Access to Justice](https://mlsanity.com/paper/2409.07713/) (2024)
- [SetCSE: Set Operations using Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2404.17606/) (2024)
- [SteuerLLM: Local specialized large language model for German tax law analysis](https://mlsanity.com/paper/2602.11081/) (2026)
- [Reconstruction of Personally Identifiable Information from Proprietary Data in Supervised Fine-Tuned Models](https://mlsanity.com/paper/2605.12264/) (2026)
- [DRIFTLENS: Measuring Memory-Induced Reasoning Drift in Personalized Language Models](https://mlsanity.com/paper/2607.02374/) (2026)
- [FinRT: Distilling Adaptive Red-Teaming Strategies into Reusable Adversarial Generators in Consumer Finance](https://mlsanity.com/paper/2609.36474/) (2026)

## Abstract (arXiv)

Seeking legal advice is often expensive. Recent advancements in machine learning for solving complex problems can be leveraged to help make legal services more accessible to the public. However, real-life applications encounter significant challenges. State-of-the-art language models are growing increasingly large, making parameter-efficient learning increasingly important. Unfortunately, parameter-efficient methods perform poorly with small amounts of data, which are common in the legal domain (where data labelling costs are high). To address these challenges, we propose parameter-efficient legal domain adaptation, which uses vast unsupervised legal data from public legal forums to perform legal pre-training. This method exceeds or matches the fewshot performance of existing models such as LEGAL-BERT on various legal tasks while tuning only approximately 0.1% of model parameters. Additionally, we show that our method can achieve calibration comparable to existing methods across several tasks. To the best of our knowledge, this work is among the first to explore parameter-efficient methods of tuning language models in the legal domain.