# Cross-Domain Generalization and Knowledge Transfer in Transformers Trained on Legal Data

arXiv: https://arxiv.org/abs/2112.07870 · HTML: https://mlsanity.com/paper/2112.07870/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 33 references that are in the map.
- First posted on arXiv 15 Dec 2021.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Neural Legal Judgment Prediction in English](https://mlsanity.com/paper/1906.02059/) (2019)
- [StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding](https://mlsanity.com/paper/1908.04577/) (2019)
- [Semi-supervised sequence tagging with bidirectional language models](https://mlsanity.com/paper/1705.00108/) (2017)
- [Identification of Rhetorical Roles of Sentences in Indian Legal Judgments](https://mlsanity.com/paper/1911.05405/) (2019)
- [Autonomy and Reliability of Continuous Active Learning for Technology-Assisted Review](https://mlsanity.com/paper/1504.06868/) (2015)
- [Legal Area Classification: A Comparative Study of Text Classifiers on Singapore Supreme Court Judgments](https://mlsanity.com/paper/1904.06470/) (2019)
- [Computer-Assisted Creation of Boolean Search Rules for Text Classification in the Legal Domain](https://mlsanity.com/paper/2112.05807/) (2021)
- [Legal Search in Case Law and Statute Law](https://mlsanity.com/paper/2108.10127/) (2021)

## Built on by (most cited first, 5 of 5)

- [Unlocking Practical Applications in Legal Domain: Evaluation of GPT for Zero-Shot Semantic Annotation of Legal Texts](https://mlsanity.com/paper/2305.04417/) (2023)
- [Data-Centric Machine Learning in the Legal Domain](https://mlsanity.com/paper/2201.06653/) (2022)
- [Toward an Intelligent Tutoring System for Argument Mining in Legal Texts](https://mlsanity.com/paper/2210.13635/) (2022)
- [Are manual annotations necessary for statutory interpretations retrieval?](https://mlsanity.com/paper/2506.13965/) (2025)
- [What Are the Facts? Automated Extraction of Court-Established Facts from Criminal-Court Opinions](https://mlsanity.com/paper/2511.05320/) (2025)

## Abstract (arXiv)

We analyze the ability of pre-trained language models to transfer knowledge among datasets annotated with different type systems and to generalize beyond the domain and dataset they were trained on. We create a meta task, over multiple datasets focused on the prediction of rhetorical roles. Prediction of the rhetorical role a sentence plays in a case decision is an important and often studied task in AI & Law. Typically, it requires the annotation of a large number of sentences to train a model, which can be time-consuming and expensive. Further, the application of the models is restrained to the same dataset it was trained on. We fine-tune language models and evaluate their performance across datasets, to investigate the models' ability to generalize across domains. Our results suggest that the approach could be helpful in overcoming the cold-start problem in active or interactvie learning, and shows the ability of the models to generalize across datasets and domains.