# Data-Centric Machine Learning in the Legal Domain

arXiv: https://arxiv.org/abs/2201.06653 · HTML: https://mlsanity.com/paper/2201.06653/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 20 references that are in the map.
- First posted on arXiv 17 Jan 2022.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Universal Sentence Encoder](https://mlsanity.com/paper/1803.11175/) (2018)
- [Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks](https://mlsanity.com/paper/2103.14749/) (2021)
- [Identification of Rhetorical Roles of Sentences in Indian Legal Judgments](https://mlsanity.com/paper/1911.05405/) (2019)
- [Lex Rosetta: Transfer of Predictive Models Across Languages, Jurisdictions, and Legal Domains](https://mlsanity.com/paper/2112.07882/) (2021)
- [Sentence Embeddings and High-speed Similarity Search for Fast Computer Assisted Annotation of Legal Documents](https://mlsanity.com/paper/2112.11494/) (2021)
- [Cross-Domain Generalization and Knowledge Transfer in Transformers Trained on Legal Data](https://mlsanity.com/paper/2112.07870/) (2021)

## Built on by (most cited first, 2 of 2)

- [SemEval 2023 Task 6: LegalEval - Understanding Legal Texts](https://mlsanity.com/paper/2304.09548/) (2023)
- [Toward an Intelligent Tutoring System for Argument Mining in Legal Texts](https://mlsanity.com/paper/2210.13635/) (2022)

## Abstract (arXiv)

Machine learning research typically starts with a fixed data set created early in the process. The focus of the experiments is finding a model and training procedure that result in the best possible performance in terms of some selected evaluation metric. This paper explores how changes in a data set influence the measured performance of a model. Using three publicly available data sets from the legal domain, we investigate how changes to their size, the train/test splits, and the human labelling accuracy impact the performance of a trained deep learning classifier. We assess the overall performance (weighted average) as well as the per-class performance. The observed effects are surprisingly pronounced, especially when the per-class performance is considered. We investigate how "semantic homogeneity" of a class, i.e., the proximity of sentences in a semantic embedding space, influences the difficulty of its classification. The presented results have far reaching implications for efforts related to data collection and curation in the field of AI & Law. The results also indicate that enhancements to a data set could be considered, alongside the advancement of the ML models, as an additional path for increasing classification performance on various tasks in AI & Law. Finally, we discuss the need for an established methodology to assess the potential effects of data set properties.