# Large Scale Legal Text Classification Using Transformer Models

arXiv: https://arxiv.org/abs/2010.12871 · HTML: https://mlsanity.com/paper/2010.12871/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2020.

## Built on by (most cited first, 19 of 19)

- [Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challenges](https://mlsanity.com/paper/2410.21306/) (2024)
- [Parameter-Efficient Legal Domain Adaptation](https://mlsanity.com/paper/2210.13712/) (2022)
- [The Lucie-7B LLM and the Lucie Training Dataset: Open resources for multilingual language generation](https://mlsanity.com/paper/2503.12294/) (2025)
- [Stratified Sampling for Extreme Multi-Label Data](https://mlsanity.com/paper/2103.03494/) (2021)
- [Impact of Latent Space Dimension on IoT Botnet Detection Performance: VAE-Encoder Versus ViT-Encoder](https://mlsanity.com/paper/2504.14879/) (2025)
- [IoT Botnet Detection: Application of Vision Transformer to Classification of Network Flow Traffic](https://mlsanity.com/paper/2504.18781/) (2025)
- [Transformer with Selective Shuffled Position Embedding and Key-Patch Exchange Strategy for Early Detection of Knee Osteoarthritis](https://mlsanity.com/paper/2304.08364/) (2023)
- [DAGAM: Data Augmentation with Generation And Modification](https://mlsanity.com/paper/2204.02633/) (2022)
- [Automatic deductive coding in discourse analysis: an application of large language models in learning analytics](https://mlsanity.com/paper/2410.01240/) (2024)
- [SAGE: Scalable Ground Truth Evaluations for Large Sparse Autoencoders](https://mlsanity.com/paper/2410.07456/) (2024)
- [CHANCERY: Evaluating Corporate Governance Reasoning Capabilities in Language Models](https://mlsanity.com/paper/2506.04636/) (2025)
- [Lexical Squad@Multimodal Hate Speech Event Detection 2023: Multimodal Hate Speech Detection using Fused Ensemble Approach](https://mlsanity.com/paper/2309.13354/) (2023)
- [Sample-based Dynamic Hierarchical Transformer with Layer and Head Flexibility via Contextual Bandit](https://mlsanity.com/paper/2312.03038/) (2023)
- [Zero-Shot Cross-Lingual Transfer in Legal Domain Using Transformer Models](https://mlsanity.com/paper/2111.14192/) (2021)
- [Automotive Multilingual Fault Diagnosis](https://mlsanity.com/paper/2210.06918/) (2022)
- [Machine Learning Technique Based Fake News Detection](https://mlsanity.com/paper/2309.13069/) (2023)
- [Transformer-based classification of user queries for medical consultancy with respect to expert specialization](https://mlsanity.com/paper/2309.14662/) (2023)
- [Transformer-Based Language Models Across Domain Verticals: Architectures, Applications and Critical Assessment](https://mlsanity.com/paper/2606.24331/) (2026)
- [VLCQ: Post-training quantization for deep neural networks using variable length coding](https://mlsanity.com/paper/2610.15092/) (2026)

## Abstract (arXiv)

Large multi-label text classification is a challenging Natural Language Processing (NLP) problem that is concerned with text classification for datasets with thousands of labels. We tackle this problem in the legal domain, where datasets, such as JRC-Acquis and EURLEX57K labeled with the EuroVoc vocabulary were created within the legal information systems of the European Union. The EuroVoc taxonomy includes around 7000 concepts. In this work, we study the performance of various recent transformer-based models in combination with strategies such as generative pretraining, gradual unfreezing and discriminative learning rates in order to reach competitive classification performance, and present new state-of-the-art results of 0.661 (F1) for JRC-Acquis and 0.754 for EURLEX57K. Furthermore, we quantify the impact of individual steps, such as language model fine-tuning or gradual unfreezing in an ablation study, and provide reference dataset splits created with an iterative stratification algorithm.