# An Exploration of Encoder-Decoder Approaches to Multi-Label Classification for Legal and Biomedical Text

arXiv: https://arxiv.org/abs/2305.05627 · HTML: https://mlsanity.com/paper/2305.05627/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 9 May 2023.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Explainable Prediction of Medical Codes from Clinical Text](https://mlsanity.com/paper/1802.05695/) (2018)
- [Large-Scale Multi-Label Text Classification on EU Legislation](https://mlsanity.com/paper/1906.02192/) (2019)
- [MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer](https://mlsanity.com/paper/2109.00904/) (2021)
- [An Empirical Study on Large-Scale Multi-Label Text Classification Including Few and Zero-Shot Labels](https://mlsanity.com/paper/2010.01653/) (2020)
- [Comprehensive Comparative Study of Multi-Label Classification Methods](https://mlsanity.com/paper/2102.07113/) (2021)
- [Improved Multi-label Classification under Temporal Concept Drift: Rethinking Group-Robust Algorithms in a Label-Wise Setting](https://mlsanity.com/paper/2203.07856/) (2022)
- [Overview of BioASQ 2021: The ninth BioASQ challenge on Large-Scale Biomedical Semantic Indexing and Question Answering](https://mlsanity.com/paper/2106.14885/) (2021)

## Built on by (most cited first, 23 of 23)

- [TURNA: A Turkish Encoder-Decoder Language Model for Enhanced Understanding and Generation](https://mlsanity.com/paper/2401.14373/) (2024)
- [Taming the Sigmoid Bottleneck: Provably Argmaxable Sparse Multi-Label Classification](https://mlsanity.com/paper/2310.10443/) (2023)
- [Rather a Nurse than a Physician -- Contrastive Explanations under Investigation](https://mlsanity.com/paper/2310.11906/) (2023)
- [LexSumm and LexT5: Benchmarking and Modeling Legal Summarization Tasks in English](https://mlsanity.com/paper/2410.09527/) (2024)
- [Exploring Contrastive Learning for Long-Tailed Multi-Label Text Classification](https://mlsanity.com/paper/2404.08720/) (2024)
- [Retrieval-augmented Multi-label Text Classification](https://mlsanity.com/paper/2305.13058/) (2023)
- [A Multi-Modal Multilingual Benchmark for Document Image Classification](https://mlsanity.com/paper/2310.16356/) (2023)
- [FAIIR: Building Toward A Conversational AI Agent Assistant for Youth Mental Health Service Provision](https://mlsanity.com/paper/2405.18553/) (2024)
- [Large Language Models for Healthcare Text Classification: A Systematic Review](https://mlsanity.com/paper/2503.01159/) (2025)
- [Entailment-Driven Privacy Policy Classification with LLMs](https://mlsanity.com/paper/2409.16621/) (2024)
- [Modelling and Classifying the Components of a Literature Review](https://mlsanity.com/paper/2508.04337/) (2025)
- [The Right Model for the Job: An Evaluation of Legal Multi-Label Classification Baselines](https://mlsanity.com/paper/2401.11852/) (2024)
- [Detecting and Characterising Mobile App Metamorphosis in Google Play Store](https://mlsanity.com/paper/2407.14565/) (2024)
- [QUAD-LLM-MLTC: Large Language Models Ensemble Learning for Healthcare Text Multi-Label Classification](https://mlsanity.com/paper/2502.14189/) (2025)
- [Divide, Cache, Conquer: Dichotomic Prompting for Efficient Multi-Label LLM-Based Classification](https://mlsanity.com/paper/2511.03830/) (2025)
- [Multi-Label Bayesian Active Learning with Inter-Label Relationships](https://mlsanity.com/paper/2411.17941/) (2024)
- [Enhancing Persona Classification in Dialogue Systems: A Graph Neural Network Approach](https://mlsanity.com/paper/2412.13283/) (2024)
- [Extreme Multi-label Completion for Semantic Document Labelling with Taxonomy-Aware Parallel Learning](https://mlsanity.com/paper/2412.13809/) (2024)
- [University of Indonesia at SemEval-2025 Task 11: Evaluating State-of-the-Art Encoders for Multi-Label Emotion Detection](https://mlsanity.com/paper/2505.16460/) (2025)
- [Skim-Aware Contrastive Learning for Efficient Document Representation](https://mlsanity.com/paper/2512.24373/) (2025)
- [Soft Contextualized Encoder For User Defined Text Classification](https://mlsanity.com/paper/2601.03450/) (2026)
- [Task-Centric Acceleration of Small-Language Models](https://mlsanity.com/paper/2602.24174/) (2026)
- [Discrete Diffusion Language Models Are Training-Free Multi-Label Classifiers](https://mlsanity.com/paper/2608.14649/) (2026)

## Abstract (arXiv)

Standard methods for multi-label text classification largely rely on encoder-only pre-trained language models, whereas encoder-decoder models have proven more effective in other classification tasks. In this study, we compare four methods for multi-label classification, two based on an encoder only, and two based on an encoder-decoder. We carry out experiments on four datasets -- two in the legal domain and two in the biomedical domain, each with two levels of label granularity -- and always depart from the same pre-trained model, T5. Our results show that encoder-decoder methods outperform encoder-only methods, with a growing advantage on more complex datasets and labeling schemes of finer granularity. Using encoder-decoder models in a non-autoregressive fashion, in particular, yields the best performance overall, so we further study this approach through ablations to better understand its strengths.