# An Empirical Study on Large-Scale Multi-Label Text Classification Including Few and Zero-Shot Labels

arXiv: https://arxiv.org/abs/2010.01653 · HTML: https://mlsanity.com/paper/2010.01653/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Oct 2020.

## Built on by (most cited first, 30 of 44)

- [LexGLUE: A Benchmark Dataset for Legal Language Understanding in English](https://mlsanity.com/paper/2110.00976/) (2021)
- [MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer](https://mlsanity.com/paper/2109.00904/) (2021)
- [CHiLS: Zero-Shot Image Classification with Hierarchical Label Sets](https://mlsanity.com/paper/2302.02551/) (2023)
- [FiNER: Financial Numeric Entity Recognition for XBRL Tagging](https://mlsanity.com/paper/2203.06482/) (2022)
- [Revisiting Transformer-based Models for Long Document Classification](https://mlsanity.com/paper/2204.06683/) (2022)
- [PLM-ICD: Automatic ICD Coding with Pretrained Language Models](https://mlsanity.com/paper/2207.05289/) (2022)
- [Language Models for Text Classification: Is In-Context Learning Enough?](https://mlsanity.com/paper/2403.17661/) (2024)
- [Making LLMs Worth Every Penny: Resource-Limited Text Classification in Banking](https://mlsanity.com/paper/2311.06102/) (2023)
- [Automated Clinical Coding: What, Why, and Where We Are?](https://mlsanity.com/paper/2203.11092/) (2022)
- [Improved Multi-label Classification under Temporal Concept Drift: Rethinking Group-Robust Algorithms in a Label-Wise Setting](https://mlsanity.com/paper/2203.07856/) (2022)
- [An Exploration of Encoder-Decoder Approaches to Multi-Label Classification for Legal and Biomedical Text](https://mlsanity.com/paper/2305.05627/) (2023)
- [Label Semantic Aware Pre-training for Few-shot Text Classification](https://mlsanity.com/paper/2204.07128/) (2022)
- [Multi-granular Legal Topic Classification on Greek Legislation](https://mlsanity.com/paper/2109.15298/) (2021)
- [Few-shot learning for medical text: A systematic review](https://mlsanity.com/paper/2204.14081/) (2022)
- [Bringing order into the realm of Transformer-based language models for artificial intelligence and law](https://mlsanity.com/paper/2308.05502/) (2023)
- [ChronosLex: Time-aware Incremental Training for Temporal Generalization of Legal Classification Tasks](https://mlsanity.com/paper/2405.14211/) (2024)
- [ICDBigBird: A Contextual Embedding Model for ICD Code Classification](https://mlsanity.com/paper/2204.10408/) (2022)
- [Open-Vocabulary Object Detection via Language Hierarchy](https://mlsanity.com/paper/2410.20371/) (2024)
- [Improving Pretrained Models for Zero-shot Multi-label Text Classification through Reinforced Label Hierarchy Reasoning](https://mlsanity.com/paper/2104.01666/) (2021)
- [CoPHE: A Count-Preserving Hierarchical Evaluation Metric in Large-Scale Multi-Label Text Classification](https://mlsanity.com/paper/2109.04853/) (2021)
- [DKEC: Domain Knowledge Enhanced Multi-Label Classification for Diagnosis Prediction](https://mlsanity.com/paper/2310.07059/) (2023)
- [Neural Natural Language Processing for Long Texts: A Survey on Classification and Summarization](https://mlsanity.com/paper/2305.16259/) (2023)
- [Adapting to the Long Tail: A Meta-Analysis of Transfer Learning Research for Language Understanding Tasks](https://mlsanity.com/paper/2111.01340/) (2021)
- [LitMC-BERT: transformer-based multi-label classification of biomedical literature with an application on COVID-19 literature curation](https://mlsanity.com/paper/2204.08649/) (2022)
- [Medical Coding with Biomedical Transformer Ensembles and Zero/Few-shot Learning](https://mlsanity.com/paper/2206.02662/) (2022)
- [TreeMAN: Tree-enhanced Multimodal Attention Network for ICD Coding](https://mlsanity.com/paper/2305.18576/) (2023)
- [Recent Advances in Hierarchical Multi-label Text Classification: A Survey](https://mlsanity.com/paper/2307.16265/) (2023)
- [PILOT: Legal Case Outcome Prediction with Case Law](https://mlsanity.com/paper/2401.15770/) (2024)
- [APT-Pipe: A Prompt-Tuning Tool for Social Data Annotation using ChatGPT](https://mlsanity.com/paper/2402.01697/) (2024)
- [Retrieval-augmented Multi-label Text Classification](https://mlsanity.com/paper/2305.13058/) (2023)

## Abstract (arXiv)

Large-scale Multi-label Text Classification (LMTC) has a wide range of Natural Language Processing (NLP) applications and presents interesting challenges. First, not all labels are well represented in the training set, due to the very large label set and the skewed label distributions of LMTC datasets. Also, label hierarchies and differences in human labelling guidelines may affect graph-aware annotation proximity. Finally, the label hierarchies are periodically updated, requiring LMTC models capable of zero-shot generalization. Current state-of-the-art LMTC models employ Label-Wise Attention Networks (LWANs), which (1) typically treat LMTC as flat multi-label classification; (2) may use the label hierarchy to improve zero-shot learning, although this practice is vastly understudied; and (3) have not been combined with pre-trained Transformers (e.g. BERT), which have led to state-of-the-art results in several NLP benchmarks. Here, for the first time, we empirically evaluate a battery of LMTC methods from vanilla LWANs to hierarchical classification approaches and transfer learning, on frequent, few, and zero-shot learning on three datasets from different domains. We show that hierarchical methods based on Probabilistic Label Trees (PLTs) outperform LWANs. Furthermore, we show that Transformer-based approaches outperform the state-of-the-art in two of the datasets, and we propose a new state-of-the-art method which combines BERT with LWANs. Finally, we propose new models that leverage the label hierarchy to improve few and zero-shot learning, considering on each dataset a graph-aware annotation proximity measure that we introduce.