# Balancing Methods for Multi-label Text Classification with Long-Tailed Class Distribution

arXiv: https://arxiv.org/abs/2109.04712 · HTML: https://mlsanity.com/paper/2109.04712/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 30 references that are in the map.
- First posted on arXiv 10 Sep 2021.

## Builds on (most cited first, 10 of 10)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [Dice Loss for Data-imbalanced NLP Tasks](https://mlsanity.com/paper/1911.02855/) (2019)
- [Learning a Deep ConvNet for Multi-label Classification with Partial Labels](https://mlsanity.com/paper/1902.09720/) (2019)
- [Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets](https://mlsanity.com/paper/2007.09654/) (2020)
- [Learning to Learn and Predict: A Meta-Learning Approach for Multi-Label Classification](https://mlsanity.com/paper/1909.04176/) (2019)
- [Order-free Learning Alleviating Exposure Bias in Multi-label Classification](https://mlsanity.com/paper/1909.03434/) (2019)

## Built on by (most cited first, 30 of 35)

- [A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processing](https://mlsanity.com/paper/2210.04675/) (2022)
- [Neural Additive Models for Location Scale and Shape: A Framework for Interpretable Neural Regression Beyond the Mean](https://mlsanity.com/paper/2301.11862/) (2023)
- [Compositional Generalization for Multi-label Text Classification: A Data-Augmentation Approach](https://mlsanity.com/paper/2312.11276/) (2023)
- [Recent Advances in Hierarchical Multi-label Text Classification: A Survey](https://mlsanity.com/paper/2307.16265/) (2023)
- [Multimodal Metadata Assignment for Cultural Heritage Artifacts](https://mlsanity.com/paper/2406.00423/) (2024)
- [Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/2406.13114/) (2024)
- [SEPSIS: I Can Catch Your Lies -- A New Paradigm for Deception Detection](https://mlsanity.com/paper/2312.00292/) (2023)
- [Curiosity Meets Cooperation: A Game-Theoretic Approach to Long-Tail Multi-Label Learning](https://mlsanity.com/paper/2510.17520/) (2025)
- [Konstruktor: A Strong Baseline for Simple Knowledge Graph Question Answering](https://mlsanity.com/paper/2409.15902/) (2024)
- [Task-Informed Anti-Curriculum by Masking Improves Downstream Performance on Text](https://mlsanity.com/paper/2502.12953/) (2025)
- [Label-semantics Aware Generative Approach for Domain-Agnostic Multilabel Classification](https://mlsanity.com/paper/2506.06806/) (2025)
- [Fairness Metric Design Exploration in Multi-Domain Moral Sentiment Classification using Transformer-Based Models](https://mlsanity.com/paper/2510.11222/) (2025)
- [Are We Really Making Much Progress in Text Classification? A Comparative Review](https://mlsanity.com/paper/2204.03954/) (2022)
- [ViralBERT: A User Focused BERT-Based Approach to Virality Prediction](https://mlsanity.com/paper/2206.10298/) (2022)
- [DOST -- Domain Obedient Self-supervised Training for Multi Label Classification with Noisy Labels](https://mlsanity.com/paper/2308.05101/) (2023)
- [Text2Tree: Aligning Text Representation to the Label Tree Hierarchy for Imbalanced Medical Classification](https://mlsanity.com/paper/2311.16650/) (2023)
- [LEGION: Harnessing Pre-trained Language Models for GitHub Topic Recommendations with Distribution-Balance Loss](https://mlsanity.com/paper/2403.05873/) (2024)
- [Empowering Interdisciplinary Research with BERT-Based Models: An Approach Through SciBERT-CNN with Topic Modeling](https://mlsanity.com/paper/2404.13078/) (2024)
- [Towards the Development of Balanced Synthetic Data for Correcting Grammatical Errors in Arabic: An Approach Based on Error Tagging Model and Synthetic Data Generating Model](https://mlsanity.com/paper/2502.05312/) (2025)
- [LabelCoRank: Revolutionizing Long Tail Multi-Label Classification with Co-Occurrence Reranking](https://mlsanity.com/paper/2503.07968/) (2025)
- [MADE: A Living Benchmark for Multi-Label Text Classification with Uncertainty Quantification of Medical Device Adverse Events](https://mlsanity.com/paper/2604.15203/) (2026)
- [A Comparison of Approaches for Imbalanced Classification Problems in the Context of Retrieving Relevant Documents for an Analysis](https://mlsanity.com/paper/2205.01600/) (2022)
- [Harnessing label semantics to extract higher performance under noisy label for Company to Industry matching](https://mlsanity.com/paper/2212.01685/) (2022)
- [Large-scale investigation of weakly-supervised deep learning for the fine-grained semantic indexing of biomedical literature](https://mlsanity.com/paper/2301.09350/) (2023)
- [APAM: Adaptive Pre-training and Adaptive Meta Learning in Language Model for Noisy Labels and Long-tailed Learning](https://mlsanity.com/paper/2302.03488/) (2023)
- [Scalable Label Distribution Learning for Multi-Label Classification](https://mlsanity.com/paper/2311.16556/) (2023)
- [A Debiased Nearest Neighbors Framework for Multi-Label Text Classification](https://mlsanity.com/paper/2408.03202/) (2024)
- [Enhancing Automatic PT Tagging for MEDLINE Citations Using Transformer-Based Models](https://mlsanity.com/paper/2506.03321/) (2025)
- [Causal Invariance and Counterfactual Learning Driven Cooperative Game for Multi-Label Classification](https://mlsanity.com/paper/2512.00812/) (2025)
- [A Scalable Curiosity-Driven Game-Theoretic Framework for Long-Tail Multi-Label Learning in Data Mining](https://mlsanity.com/paper/2602.15330/) (2026)

## Abstract (arXiv)

Multi-label text classification is a challenging task because it requires capturing label dependencies. It becomes even more challenging when class distribution is long-tailed. Resampling and re-weighting are common approaches used for addressing the class imbalance problem, however, they are not effective when there is label dependency besides class imbalance because they result in oversampling of common labels. Here, we introduce the application of balancing loss functions for multi-label text classification. We perform experiments on a general domain dataset with 90 labels (Reuters-21578) and a domain-specific dataset from PubMed with 18211 labels. We find that a distribution-balanced loss function, which inherently addresses both the class imbalance and label linkage problems, outperforms commonly used loss functions. Distribution balancing methods have been successfully used in the image recognition field. Here, we show their effectiveness in natural language processing. Source code is available at https://github.com/Roche/BalancedLossNLP.