# Stratified Sampling for Extreme Multi-Label Data

arXiv: https://arxiv.org/abs/2103.03494 · HTML: https://mlsanity.com/paper/2103.03494/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 16 references that are in the map.
- First posted on arXiv 5 Mar 2021.

## Builds on (most cited first, 5 of 5)

- [AttentionXML: Label Tree-based Attention-Aware Deep Model for High-Performance Extreme Multi-Label Text Classification](https://mlsanity.com/paper/1811.01727/) (2018)
- [LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation](https://mlsanity.com/paper/2005.04322/) (2020)
- [A Network Perspective on Stratification of Multi-Label Data](https://mlsanity.com/paper/1704.08756/) (2017)
- [Multi-label Few/Zero-shot Learning with Knowledge Aggregated from Multiple Label Graphs](https://mlsanity.com/paper/2010.07459/) (2020)
- [Large Scale Legal Text Classification Using Transformer Models](https://mlsanity.com/paper/2010.12871/) (2020)

## Built on by (most cited first, 7 of 7)

- [Novel split quality measures for stratified multilabel Cross Validation with application to large and sparse gene ontology datasets](https://mlsanity.com/paper/2109.01425/) (2021)
- [GUDN: A novel guide network with label reinforcement strategy for extreme multi-label text classification](https://mlsanity.com/paper/2201.11582/) (2022)
- [TEPI: Taxonomy-aware Embedding and Pseudo-Imaging for Scarcely-labeled Zero-shot Genome Classification](https://mlsanity.com/paper/2401.13219/) (2024)
- [Stratify or Die: Rethinking Data Splits in Image Segmentation](https://mlsanity.com/paper/2509.21056/) (2025)
- [Enhancing Automatic PT Tagging for MEDLINE Citations Using Transformer-Based Models](https://mlsanity.com/paper/2506.03321/) (2025)
- [Does generative AI supersede supervised XMLC? A Benchmark Study on Automated Subject Indexing with German Scientific Literature](https://mlsanity.com/paper/2607.14882/) (2026)
- [Not All Nodes Are Created Equal: Homophily-Aware Stratification for Stable GNN Evaluation](https://mlsanity.com/paper/2609.19210/) (2026)

## Abstract (arXiv)

Extreme multi-label classification (XML) is becoming increasingly relevant in the era of big data. Yet, there is no method for effectively generating stratified partitions of XML datasets. Instead, researchers typically rely on provided test-train splits that, 1) aren't always representative of the entire dataset, and 2) are missing many of the labels. This can lead to poor generalization ability and unreliable performance estimates, as has been established in the binary and multi-class settings. As such, this paper presents a new and simple algorithm that can efficiently generate stratified partitions of XML datasets with millions of unique labels. We also examine the label distributions of prevailing benchmark splits, and investigate the issues that arise from using unrepresentative subsets of data for model development. The results highlight the difficulty of stratifying XML data, and demonstrate the importance of using stratified partitions for training and evaluation.