# Fast Multi-Resolution Transformer Fine-tuning for Extreme Multi-label Text Classification

arXiv: https://arxiv.org/abs/2110.00685 · HTML: https://mlsanity.com/paper/2110.00685/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 62 references that are in the map.
- First posted on arXiv 1 Oct 2021.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs](https://mlsanity.com/paper/1603.09320/) (2016)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [Deep Laplacian Pyramid Networks for Fast and Accurate Super-Resolution](https://mlsanity.com/paper/1704.03915/) (2017)
- [Pre-training Tasks for Embedding-based Large-scale Retrieval](https://mlsanity.com/paper/2002.03932/) (2020)
- [AttentionXML: Label Tree-based Attention-Aware Deep Model for High-Performance Extreme Multi-Label Text Classification](https://mlsanity.com/paper/1811.01727/) (2018)
- [Self-Supervised Generalisation with Meta Auxiliary Learning](https://mlsanity.com/paper/1901.08933/) (2019)
- [DiSMEC - Distributed Sparse Machines for Extreme Multi-label Classification](https://mlsanity.com/paper/1609.02521/) (2016)
- [Efficient softmax approximation for GPUs](https://mlsanity.com/paper/1609.04309/) (2016)
- [Bonsai -- Diverse and Shallow Trees for Extreme Multi-label Classification](https://mlsanity.com/paper/1904.08249/) (2019)
- [LightXML: Transformer with Dynamic Negative Sampling for High-Performance Extreme Multi-label Text Classification](https://mlsanity.com/paper/2101.03305/) (2021)
- [A no-regret generalization of hierarchical softmax to extreme multi-label classification](https://mlsanity.com/paper/1810.11671/) (2018)
- [PECOS: Prediction for Enormous and Correlated Output Spaces](https://mlsanity.com/paper/2010.05878/) (2020)
- [ECLARE: Extreme Classification with Label Graph Correlations](https://mlsanity.com/paper/2108.00261/) (2021)
- [Extreme Classification in Log Memory using Count-Min Sketch: A Case Study of Amazon Search with 50M Products](https://mlsanity.com/paper/1910.13830/) (2019)
- [DECAF: Deep Extreme Classification with Label Features](https://mlsanity.com/paper/2108.00368/) (2021)
- [Extreme Multi-label Learning for Semantic Matching in Product Search](https://mlsanity.com/paper/2106.12657/) (2021)
- [Pretrained Generalized Autoregressive Model with Adaptive Probabilistic Label Clusters for Extreme Multi-label Text Classification](https://mlsanity.com/paper/2007.02439/) (2020)
- [Probabilistic Label Trees for Extreme Multi-label Classification](https://mlsanity.com/paper/2009.11218/) (2020)
- [Extreme Regression for Dynamic Search Advertising](https://mlsanity.com/paper/2001.05228/) (2020)
- [Logarithmic Time Online Multiclass prediction](https://mlsanity.com/paper/1406.1822/) (2014)
- [Cost-Sensitive Label Embedding for Multi-Label Classification](https://mlsanity.com/paper/1603.09048/) (2016)
- [Logarithmic Time One-Against-Some](https://mlsanity.com/paper/1606.04988/) (2016)
- [Online probabilistic label trees](https://mlsanity.com/paper/2007.04451/) (2020)

## Built on by (most cited first, 30 of 67)

- [Large Language Models on Graphs: A Comprehensive Survey](https://mlsanity.com/paper/2312.02783/) (2023)
- [Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction](https://mlsanity.com/paper/2111.00064/) (2021)
- [14 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathon](https://mlsanity.com/paper/2306.06283/) (2023)
- [A Comprehensive Survey on Deep Clustering: Taxonomy, Challenges, and Future Directions](https://mlsanity.com/paper/2206.07579/) (2022)
- [SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learning](https://mlsanity.com/paper/2308.02565/) (2023)
- [GLBench: A Comprehensive Benchmark for Graph with Large Language Models](https://mlsanity.com/paper/2407.07457/) (2024)
- [NGAME: Negative Mining-aware Mini-batching for Extreme Classification](https://mlsanity.com/paper/2207.04452/) (2022)
- [The Effect of Metadata on Scientific Literature Tagging: A Cross-Field Cross-Model Study](https://mlsanity.com/paper/2302.03341/) (2023)
- [Extreme Zero-Shot Learning for Extreme Text Classification](https://mlsanity.com/paper/2112.08652/) (2021)
- [ELIAS: End-to-End Learning to Index and Search in Large Output Spaces](https://mlsanity.com/paper/2210.08410/) (2022)
- [Large Language Model Meets Graph Neural Network in Knowledge Distillation](https://mlsanity.com/paper/2402.05894/) (2024)
- [Hierarchical Label-wise Attention Transformer Model for Explainable ICD Coding](https://mlsanity.com/paper/2204.10716/) (2022)
- [CascadeXML: Rethinking Transformers for End-to-end Multi-resolution Training in Extreme Multi-label Classification](https://mlsanity.com/paper/2211.00640/) (2022)
- [Exploring Graph Learning Tasks with Pure LLMs: A Comprehensive Benchmark and Investigation](https://mlsanity.com/paper/2502.18771/) (2025)
- [E2EG: End-to-End Node Classification Using Graph Topology and Text-based Node Attributes](https://mlsanity.com/paper/2208.04609/) (2022)
- [A Survey on Extreme Multi-label Learning](https://mlsanity.com/paper/2210.03968/) (2022)
- [PINA: Leveraging Side Information in eXtreme Multi-label Classification via Predicted Instance Neighborhood Aggregation](https://mlsanity.com/paper/2305.12349/) (2023)
- [Dual-Encoders for Extreme Multi-Label Classification](https://mlsanity.com/paper/2310.10636/) (2023)
- [HyperBERT: Mixing Hypergraph-Aware Layers with Language Models for Node Classification on Text-Attributed Hypergraphs](https://mlsanity.com/paper/2402.07309/) (2024)
- [SemSup-XC: Semantic Supervision for Zero and Few-shot Extreme Classification](https://mlsanity.com/paper/2301.11309/) (2023)
- [Towards Heterogeneous Long-tailed Learning: Benchmarking, Metrics, and Toolbox](https://mlsanity.com/paper/2307.08235/) (2023)
- [Recent Advances in Hierarchical Multi-label Text Classification: A Survey](https://mlsanity.com/paper/2307.16265/) (2023)
- [Generalized test utilities for long-tail performance in extreme multi-label classification](https://mlsanity.com/paper/2311.05081/) (2023)
- [Automated ICD Coding using Extreme Multi-label Long Text Transformer-based Models](https://mlsanity.com/paper/2212.05857/) (2022)
- [Hypergraph Foundation Model](https://mlsanity.com/paper/2503.01203/) (2025)
- [Efficient Text Encoders for Labor Market Analysis](https://mlsanity.com/paper/2505.24640/) (2025)
- [PEFA: Parameter-Free Adapters for Large-scale Embedding-based Retrieval Models](https://mlsanity.com/paper/2312.02429/) (2023)
- [Noise Contrastive Estimation-based Matching Framework for Low-Resource Security Attack Pattern Recognition](https://mlsanity.com/paper/2401.10337/) (2024)
- [DIOR-ViT: Differential Ordinal Learning Vision Transformer for Cancer Classification in Pathology Images](https://mlsanity.com/paper/2407.08503/) (2024)
- [DynaSpec: Context-aware Dynamic Speculative Sampling for Large-Vocabulary Language Models](https://mlsanity.com/paper/2510.13847/) (2025)

## Abstract (arXiv)

Extreme multi-label text classification (XMC) seeks to find relevant labels from an extreme large label collection for a given text input. Many real-world applications can be formulated as XMC problems, such as recommendation systems, document tagging and semantic search. Recently, transformer based XMC methods, such as X-Transformer and LightXML, have shown significant improvement over other XMC methods. Despite leveraging pre-trained transformer models for text representation, the fine-tuning procedure of transformer models on large label space still has lengthy computational time even with powerful GPUs. In this paper, we propose a novel recursive approach, XR-Transformer to accelerate the procedure through recursively fine-tuning transformer models on a series of multi-resolution objectives related to the original XMC objective function. Empirical results show that XR-Transformer takes significantly less training time compared to other transformer-based XMC models while yielding better state-of-the-art results. In particular, on the public Amazon-3M dataset with 3 million labels, XR-Transformer is not only 20x faster than X-Transformer but also improves the Precision@1 from 51% to 54%.