# ML-Decoder: Scalable and Versatile Classification Head

arXiv: https://arxiv.org/abs/2111.12933 · HTML: https://mlsanity.com/paper/2111.12933/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 52 references that are in the map.
- First posted on arXiv 25 Nov 2021.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [An Overview of Multi-Task Learning in Deep Neural Networks](https://mlsanity.com/paper/1706.05098/) (2017)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [Fine-tuning CNN Image Retrieval with No Human Annotation](https://mlsanity.com/paper/1711.02512/) (2017)
- [Asymmetric Loss For Multi-Label Classification](https://mlsanity.com/paper/2009.14119/) (2020)
- [ResNet strikes back: An improved training procedure in timm](https://mlsanity.com/paper/2110.00476/) (2021)
- [Multi-Label Image Recognition with Graph Convolutional Networks](https://mlsanity.com/paper/1904.03582/) (2019)
- [Bilinear Attention Networks](https://mlsanity.com/paper/1805.07932/) (2018)
- [Zero-Shot Learning -- The Good, the Bad and the Ugly](https://mlsanity.com/paper/1703.04394/) (2017)
- [Zero-Shot Learning by Convex Combination of Semantic Embeddings](https://mlsanity.com/paper/1312.5650/) (2013)
- [Query2Label: A Simple Transformer Way to Multi-Label Classification](https://mlsanity.com/paper/2107.10834/) (2021)
- [Learning a Deep ConvNet for Multi-label Classification with Partial Labels](https://mlsanity.com/paper/1902.09720/) (2019)
- [Distribution-Balanced Loss for Multi-Label Classification in Long-Tailed Datasets](https://mlsanity.com/paper/2007.09654/) (2020)
- [TResNet: High Performance GPU-Dedicated Architecture](https://mlsanity.com/paper/2003.13630/) (2020)
- [Learning Semantic-Specific Graph Representation for Multi-Label Image Recognition](https://mlsanity.com/paper/1908.07325/) (2019)
- [word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method](https://mlsanity.com/paper/1402.3722/) (2014)
- [Multi-label Image Recognition by Recurrently Discovering Attentional Regions](https://mlsanity.com/paper/1711.02816/) (2017)
- [Cross-Modality Attention with Semantic Graph Embedding for Multi-Label Classification](https://mlsanity.com/paper/1912.07872/) (2019)
- [Large-scale Multi-label Learning with Missing Labels](https://mlsanity.com/paper/1307.5101/) (2013)
- [Residual Attention: A Simple but Effective Method for Multi-Label Recognition](https://mlsanity.com/paper/2108.02456/) (2021)
- [Learning to Discover Multi-Class Attentional Regions for Multi-Label Image Recognition](https://mlsanity.com/paper/2007.01755/) (2020)
- [Exploit Bounding Box Annotations for Multi-label Object Recognition](https://mlsanity.com/paper/1504.05843/) (2015)
- [Multi-Label Image Classification via Knowledge Distillation from Weakly-Supervised Detection](https://mlsanity.com/paper/1809.05884/) (2018)
- [Extreme Classification in Log Memory using Count-Min Sketch: A Case Study of Amazon Search with 50M Products](https://mlsanity.com/paper/1910.13830/) (2019)
- [Discriminative Region-based Multi-Label Zero-Shot Learning](https://mlsanity.com/paper/2108.09301/) (2021)
- [Multi-label Classification with Partial Annotations using Class-aware Selective Loss](https://mlsanity.com/paper/2110.10955/) (2021)
- [Deep Extreme Multi-label Learning](https://mlsanity.com/paper/1704.03718/) (2017)

## Built on by (most cited first, 30 of 79)

- [Recognize Anything: A Strong Image Tagging Model](https://mlsanity.com/paper/2306.03514/) (2023)
- [Tag2Text: Guiding Vision-Language Model via Image Tagging](https://mlsanity.com/paper/2303.05657/) (2023)
- [Open-Set Image Tagging with Multi-Grained Text Supervision](https://mlsanity.com/paper/2310.15200/) (2023)
- [Prompt Stealing Attacks Against Text-to-Image Generation Models](https://mlsanity.com/paper/2302.09923/) (2023)
- [MAT-Agent: Adaptive Multi-Agent Training Optimization](https://mlsanity.com/paper/2510.17845/) (2025)
- [FlowTransformer: A Transformer Framework for Flow-based Network Intrusion Detection Systems](https://mlsanity.com/paper/2304.14746/) (2023)
- [Towards long-tailed, multi-label disease classification from chest X-ray: Overview of the CXR-LT challenge](https://mlsanity.com/paper/2310.16112/) (2023)
- [Deep Learning for Multi-Label Learning: A Comprehensive Survey](https://mlsanity.com/paper/2401.16549/) (2024)
- [PIDray: A Large-scale X-ray Benchmark for Real-World Prohibited Item Detection](https://mlsanity.com/paper/2211.10763/) (2022)
- [An Evolutionary Approach to Dynamic Introduction of Tasks in Large-scale Multitask Learning Systems](https://mlsanity.com/paper/2205.12755/) (2022)
- [Class-Distribution-Aware Pseudo Labeling for Semi-Supervised Multi-Label Learning](https://mlsanity.com/paper/2305.02795/) (2023)
- [LMPT: Prompt Tuning with Class-Specific Embedding Loss for Long-tailed Multi-Label Visual Recognition](https://mlsanity.com/paper/2305.04536/) (2023)
- [IDEA: Increasing Text Diversity via Online Multi-Label Recognition for Vision-Language Pre-training](https://mlsanity.com/paper/2207.05333/) (2022)
- [Neural Collapse in Multi-label Learning with Pick-all-label Loss](https://mlsanity.com/paper/2310.15903/) (2023)
- [Deep transfer learning for image classification: a survey](https://mlsanity.com/paper/2205.09904/) (2022)
- [Obj2Seq: Formatting Objects as Sequences with Class Prompt for Visual Tasks](https://mlsanity.com/paper/2209.13948/) (2022)
- [CheXFusion: Effective Fusion of Multi-View Features using Transformers for Long-Tailed Chest X-Ray Classification](https://mlsanity.com/paper/2308.03968/) (2023)
- [Multi-Label Knowledge Distillation](https://mlsanity.com/paper/2308.06453/) (2023)
- [Object Recognition as Next Token Prediction](https://mlsanity.com/paper/2312.02142/) (2023)
- [Graph Attention Transformer Network for Multi-Label Image Classification](https://mlsanity.com/paper/2203.04049/) (2022)
- [Learning to Generate Semantic Layouts for Higher Text-Image Correspondence in Text-to-Image Synthesis](https://mlsanity.com/paper/2308.08157/) (2023)
- [Toward Robustness in Multi-label Classification: A Data Augmentation Strategy against Imbalance and Noise](https://mlsanity.com/paper/2312.07087/) (2023)
- [CatLIP: CLIP-level Visual Recognition Accuracy with 2.7x Faster Pre-training on Web-scale Image-Text Data](https://mlsanity.com/paper/2404.15653/) (2024)
- [CXR-LT 2024: A MICCAI challenge on long-tailed, multi-label, and zero-shot disease classification from chest X-ray](https://mlsanity.com/paper/2506.07984/) (2025)
- [CLIP-Decoder : ZeroShot Multilabel Classification using Multimodal CLIP Aligned Representation](https://mlsanity.com/paper/2406.14830/) (2024)
- [MuMIC -- Multimodal Embedding for Multi-label Image Classification with Tempered Sigmoid](https://mlsanity.com/paper/2211.05232/) (2022)
- [Counterfactual Reasoning for Multi-Label Image Classification via Patching-Based Training](https://mlsanity.com/paper/2404.06287/) (2024)
- [Open Vocabulary Multi-Label Classification with Dual-Modal Decoder on Aligned Visual-Textual Features](https://mlsanity.com/paper/2208.09562/) (2022)
- [Astroformer: More Data Might not be all you need for Classification](https://mlsanity.com/paper/2304.05350/) (2023)
- [Learning to Rank Patches for Unbiased Image Redundancy Reduction](https://mlsanity.com/paper/2404.00680/) (2024)

## Abstract (arXiv)

In this paper, we introduce ML-Decoder, a new attention-based classification head. ML-Decoder predicts the existence of class labels via queries, and enables better utilization of spatial data compared to global average pooling. By redesigning the decoder architecture, and using a novel group-decoding scheme, ML-Decoder is highly efficient, and can scale well to thousands of classes. Compared to using a larger backbone, ML-Decoder consistently provides a better speed-accuracy trade-off. ML-Decoder is also versatile - it can be used as a drop-in replacement for various classification heads, and generalize to unseen classes when operated with word queries. Novel query augmentations further improve its generalization ability. Using ML-Decoder, we achieve state-of-the-art results on several classification tasks: on MS-COCO multi-label, we reach 91.4% mAP; on NUS-WIDE zero-shot, we reach 31.1% ZSL mAP; and on ImageNet single-label, we reach with vanilla ResNet50 backbone a new top score of 80.7%, without extra data or distillation. Public code is available at: https://github.com/Alibaba-MIIL/ML_Decoder