# General Multi-label Image Classification with Transformers

arXiv: https://arxiv.org/abs/2011.14027 · HTML: https://mlsanity.com/paper/2011.14027/

## Key facts

- Cited by 127 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Nov 2020.

## Built on by (most cited first, 30 of 127)

- [Query2Label: A Simple Transformer Way to Multi-Label Classification](https://mlsanity.com/paper/2107.10834/) (2021)
- [Coarse-to-Fine Sparse Transformer for Hyperspectral Image Reconstruction](https://mlsanity.com/paper/2203.04845/) (2022)
- [Instance-level Image Retrieval using Reranking Transformers](https://mlsanity.com/paper/2103.12236/) (2021)
- [UniHCP: A Unified Model for Human-Centric Perceptions](https://mlsanity.com/paper/2303.02936/) (2023)
- [Transformer-based Dual Relation Graph for Multi-label Image Recognition](https://mlsanity.com/paper/2110.04722/) (2021)
- [Large Loss Matters in Weakly Supervised Multi-Label Classification](https://mlsanity.com/paper/2206.03740/) (2022)
- [MAT-Agent: Adaptive Multi-Agent Training Optimization](https://mlsanity.com/paper/2510.17845/) (2025)
- [Multi-Task Learning with Multi-Query Transformer for Dense Prediction](https://mlsanity.com/paper/2205.14354/) (2022)
- [Open-Vocabulary Multi-Label Classification via Multi-Modal Knowledge Transfer](https://mlsanity.com/paper/2207.01887/) (2022)
- [Semi-Supervised Object Detection with Adaptive Class-Rebalancing Self-Training](https://mlsanity.com/paper/2107.05031/) (2021)
- [Multi-label Classification with Partial Annotations using Class-aware Selective Loss](https://mlsanity.com/paper/2110.10955/) (2021)
- [Integrative Few-Shot Learning for Classification and Segmentation](https://mlsanity.com/paper/2203.15712/) (2022)
- [Learning Local and Global Temporal Contexts for Video Semantic Segmentation](https://mlsanity.com/paper/2204.03330/) (2022)
- [Label2Label: A Language Modeling Framework for Multi-Attribute Learning](https://mlsanity.com/paper/2207.08677/) (2022)
- [One Positive Label is Sufficient: Single-Positive Multi-Label Learning with Label Enhancement](https://mlsanity.com/paper/2206.00517/) (2022)
- [Hulk: A Universal Knowledge Translator for Human-Centric Tasks](https://mlsanity.com/paper/2312.01697/) (2023)
- [Out-of-distribution Detection in Medical Image Analysis: A survey](https://mlsanity.com/paper/2404.18279/) (2024)
- [MlTr: Multi-label Classification with Transformer](https://mlsanity.com/paper/2106.06195/) (2021)
- [PatchCT: Aligning Patch Set and Label Set with Conditional Transport for Multi-Label Image Classification](https://mlsanity.com/paper/2307.09066/) (2023)
- [Multi-Label Retinal Disease Classification using Transformers](https://mlsanity.com/paper/2207.02335/) (2022)
- [PARFormer: Transformer-based Multi-Task Network for Pedestrian Attribute Recognition](https://mlsanity.com/paper/2304.07230/) (2023)
- [Deep Learning for Multi-Label Learning: A Comprehensive Survey](https://mlsanity.com/paper/2401.16549/) (2024)
- [Multi-label affordance mapping from egocentric vision](https://mlsanity.com/paper/2309.02120/) (2023)
- [Incomplete Multi-View Multi-Label Learning via Label-Guided Masked View- and Category-Aware Transformers](https://mlsanity.com/paper/2303.07180/) (2023)
- [Few-Shot Learning Meets Transformer: Unified Query-Support Transformers for Few-Shot Classification](https://mlsanity.com/paper/2208.12398/) (2022)
- [Class-Distribution-Aware Pseudo Labeling for Semi-Supervised Multi-Label Learning](https://mlsanity.com/paper/2305.02795/) (2023)
- [Inferring Prototypes for Multi-Label Few-Shot Image Classification with Word Vector Guided Attention](https://mlsanity.com/paper/2112.01037/) (2021)
- [Learning Semantic Segmentation from Multiple Datasets with Label Shifts](https://mlsanity.com/paper/2202.14030/) (2022)
- [Vision Transformer Adapters for Generalizable Multitask Learning](https://mlsanity.com/paper/2308.12372/) (2023)
- [Neural Collapse in Multi-label Learning with Pick-all-label Loss](https://mlsanity.com/paper/2310.15903/) (2023)

## Abstract (arXiv)

Multi-label image classification is the task of predicting a set of labels corresponding to objects, attributes or other entities present in an image. In this work we propose the Classification Transformer (C-Tran), a general framework for multi-label image classification that leverages Transformers to exploit the complex dependencies among visual features and labels. Our approach consists of a Transformer encoder trained to predict a set of target labels given an input set of masked labels, and visual features from a convolutional neural network. A key ingredient of our method is a label mask training objective that uses a ternary encoding scheme to represent the state of the labels as positive, negative, or unknown during training. Our model shows state-of-the-art performance on challenging datasets such as COCO and Visual Genome. Moreover, because our model explicitly represents the uncertainty of labels during training, it is more general by allowing us to produce improved results for images with partial or extra label annotations during inference. We demonstrate this additional capability in the COCO, Visual Genome, News500, and CUB image datasets.