# Visual Transformers with Primal Object Queries for Multi-Label Image Classification

arXiv: https://arxiv.org/abs/2112.05485 · HTML: https://mlsanity.com/paper/2112.05485/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 33 references that are in the map.
- First posted on arXiv 10 Dec 2021.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Meshed-Memory Transformer for Image Captioning](https://mlsanity.com/paper/1912.08226/) (2019)
- [Multi-Label Image Recognition with Graph Convolutional Networks](https://mlsanity.com/paper/1904.03582/) (2019)
- [On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks](https://mlsanity.com/paper/1905.11001/) (2019)
- [CNN-RNN: A Unified Framework for Multi-label Image Classification](https://mlsanity.com/paper/1604.04573/) (2016)
- [General Multi-label Image Classification with Transformers](https://mlsanity.com/paper/2011.14027/) (2020)
- [Learning Spatial Regularization with Image-level Supervisions for Multi-label Image Classification](https://mlsanity.com/paper/1702.05891/) (2017)
- [Bag of Freebies for Training Object Detection Neural Networks](https://mlsanity.com/paper/1902.04103/) (2019)
- [Orderless Recurrent Models for Multi-label Classification](https://mlsanity.com/paper/1911.09996/) (2019)
- [Improving Pairwise Ranking for Multi-label Image Classification](https://mlsanity.com/paper/1704.03135/) (2017)
- [Order-Free RNN with Visual Attention for Multi-Label Classification](https://mlsanity.com/paper/1707.05495/) (2017)
- [Semantic Regularisation for Recurrent Image Annotation](https://mlsanity.com/paper/1611.05490/) (2016)
- [Annotation Order Matters: Recurrent Image Annotator for Arbitrary Length Image Tagging](https://mlsanity.com/paper/1604.05225/) (2016)
- [Feature Binding with Category-Dependant MixUp for Semantic Segmentation and Adversarial Robustness](https://mlsanity.com/paper/2008.05667/) (2020)
- [Love Thy Neighbors: Image Annotation by Exploiting Image Metadata](https://mlsanity.com/paper/1508.07647/) (2015)

## Abstract (arXiv)

Multi-label image classification is about predicting a set of class labels that can be considered as orderless sequential data. Transformers process the sequential data as a whole, therefore they are inherently good at set prediction. The first vision-based transformer model, which was proposed for the object detection task introduced the concept of object queries. Object queries are learnable positional encodings that are used by attention modules in decoder layers to decode the object classes or bounding boxes using the region of interests in an image. However, inputting the same set of object queries to different decoder layers hinders the training: it results in lower performance and delays convergence. In this paper, we propose the usage of primal object queries that are only provided at the start of the transformer decoder stack. In addition, we improve the mixup technique proposed for multi-label classification. The proposed transformer model with primal object queries improves the state-of-the-art class wise F1 metric by 2.1% and 1.8%; and speeds up the convergence by 79.0% and 38.6% on MS-COCO and NUS-WIDE datasets respectively.