# Enhancing Classification with Hierarchical Scalable Query on Fusion Transformer

arXiv: https://arxiv.org/abs/2302.14487 · HTML: https://mlsanity.com/paper/2302.14487/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 28 Feb 2023.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [CvT: Introducing Convolutions to Vision Transformers](https://mlsanity.com/paper/2103.15808/) (2021)
- [Graph Convolutional Networks for Text Classification](https://mlsanity.com/paper/1809.05679/) (2018)
- [Query2Label: A Simple Transformer Way to Multi-Label Classification](https://mlsanity.com/paper/2107.10834/) (2021)
- [Parametric UMAP embeddings for representation and semi-supervised learning](https://mlsanity.com/paper/2009.12981/) (2020)
- [ML-Decoder: Scalable and Versatile Classification Head](https://mlsanity.com/paper/2111.12933/) (2021)
- [Every Document Owns Its Structure: Inductive Text Classification via Graph Neural Networks](https://mlsanity.com/paper/2004.13826/) (2020)
- [Inferring Concept Hierarchies from Text Corpora via Hyperbolic Embeddings](https://mlsanity.com/paper/1902.00913/) (2019)
- [Tensor Graph Convolutional Networks for Text Classification](https://mlsanity.com/paper/2001.05313/) (2020)
- [Hierarchical Image Classification using Entailment Cone Embeddings](https://mlsanity.com/paper/2004.03459/) (2020)
- [Injecting Semantic Concepts into End-to-End Image Captioning](https://mlsanity.com/paper/2112.05230/) (2021)
- [Can CNNs Be More Robust Than Transformers?](https://mlsanity.com/paper/2206.03452/) (2022)
- [Fine-Grained Representation Learning and Recognition by Exploiting Hierarchical Semantic Embedding](https://mlsanity.com/paper/1808.04505/) (2018)
- [No Cost Likelihood Manipulation at Test Time for Making Better Mistakes in Deep Networks](https://mlsanity.com/paper/2104.00795/) (2021)
- [A Hierarchical Grocery Store Image Dataset with Visual and Semantic Labels](https://mlsanity.com/paper/1901.00711/) (2019)
- [Hierarchical Image Classification with A Literally Toy Dataset](https://mlsanity.com/paper/2111.00892/) (2021)
- [Semantic Driven Energy based Out-of-Distribution Detection](https://mlsanity.com/paper/2208.10787/) (2022)

## Abstract (arXiv)

Real-world vision based applications require fine-grained classification for various area of interest like e-commerce, mobile applications, warehouse management, etc. where reducing the severity of mistakes and improving the classification accuracy is of utmost importance. This paper proposes a method to boost fine-grained classification through a hierarchical approach via learnable independent query embeddings. This is achieved through a classification network that uses coarse class predictions to improve the fine class accuracy in a stage-wise sequential manner. We exploit the idea of hierarchy to learn query embeddings that are scalable across all levels, thus making this a relevant approach even for extreme classification where we have a large number of classes. The query is initialized with a weighted Eigen image calculated from training samples to best represent and capture the variance of the object. We introduce transformer blocks to fuse intermediate layers at which query attention happens to enhance the spatial representation of feature maps at different scales. This multi-scale fusion helps improve the accuracy of small-size objects. We propose a two-fold approach for the unique representation of learnable queries. First, at each hierarchical level, we leverage cluster based loss that ensures maximum separation between inter-class query embeddings and helps learn a better (query) representation in higher dimensional spaces. Second, we fuse coarse level queries with finer level queries weighted by a learned scale factor. We additionally introduce a novel block called Cross Attention on Multi-level queries with Prior (CAMP) Block that helps reduce error propagation from coarse level to finer level, which is a common problem in all hierarchical classifiers. Our method is able to outperform the existing methods with an improvement of ~11% at the fine-grained classification.