# Feature Fusion Vision Transformer for Fine-Grained Visual Categorization

arXiv: https://arxiv.org/abs/2107.02341 · HTML: https://mlsanity.com/paper/2107.02341/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 55 references that are in the map.
- First posted on arXiv 6 Jul 2021.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [Learning to Navigate for Fine-grained Classification](https://mlsanity.com/paper/1809.00287/) (2018)
- [Fine-Grained Visual Classification via Progressive Multi-Granularity Training of Jigsaw Patches](https://mlsanity.com/paper/2003.03836/) (2020)
- [Training Vision Transformers for Image Retrieval](https://mlsanity.com/paper/2102.05644/) (2021)
- [Learning Attentive Pairwise Interaction for Fine-Grained Classification](https://mlsanity.com/paper/2002.10191/) (2020)
- [The Application of Two-level Attention Models in Deep Convolutional Neural Network for Fine-grained Image Classification](https://mlsanity.com/paper/1411.6447/) (2014)
- [Weakly Supervised Complementary Parts Models for Fine-Grained Image Classification from the Bottom Up](https://mlsanity.com/paper/1903.02827/) (2019)
- [Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization](https://mlsanity.com/paper/1712.01034/) (2017)
- [Maximum-Entropy Fine-Grained Classification](https://mlsanity.com/paper/1809.05934/) (2018)
- [Part-Stacked CNN for Fine-Grained Visual Categorization](https://mlsanity.com/paper/1512.08086/) (2015)
- [Cross-X Learning for Fine-Grained Visual Categorization](https://mlsanity.com/paper/1909.04412/) (2019)
- [Learning Deep Bilinear Transformation for Fine-grained Image Representation](https://mlsanity.com/paper/1911.03621/) (2019)
- [Attention Convolutional Binary Neural Tree for Fine-Grained Visual Categorization](https://mlsanity.com/paper/1909.11378/) (2019)
- [Channel Interaction Networks for Fine-Grained Image Categorization](https://mlsanity.com/paper/2003.05235/) (2020)
- [Improved Bilinear Pooling with CNNs](https://mlsanity.com/paper/1707.06772/) (2017)
- [Diversified Visual Attention Networks for Fine-Grained Object Classification](https://mlsanity.com/paper/1606.08572/) (2016)
- [Fine-grained Recognition: Accounting for Subtle Differences between Similar Classes](https://mlsanity.com/paper/1912.06842/) (2019)
- [Patchy Image Structure Classification Using Multi-Orientation Region Transform](https://mlsanity.com/paper/1912.00622/) (2019)
- [Learning Semantically Enhanced Feature for Fine-Grained Image Classification](https://mlsanity.com/paper/2006.13457/) (2020)
- [Mask Guided Attention For Fine-Grained Patchy Image Classification](https://mlsanity.com/paper/2102.02771/) (2021)
- [Boosted EfficientNet: Detection of Lymph Node Metastases in Breast Cancer Using Convolutional Neural Network](https://mlsanity.com/paper/2010.05027/) (2020)
- [EAR-NET: Error Attention Refining Network For Retinal Vessel Segmentation](https://mlsanity.com/paper/2107.01351/) (2021)

## Built on by (most cited first, 30 of 40)

- [Dual Cross-Attention Learning for Fine-Grained Visual Categorization and Object Re-Identification](https://mlsanity.com/paper/2205.02151/) (2022)
- [Which Tokens to Use? Investigating Token Reduction in Vision Transformers](https://mlsanity.com/paper/2308.04657/) (2023)
- [Fine-grained Visual Classification with High-temperature Refinement and Background Suppression](https://mlsanity.com/paper/2303.06442/) (2023)
- [Novel Class Discovery for Ultra-Fine-Grained Visual Categorization](https://mlsanity.com/paper/2405.06283/) (2024)
- [Agglomerative Token Clustering](https://mlsanity.com/paper/2409.11923/) (2024)
- [A Novel Plug-in Module for Fine-Grained Visual Classification](https://mlsanity.com/paper/2202.03822/) (2022)
- [Re-Attention Transformer for Weakly Supervised Object Localization](https://mlsanity.com/paper/2208.01838/) (2022)
- [On the Eigenvalues of Global Covariance Pooling for Fine-grained Visual Recognition](https://mlsanity.com/paper/2205.13282/) (2022)
- [Dividing and Conquering a BlackBox to a Mixture of Interpretable Models: Route, Interpret, Repeat](https://mlsanity.com/paper/2307.05350/) (2023)
- [Learning Contrastive Self-Distillation for Ultra-Fine-Grained Visual Categorization Targeting Limited Samples](https://mlsanity.com/paper/2311.06056/) (2023)
- [R2-Trans:Fine-Grained Visual Categorization with Redundancy Reduction](https://mlsanity.com/paper/2204.10095/) (2022)
- [A Novel Multi-Task Model Imitating Dermatologists for Accurate Differential Diagnosis of Skin Diseases in Clinical Images](https://mlsanity.com/paper/2307.08308/) (2023)
- [Convolutional Fine-Grained Classification with Self-Supervised Target Relation Regularization](https://mlsanity.com/paper/2208.01997/) (2022)
- [Foreground Guidance and Multi-Layer Feature Fusion for Unsupervised Object Discovery with Transformers](https://mlsanity.com/paper/2210.13053/) (2022)
- [Coping with Change: Learning Invariant and Minimum Sufficient Representations for Fine-Grained Visual Categorization](https://mlsanity.com/paper/2306.04893/) (2023)
- [Global-Local Similarity for Efficient Fine-Grained Image Recognition with Vision Transformers](https://mlsanity.com/paper/2407.12891/) (2024)
- [Salient Mask-Guided Vision Transformer for Fine-Grained Classification](https://mlsanity.com/paper/2305.07102/) (2023)
- [Tackling Shortcut Learning in Deep Neural Networks: An Iterative Approach with Interpretable Models](https://mlsanity.com/paper/2302.10289/) (2023)
- [Concept Drift and Long-Tailed Distribution in Fine-Grained Visual Categorization: Benchmark and Method](https://mlsanity.com/paper/2306.02346/) (2023)
- [M2Former: Multi-Scale Patch Selection for Fine-Grained Visual Recognition](https://mlsanity.com/paper/2308.02161/) (2023)
- [ViT-FOD: A Vision Transformer based Fine-grained Object Discriminator](https://mlsanity.com/paper/2203.12816/) (2022)
- [Conviformers: Convolutionally guided Vision Transformer](https://mlsanity.com/paper/2208.08900/) (2022)
- [Relational Proxies: Emergent Relationships as Fine-Grained Discriminators](https://mlsanity.com/paper/2210.02149/) (2022)
- [ViTree: Single-path Neural Tree for Step-wise Interpretable Fine-grained Visual Categorization](https://mlsanity.com/paper/2401.17050/) (2024)
- [ConceptHash: Interpretable Fine-Grained Hashing via Concept Discovery](https://mlsanity.com/paper/2406.08457/) (2024)
- [Extract More from Less: Efficient Fine-Grained Visual Recognition in Low-Data Regimes](https://mlsanity.com/paper/2406.19814/) (2024)
- [Learning from the few: Fine-grained approach to pediatric wrist pathology recognition on a limited dataset](https://mlsanity.com/paper/2408.13542/) (2024)
- [Down-Sampling Inter-Layer Adapter for Parameter and Computation Efficient Ultra-Fine-Grained Image Recognition](https://mlsanity.com/paper/2409.11051/) (2024)
- [Cross-Layer Cache Aggregation for Token Reduction in Ultra-Fine-Grained Image Recognition](https://mlsanity.com/paper/2501.00243/) (2024)
- [Vocabulary-free Fine-grained Visual Recognition via Enriched Contextually Grounded Vision-Language Model](https://mlsanity.com/paper/2507.23070/) (2025)

## Abstract (arXiv)

The core for tackling the fine-grained visual categorization (FGVC) is to learn subtle yet discriminative features. Most previous works achieve this by explicitly selecting the discriminative parts or integrating the attention mechanism via CNN-based approaches.However, these methods enhance the computational complexity and make the modeldominated by the regions containing the most of the objects. Recently, vision trans-former (ViT) has achieved SOTA performance on general image recognition tasks. Theself-attention mechanism aggregates and weights the information from all patches to the classification token, making it perfectly suitable for FGVC. Nonetheless, the classifi-cation token in the deep layer pays more attention to the global information, lacking the local and low-level features that are essential for FGVC. In this work, we proposea novel pure transformer-based framework Feature Fusion Vision Transformer (FFVT)where we aggregate the important tokens from each transformer layer to compensate thelocal, low-level and middle-level information. We design a novel token selection mod-ule called mutual attention weight selection (MAWS) to guide the network effectively and efficiently towards selecting discriminative tokens without introducing extra param-eters. We verify the effectiveness of FFVT on three benchmarks where FFVT achieves the state-of-the-art performance.