# SIM-Trans: Structure Information Modeling Transformer for Fine-grained Visual Categorization

arXiv: https://arxiv.org/abs/2208.14607 · HTML: https://mlsanity.com/paper/2208.14607/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 42 references that are in the map.
- First posted on arXiv 31 Aug 2022.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers](https://mlsanity.com/paper/2012.15840/) (2020)
- [TransReID: Transformer-based Object Re-Identification](https://mlsanity.com/paper/2102.04378/) (2021)
- [Fine-Grained Image Analysis with Deep Learning: A Survey](https://mlsanity.com/paper/2111.06119/) (2021)
- [Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification](https://mlsanity.com/paper/2108.08728/) (2021)
- [Part-based R-CNNs for Fine-grained Category Detection](https://mlsanity.com/paper/1407.3867/) (2014)
- [Learning to Navigate for Fine-grained Classification](https://mlsanity.com/paper/1809.00287/) (2018)
- [Looking for the Devil in the Details: Learning Trilinear Attention Sampling Network for Fine-grained Image Recognition](https://mlsanity.com/paper/1903.06150/) (2019)
- [Fine-Grained Visual Classification via Progressive Multi-Granularity Training of Jigsaw Patches](https://mlsanity.com/paper/2003.03836/) (2020)
- [Interpretable and Accurate Fine-grained Recognition via Region Grouping](https://mlsanity.com/paper/2005.10411/) (2020)
- [Weakly Supervised Complementary Parts Models for Fine-Grained Image Classification from the Bottom Up](https://mlsanity.com/paper/1903.02827/) (2019)
- [Learning to Zoom: a Saliency-Based Sampling Layer for Neural Networks](https://mlsanity.com/paper/1809.03355/) (2018)
- [Object-Part Attention Model for Fine-grained Image Classification](https://mlsanity.com/paper/1704.01740/) (2017)
- [Bird Species Categorization Using Pose Normalized Deep Convolutional Nets](https://mlsanity.com/paper/1406.2952/) (2014)
- [RAMS-Trans: Recurrent Attention Multi-scale Transformer forFine-grained Image Recognition](https://mlsanity.com/paper/2107.08192/) (2021)
- [Fine-grained Recognition: Accounting for Subtle Differences between Similar Classes](https://mlsanity.com/paper/1912.06842/) (2019)
- [Look-into-Object: Self-supervised Structure Modeling for Object Recognition](https://mlsanity.com/paper/2003.14142/) (2020)
- [Fast Fine-grained Image Classification via Weakly Supervised Discriminative Localization](https://mlsanity.com/paper/1710.01168/) (2017)
- [Learning Canonical 3D Object Representation for Fine-Grained Recognition](https://mlsanity.com/paper/2108.04628/) (2021)

## Built on by (most cited first, 19 of 19)

- [Fine-grained Visual Classification with High-temperature Refinement and Background Suppression](https://mlsanity.com/paper/2303.06442/) (2023)
- [Delving into Multimodal Prompting for Fine-grained Visual Classification](https://mlsanity.com/paper/2309.08912/) (2023)
- [Learning Contrastive Self-Distillation for Ultra-Fine-Grained Visual Categorization Targeting Limited Samples](https://mlsanity.com/paper/2311.06056/) (2023)
- [Robust Saliency-Aware Distillation for Few-shot Fine-grained Visual Recognition](https://mlsanity.com/paper/2305.07180/) (2023)
- [Coping with Change: Learning Invariant and Minimum Sufficient Representations for Fine-Grained Visual Categorization](https://mlsanity.com/paper/2306.04893/) (2023)
- [Global-Local Similarity for Efficient Fine-Grained Image Recognition with Vision Transformers](https://mlsanity.com/paper/2407.12891/) (2024)
- [Salient Mask-Guided Vision Transformer for Fine-Grained Classification](https://mlsanity.com/paper/2305.07102/) (2023)
- [Fine-Grained Domain Generalization with Feature Structuralization](https://mlsanity.com/paper/2406.09166/) (2024)
- [Concept Drift and Long-Tailed Distribution in Fine-Grained Visual Categorization: Benchmark and Method](https://mlsanity.com/paper/2306.02346/) (2023)
- [Down-Sampling Inter-Layer Adapter for Parameter and Computation Efficient Ultra-Fine-Grained Image Recognition](https://mlsanity.com/paper/2409.11051/) (2024)
- [Cross-Layer Cache Aggregation for Token Reduction in Ultra-Fine-Grained Image Recognition](https://mlsanity.com/paper/2501.00243/) (2024)
- [Semantic Feature Integration network for Fine-grained Visual Classification](https://mlsanity.com/paper/2302.10275/) (2023)
- [Exploration of Class Center for Fine-Grained Visual Classification](https://mlsanity.com/paper/2407.04243/) (2024)
- [Extremely Fine-Grained Visual Classification over Resembling Glyphs in the Wild](https://mlsanity.com/paper/2408.13774/) (2024)
- [Navigating limitations with precision: A fine-grained ensemble approach to wrist pathology recognition on a limited x-ray dataset](https://mlsanity.com/paper/2412.13884/) (2024)
- [CausalFSFG: Rethinking Few-Shot Fine-Grained Visual Categorization from Causal Perspective](https://mlsanity.com/paper/2512.21617/) (2025)
- [Expert Knowledge-Guided Decision Calibration for Accurate Fine-Grained Tree Species Classification](https://mlsanity.com/paper/2601.16498/) (2026)
- [ToolFG: Towards Well-Grounded Fine-Grained Image Classification](https://mlsanity.com/paper/2606.02518/) (2026)
- [ReCalMatch:Reliability-Calibrated Semantic Guidance for Semi-Supervised Fine-Grained Recognition](https://mlsanity.com/paper/2609.29678/) (2026)

## Abstract (arXiv)

Fine-grained visual categorization (FGVC) aims at recognizing objects from similar subordinate categories, which is challenging and practical for human's accurate automatic recognition needs. Most FGVC approaches focus on the attention mechanism research for discriminative regions mining while neglecting their interdependencies and composed holistic object structure, which are essential for model's discriminative information localization and understanding ability. To address the above limitations, we propose the Structure Information Modeling Transformer (SIM-Trans) to incorporate object structure information into transformer for enhancing discriminative representation learning to contain both the appearance information and structure information. Specifically, we encode the image into a sequence of patch tokens and build a strong vision transformer framework with two well-designed modules: (i) the structure information learning (SIL) module is proposed to mine the spatial context relation of significant patches within the object extent with the help of the transformer's self-attention weights, which is further injected into the model for importing structure information; (ii) the multi-level feature boosting (MFB) module is introduced to exploit the complementary of multi-level features and contrastive learning among classes to enhance feature robustness for accurate recognition. The proposed two modules are light-weighted and can be plugged into any transformer network and trained end-to-end easily, which only depends on the attention weights that come with the vision transformer itself. Extensive experiments and analyses demonstrate that the proposed SIM-Trans achieves state-of-the-art performance on fine-grained visual categorization benchmarks. The code is available at https://github.com/PKU-ICST-MIPL/SIM-Trans_ACMMM2022.