# H3Former: Hypergraph-based Semantic-Aware Aggregation via Hyperbolic Hierarchical Contrastive Loss for Fine-Grained Visual Classification

arXiv: https://arxiv.org/abs/2511.10260 · HTML: https://mlsanity.com/paper/2511.10260/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 40 of its 84 references that are in the map.
- First posted on arXiv 13 Nov 2025.

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [A Survey on Visual Transformer](https://mlsanity.com/paper/2012.12556/) (2020)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [Bilinear CNNs for Fine-grained Visual Recognition](https://mlsanity.com/paper/1504.07889/) (2015)
- [Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning](https://mlsanity.com/paper/1806.06193/) (2018)
- [Fine-Grained Image Analysis with Deep Learning: A Survey](https://mlsanity.com/paper/2111.06119/) (2021)
- [TransFG: A Transformer Architecture for Fine-grained Recognition](https://mlsanity.com/paper/2103.07976/) (2021)
- [Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification](https://mlsanity.com/paper/2108.08728/) (2021)
- [Learning Attentive Pairwise Interaction for Fine-Grained Classification](https://mlsanity.com/paper/2002.10191/) (2020)
- [Efficient Reasoning with Hidden Thinking](https://mlsanity.com/paper/2501.19201/) (2025)
- [The Devil is in the Channels: Mutual-Channel Loss for Fine-Grained Image Classification](https://mlsanity.com/paper/2002.04264/) (2020)
- [Object-Part Attention Model for Fine-grained Image Classification](https://mlsanity.com/paper/1704.01740/) (2017)
- [Bird Species Categorization Using Pose Normalized Deep Convolutional Nets](https://mlsanity.com/paper/1406.2952/) (2014)
- [Borrowing Treasures from the Wealthy: Deep Transfer Learning through Selective Joint Fine-tuning](https://mlsanity.com/paper/1702.08690/) (2017)
- [Deep Learning on Small Datasets without Pre-Training using Cosine Loss](https://mlsanity.com/paper/1901.09054/) (2019)
- [Feature Fusion Vision Transformer for Fine-Grained Visual Categorization](https://mlsanity.com/paper/2107.02341/) (2021)
- [RAMS-Trans: Recurrent Attention Multi-scale Transformer forFine-grained Image Recognition](https://mlsanity.com/paper/2107.08192/) (2021)
- [M3Net: Multi-view Encoding, Matching, and Fusion for Few-shot Fine-grained Action Recognition](https://mlsanity.com/paper/2308.03063/) (2023)
- [Context-aware Attentional Pooling (CAP) for Fine-grained Visual Classification](https://mlsanity.com/paper/2101.06635/) (2021)
- [Hyper-YOLO: When Visual Object Detection Meets Hypergraph Computation](https://mlsanity.com/paper/2408.04804/) (2024)
- [MVP-Shot: Multi-Velocity Progressive-Alignment Framework for Few-Shot Action Recognition](https://mlsanity.com/paper/2405.02077/) (2024)
- [Fine-Grained Object Classification via Self-Supervised Pose Alignment](https://mlsanity.com/paper/2203.15987/) (2022)
- [SR-GNN: Spatial Relation-aware Graph Neural Network for Fine-Grained Image Categorization](https://mlsanity.com/paper/2209.02109/) (2022)
- [Vision-centric Token Compression in Large Language Model](https://mlsanity.com/paper/2502.00791/) (2025)
- [End-to-End Spatio-Temporal Action Localisation with Video Transformers](https://mlsanity.com/paper/2304.12160/) (2023)
- [Hyperbolic Category Discovery](https://mlsanity.com/paper/2504.06120/) (2025)
- [HLFormer: Enhancing Partially Relevant Video Retrieval with Hyperbolic Learning](https://mlsanity.com/paper/2507.17402/) (2025)
- [Learning Gabor Texture Features for Fine-Grained Recognition](https://mlsanity.com/paper/2308.05396/) (2023)

## Abstract (arXiv)

Fine-Grained Visual Classification (FGVC) remains a challenging task due to subtle inter-class differences and large intra-class variations. Existing approaches typically rely on feature-selection mechanisms or region-proposal strategies to localize discriminative regions for semantic analysis. However, these methods often fail to capture discriminative cues comprehensively while introducing substantial category-agnostic redundancy. To address these limitations, we propose H3Former, a novel token-to-region framework that leverages high-order semantic relations to aggregate local fine-grained representations with structured region-level modeling. Specifically, we propose the Semantic-Aware Aggregation Module (SAAM), which exploits multi-scale contextual cues to dynamically construct a weighted hypergraph among tokens. By applying hypergraph convolution, SAAM captures high-order semantic dependencies and progressively aggregates token features into compact region-level representations. Furthermore, we introduce the Hyperbolic Hierarchical Contrastive Loss (HHCL), which enforces hierarchical semantic constraints in a non-Euclidean embedding space. The HHCL enhances inter-class separability and intra-class consistency while preserving the intrinsic hierarchical relationships among fine-grained categories. Comprehensive experiments conducted on four standard FGVC benchmarks validate the superiority of our H3Former framework.