# Fine-Grained Representation Learning and Recognition by Exploiting Hierarchical Semantic Embedding

arXiv: https://arxiv.org/abs/1808.04505 · HTML: https://mlsanity.com/paper/1808.04505/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Aug 2018.

## Built on by (most cited first, 28 of 28)

- [Deep Hierarchical Semantic Segmentation](https://mlsanity.com/paper/2203.14335/) (2022)
- [Mixture Outlier Exposure: Towards Out-of-Distribution Detection in Fine-grained Environments](https://mlsanity.com/paper/2106.03917/) (2021)
- [Learn to Categorize or Categorize to Learn? Self-Coding for Generalized Category Discovery](https://mlsanity.com/paper/2310.19776/) (2023)
- [Label Relation Graphs Enhanced Hierarchical Residual Network for Hierarchical Multi-Granularity Classification](https://mlsanity.com/paper/2201.03194/) (2022)
- [Fine-Grained Object Classification via Self-Supervised Pose Alignment](https://mlsanity.com/paper/2203.15987/) (2022)
- [HGCLIP: Exploring Vision-Language Models with Graph Representations for Hierarchical Understanding](https://mlsanity.com/paper/2311.14064/) (2023)
- [Meta-Learning with Less Forgetting on Large-Scale Non-Stationary Task Distributions](https://mlsanity.com/paper/2209.01501/) (2022)
- [Comprehensible Convolutional Neural Networks via Guided Concept Learning](https://mlsanity.com/paper/2101.03919/) (2021)
- [A Survey of Neural Trees](https://mlsanity.com/paper/2209.03415/) (2022)
- [Take 5: Interpretable Image Classification with a Handful of Features](https://mlsanity.com/paper/2303.13166/) (2023)
- [Convolutional Fine-Grained Classification with Self-Supervised Target Relation Regularization](https://mlsanity.com/paper/2208.01997/) (2022)
- [Digital Taxonomist: Identifying Plant Species in Community Scientists' Photographs](https://mlsanity.com/paper/2106.03774/) (2021)
- [Multi-Granularity Archaeological Dating of Chinese Bronze Dings Based on a Knowledge-Guided Relation Graph](https://mlsanity.com/paper/2303.15266/) (2023)
- [Diffusion based Semantic Outlier Generation via Nuisance Awareness for Out-of-Distribution Detection](https://mlsanity.com/paper/2408.14841/) (2024)
- [Label Hierarchy Transition: Delving into Class Hierarchies to Enhance Deep Classifiers](https://mlsanity.com/paper/2112.02353/) (2021)
- [Hierarchical Material Recognition from Local Appearance](https://mlsanity.com/paper/2505.22911/) (2025)
- [PP-SSL : Priority-Perception Self-Supervised Learning for Fine-Grained Recognition](https://mlsanity.com/paper/2412.00134/) (2024)
- [Cross-Hierarchical Bidirectional Consistency Learning for Fine-Grained Visual Classification](https://mlsanity.com/paper/2504.13608/) (2025)
- [MATANet: A Multi-context Attention and Taxonomy-Aware Network for Fine-Grained Underwater Recognition of Marine Species](https://mlsanity.com/paper/2601.03729/) (2026)
- [Graceful Degradation and Related Fields](https://mlsanity.com/paper/2106.11119/) (2021)
- [An attention-driven hierarchical multi-scale representation for visual recognition](https://mlsanity.com/paper/2110.12178/) (2021)
- [Enhancing Classification with Hierarchical Scalable Query on Fusion Transformer](https://mlsanity.com/paper/2302.14487/) (2023)
- [Collision-based Dynamics for Multi-Marginal Optimal Transport](https://mlsanity.com/paper/2412.16385/) (2024)
- [Descriptive Image-Text Matching with Graded Contextual Similarity](https://mlsanity.com/paper/2505.09997/) (2025)
- [EnGraf-Net: Multiple Granularity Branch Network with Fine-Coarse Graft Grained for Classification Task](https://mlsanity.com/paper/2509.21061/) (2025)
- [Hierarchy-Aware Fine-Tuning of Vision-Language Models](https://mlsanity.com/paper/2512.21529/) (2025)
- [DeepTaxon: An Interpretable Retrieval-Augmented Multimodal Framework for Unified Species Identification and Discovery](https://mlsanity.com/paper/2604.24029/) (2026)
- [Dual Feature Decoupling for Fine-Grained OOD Detection](https://mlsanity.com/paper/2606.05536/) (2026)

## Abstract (arXiv)

Object categories inherently form a hierarchy with different levels of concept abstraction, especially for fine-grained categories. For example, birds (Aves) can be categorized according to a four-level hierarchy of order, family, genus, and species. This hierarchy encodes rich correlations among various categories across different levels, which can effectively regularize the semantic space and thus make prediction less ambiguous. However, previous studies of fine-grained image recognition primarily focus on categories of one certain level and usually overlook this correlation information. In this work, we investigate simultaneously predicting categories of different levels in the hierarchy and integrating this structured correlation information into the deep neural network by developing a novel Hierarchical Semantic Embedding (HSE) framework. Specifically, the HSE framework sequentially predicts the category score vector of each level in the hierarchy, from highest to lowest. At each level, it incorporates the predicted score vector of the higher level as prior knowledge to learn finer-grained feature representation. During training, the predicted score vector of the higher level is also employed to regularize label prediction by using it as soft targets of corresponding sub-categories. To evaluate the proposed framework, we organize the 200 bird species of the Caltech-UCSD birds dataset with the four-level category hierarchy and construct a large-scale butterfly dataset that also covers four level categories. Extensive experiments on these two and the newly-released VegFru datasets demonstrate the superiority of our HSE framework over the baseline methods and existing competitors.