# Label Relation Graphs Enhanced Hierarchical Residual Network for Hierarchical Multi-Granularity Classification

arXiv: https://arxiv.org/abs/2201.03194 · HTML: https://mlsanity.com/paper/2201.03194/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 42 references that are in the map.
- First posted on arXiv 10 Jan 2022.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [Coherent Hierarchical Multi-Label Classification Networks](https://mlsanity.com/paper/2010.10151/) (2020)
- [Weakly-Supervised Hierarchical Text Classification](https://mlsanity.com/paper/1812.11270/) (2018)
- [Hierarchical Text Classification with Reinforced Label Assignment](https://mlsanity.com/paper/1908.10419/) (2019)
- [Hyperbolic Interaction Model For Hierarchical Multi-Label Classification](https://mlsanity.com/paper/1905.10802/) (2019)
- [Fine-Grained Representation Learning and Recognition by Exploiting Hierarchical Semantic Embedding](https://mlsanity.com/paper/1808.04505/) (2018)
- [Embedding Label Structures for Fine-Grained Feature Representation](https://mlsanity.com/paper/1512.02895/) (2015)
- [Fine-grained Image Classification by Exploring Bipartite-Graph Labels](https://mlsanity.com/paper/1512.02665/) (2015)
- [Embedding Visual Hierarchy with Deep Networks for Large-Scale Visual Recognition](https://mlsanity.com/paper/1707.02406/) (2017)

## Built on by (most cited first, 29 of 29)

- [Deep Learning with Logical Constraints](https://mlsanity.com/paper/2205.00523/) (2022)
- [HIER: Metric Learning Beyond Class Labels via Hierarchical Regularization](https://mlsanity.com/paper/2212.14258/) (2022)
- [Visually Consistent Hierarchical Image Classification](https://mlsanity.com/paper/2406.11608/) (2024)
- [Language-guided Hierarchical Fine-grained Image Forgery Detection and Localization](https://mlsanity.com/paper/2410.23556/) (2024)
- [HMIL: Hierarchical Multi-Instance Learning for Fine-Grained Whole Slide Image Classification](https://mlsanity.com/paper/2411.07660/) (2024)
- [A Survey of Neural Trees](https://mlsanity.com/paper/2209.03415/) (2022)
- [The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognition](https://mlsanity.com/paper/2505.24840/) (2025)
- [Multi-Granularity Archaeological Dating of Chinese Bronze Dings Based on a Knowledge-Guided Relation Graph](https://mlsanity.com/paper/2303.15266/) (2023)
- [Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Models](https://mlsanity.com/paper/2603.00431/) (2026)
- [Adaptive Location Hierarchy Learning for Long-Tailed Mobility Prediction](https://mlsanity.com/paper/2505.19965/) (2025)
- [Free-Grained Hierarchical Visual Recognition](https://mlsanity.com/paper/2510.14737/) (2025)
- [Multi-view Multi-label Anomaly Network Traffic Classification based on MLP-Mixer Neural Network](https://mlsanity.com/paper/2210.16719/) (2022)
- [Hierarchy-Consistent Learning and Adaptive Loss Balancing for Hierarchical Multi-Label Classification](https://mlsanity.com/paper/2508.13452/) (2025)
- [Generalized Fine-Grained Category Discovery with Multi-Granularity Conceptual Experts](https://mlsanity.com/paper/2509.26227/) (2025)
- [A Fine-Grained Vehicle Detection (FGVD) Dataset for Unconstrained Roads](https://mlsanity.com/paper/2212.14569/) (2022)
- [Cross-Hierarchical Bidirectional Consistency Learning for Fine-Grained Visual Classification](https://mlsanity.com/paper/2504.13608/) (2025)
- [Semantic Guided Level-Category Hybrid Prediction Network for Hierarchical Image Classification](https://mlsanity.com/paper/2211.12277/) (2022)
- [Research on fault diagnosis of nuclear power first-second circuit based on hierarchical multi-granularity classification network](https://mlsanity.com/paper/2411.07453/) (2024)
- [HMCGeo: IP Region Prediction Based on Hierarchical Multi-label Classification](https://mlsanity.com/paper/2501.16392/) (2025)
- [Semantics-Aware Hierarchical Consensus Learning for Remote Sensing Image Classification](https://mlsanity.com/paper/2510.04916/) (2025)
- [Multi-Granularity Mutual Refinement Network for Zero-Shot Learning](https://mlsanity.com/paper/2511.08163/) (2025)
- [Learning Consistent Taxonomic Classification through Hierarchical Reasoning](https://mlsanity.com/paper/2601.14610/) (2026)
- [HELM: Hierarchical and Explicit Label Modeling with Graph Learning for Multi-Label Image Classification](https://mlsanity.com/paper/2603.11783/) (2026)
- [MAPLE: Multi-Path Adaptive Propagation with Level-Aware Embeddings for Hierarchical Multi-Label Image Classification](https://mlsanity.com/paper/2603.29784/) (2026)
- [Beyond Flat Labels: Level-Restricted Contrastive Learning for Hierarchical Fine-Grained Vision Classification](https://mlsanity.com/paper/2606.21838/) (2026)
- [Learning Taxonomic Trees with Hierarchical Representation Regularization for Large Multimodal Models](https://mlsanity.com/paper/2607.02909/) (2026)
- [Deep learning-based hierarchical insect classification using camera trap imagery](https://mlsanity.com/paper/2607.28005/) (2026)
- [Latent Softmax for Data-Efficient Phoneme-Based Multilingual ASR Across Tonal and Non-Tonal Languages](https://mlsanity.com/paper/2608.01281/) (2026)
- [Label Granularity Skew in Federated Learning with Hierarchical Image Classification](https://mlsanity.com/paper/2608.09236/) (2026)

## Abstract (arXiv)

Hierarchical multi-granularity classification (HMC) assigns hierarchical multi-granularity labels to each object and focuses on encoding the label hierarchy, e.g., ["Albatross", "Laysan Albatross"] from coarse-to-fine levels. However, the definition of what is fine-grained is subjective, and the image quality may affect the identification. Thus, samples could be observed at any level of the hierarchy, e.g., ["Albatross"] or ["Albatross", "Laysan Albatross"], and examples discerned at coarse categories are often neglected in the conventional setting of HMC. In this paper, we study the HMC problem in which objects are labeled at any level of the hierarchy. The essential designs of the proposed method are derived from two motivations: (1) learning with objects labeled at various levels should transfer hierarchical knowledge between levels; (2) lower-level classes should inherit attributes related to upper-level superclasses. The proposed combinatorial loss maximizes the marginal probability of the observed ground truth label by aggregating information from related labels defined in the tree hierarchy. If the observed label is at the leaf level, the combinatorial loss further imposes the multi-class cross-entropy loss to increase the weight of fine-grained classification loss. Considering the hierarchical feature interaction, we propose a hierarchical residual network (HRN), in which granularity-specific features from parent levels acting as residual connections are added to features of children levels. Experiments on three commonly used datasets demonstrate the effectiveness of our approach compared to the state-of-the-art HMC approaches and fine-grained visual classification (FGVC) methods exploiting the label hierarchy.