# Do Convolutional Neural Networks Learn Class Hierarchy?

arXiv: https://arxiv.org/abs/1710.06501 · HTML: https://mlsanity.com/paper/1710.06501/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Oct 2017.

## Built on by (most cited first, 30 of 61)

- [BioCLIP: A Vision Foundation Model for the Tree of Life](https://mlsanity.com/paper/2311.18803/) (2023)
- [Deep Hierarchical Semantic Segmentation](https://mlsanity.com/paper/2203.14335/) (2022)
- [Adaptive Inference through Early-Exit Networks: Design, Challenges and Directions](https://mlsanity.com/paper/2106.05022/) (2021)
- [Use All The Labels: A Hierarchical Multi-Label Contrastive Learning Framework](https://mlsanity.com/paper/2204.13207/) (2022)
- [Self-Damaging Contrastive Learning](https://mlsanity.com/paper/2106.02990/) (2021)
- [Improving neural network representations using human similarity judgments](https://mlsanity.com/paper/2306.04507/) (2023)
- [M2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysis](https://mlsanity.com/paper/2107.08264/) (2021)
- [The State of the Art in Enhancing Trust in Machine Learning Models with the Use of Visualizations](https://mlsanity.com/paper/2212.11737/) (2022)
- [Aligning Eyes between Humans and Deep Neural Network through Interactive Attention Alignment](https://mlsanity.com/paper/2202.02838/) (2022)
- [Learning Hierarchy Aware Features for Reducing Mistake Severity](https://mlsanity.com/paper/2207.12646/) (2022)
- [Visual Analytics For Machine Learning: A Data Perspective Survey](https://mlsanity.com/paper/2307.07712/) (2023)
- [No Cost Likelihood Manipulation at Test Time for Making Better Mistakes in Deep Networks](https://mlsanity.com/paper/2104.00795/) (2021)
- [Neo: Generalizing Confusion Matrix Visualization to Hierarchical and Multi-Output Labels](https://mlsanity.com/paper/2110.12536/) (2021)
- [Diagnosing Ensemble Few-Shot Classifiers](https://mlsanity.com/paper/2206.04372/) (2022)
- [Inducing Neural Collapse to a Fixed Hierarchy-Aware Frame for Reducing Mistake Severity](https://mlsanity.com/paper/2303.05689/) (2023)
- [On Label Granularity and Object Localization](https://mlsanity.com/paper/2207.10225/) (2022)
- [VAC-CNN: A Visual Analytics System for Comparative Studies of Deep Convolutional Neural Networks](https://mlsanity.com/paper/2110.13252/) (2021)
- [Coarse-to-Fine Curriculum Learning](https://mlsanity.com/paper/2106.04072/) (2021)
- [Fine-grain Inference on Out-of-Distribution Data with Hierarchical Classification](https://mlsanity.com/paper/2209.04493/) (2022)
- [E-PixelHop: An Enhanced PixelHop Method for Object Classification](https://mlsanity.com/paper/2107.02966/) (2021)
- [A Unified Interactive Model Evaluation for Classification, Object Detection, and Instance Segmentation in Computer Vision](https://mlsanity.com/paper/2308.05168/) (2023)
- [HIERMATCH: Leveraging Label Hierarchies for Improving Semi-Supervised Learning](https://mlsanity.com/paper/2111.00164/) (2021)
- [A Survey of Neural Trees](https://mlsanity.com/paper/2209.03415/) (2022)
- [SEAL: Simultaneous Label Hierarchy Exploration And Learning](https://mlsanity.com/paper/2304.13374/) (2023)
- [Test-Time Amendment with a Coarse Classifier for Fine-Grained Classification](https://mlsanity.com/paper/2302.00368/) (2023)
- [Analyzing Hierarchical Structure in Vision Models with Sparse Autoencoders](https://mlsanity.com/paper/2505.15970/) (2025)
- [Visual Analytics for Efficient Image Exploration and User-Guided Image Captioning](https://mlsanity.com/paper/2311.01016/) (2023)
- [Reviving the Context: Camera Trap Species Classification as Link Prediction on Multimodal Knowledge Graphs](https://mlsanity.com/paper/2401.00608/) (2023)
- [United We Learn Better: Harvesting Learning Improvements From Class Hierarchies Across Tasks](https://mlsanity.com/paper/2107.13627/) (2021)
- [Label Hierarchy Transition: Delving into Class Hierarchies to Enhance Deep Classifiers](https://mlsanity.com/paper/2112.02353/) (2021)

## Abstract (arXiv)

Convolutional Neural Networks (CNNs) currently achieve state-of-the-art accuracy in image classification. With a growing number of classes, the accuracy usually drops as the possibilities of confusion increase. Interestingly, the class confusion patterns follow a hierarchical structure over the classes. We present visual-analytics methods to reveal and analyze this hierarchy of similar classes in relation with CNN-internal data. We found that this hierarchy not only dictates the confusion patterns between the classes, it furthermore dictates the learning behavior of CNNs. In particular, the early layers in these networks develop feature detectors that can separate high-level groups of classes quite well, even after a few training epochs. In contrast, the latter layers require substantially more epochs to develop specialized feature detectors that can separate individual classes. We demonstrate how these insights are key to significant improvement in accuracy by designing hierarchy-aware CNNs that accelerate model convergence and alleviate overfitting. We further demonstrate how our methods help in identifying various quality issues in the training data.