# Hierarchical classification at multiple operating points

arXiv: https://arxiv.org/abs/2210.10929 · HTML: https://mlsanity.com/paper/2210.10929/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 44 references that are in the map.
- First posted on arXiv 19 Oct 2022.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [YOLO9000: Better, Faster, Stronger](https://mlsanity.com/paper/1612.08242/) (2016)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [Long-tail learning via logit adjustment](https://mlsanity.com/paper/2007.07314/) (2020)
- [Benchmarking Representation Learning for Natural World Image Collections](https://mlsanity.com/paper/2103.16483/) (2021)
- [Making Better Mistakes: Leveraging Class Hierarchies with Deep Networks](https://mlsanity.com/paper/1912.09393/) (2019)
- [Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrieval](https://mlsanity.com/paper/2007.12163/) (2020)
- [Open Vocabulary Scene Parsing](https://mlsanity.com/paper/1703.08769/) (2017)
- [Classical Structured Prediction Losses for Sequence to Sequence Learning](https://mlsanity.com/paper/1711.04956/) (2017)
- [Network of Experts for Large-Scale Image Categorization](https://mlsanity.com/paper/1604.06119/) (2016)
- [Evaluation Measures for Hierarchical Classification: a unified view and novel approaches](https://mlsanity.com/paper/1306.6802/) (2013)
- [Classifying and Segmenting Microscopy Images Using Convolutional Multiple Instance Learning](https://mlsanity.com/paper/1511.05286/) (2015)
- [B-CNN: Branch Convolutional Neural Network for Hierarchical Classification](https://mlsanity.com/paper/1709.09890/) (2017)
- [Solving Long-tailed Recognition with Deep Realistic Taxonomic Classifier](https://mlsanity.com/paper/2007.09898/) (2020)
- [Integrating domain knowledge: using hierarchies to improve deep classifiers](https://mlsanity.com/paper/1811.07125/) (2018)
- [No Cost Likelihood Manipulation at Test Time for Making Better Mistakes in Deep Networks](https://mlsanity.com/paper/2104.00795/) (2021)
- [A hierarchical loss and its problems when classifying non-hierarchically](https://mlsanity.com/paper/1709.01062/) (2017)

## Built on by (most cited first, 14 of 14)

- [Visually Consistent Hierarchical Image Classification](https://mlsanity.com/paper/2406.11608/) (2024)
- [LCA-on-the-Line: Benchmarking Out-of-Distribution Generalization with Class Taxonomies](https://mlsanity.com/paper/2407.16067/) (2024)
- [Flattening the Parent Bias: Hierarchical Semantic Segmentation in the Poincar'e Ball](https://mlsanity.com/paper/2404.03778/) (2024)
- [Hierarchical Selective Classification](https://mlsanity.com/paper/2405.11533/) (2024)
- [Revisiting Hierarchical Text Classification: Inference and Metrics](https://mlsanity.com/paper/2410.01305/) (2024)
- [Test-Time Amendment with a Coarse Classifier for Fine-Grained Classification](https://mlsanity.com/paper/2302.00368/) (2023)
- [Conformal Prediction in Hierarchical Classification with Constrained Representation Complexity](https://mlsanity.com/paper/2501.19038/) (2025)
- [Do Schwartz Higher-Order Values Help Sentence-Level Human Value Detection? A Study of Hierarchical Gating and Calibration](https://mlsanity.com/paper/2602.00913/) (2026)
- [Tree semantic segmentation from aerial image time series](https://mlsanity.com/paper/2407.13102/) (2024)
- [T-DuMpRa: Teacher-guided Dual-path Multi-prototype Retrieval Augmented framework for fine-grained medical image classification](https://mlsanity.com/paper/2604.17360/) (2026)
- [Enhancing VICReg: Random-Walk Pairing for Improved Generalization and Better Global Semantics Capturing](https://mlsanity.com/paper/2506.18104/) (2025)
- [Hierarchy-Aware Fine-Tuning of Vision-Language Models](https://mlsanity.com/paper/2512.21529/) (2025)
- [TaxDistill: Improving Metagenomic Taxonomic Annotation via Distilled Genomic Foundation Models](https://mlsanity.com/paper/2605.28868/) (2026)
- [Label Granularity Skew in Federated Learning with Hierarchical Image Classification](https://mlsanity.com/paper/2608.09236/) (2026)

## Abstract (arXiv)

Many classification problems consider classes that form a hierarchy. Classifiers that are aware of this hierarchy may be able to make confident predictions at a coarse level despite being uncertain at the fine-grained level. While it is generally possible to vary the granularity of predictions using a threshold at inference time, most contemporary work considers only leaf-node prediction, and almost no prior work has compared methods at multiple operating points. We present an efficient algorithm to produce operating characteristic curves for any method that assigns a score to every class in the hierarchy. Applying this technique to evaluate existing methods reveals that top-down classifiers are dominated by a naive flat softmax classifier across the entire operating range. We further propose two novel loss functions and show that a soft variant of the structured hinge loss is able to significantly outperform the flat baseline. Finally, we investigate the poor accuracy of top-down classifiers and demonstrate that they perform relatively well on unseen classes. Code is available online at https://github.com/jvlmdr/hiercls.