# Hierarchy-based Image Embeddings for Semantic Image Retrieval

arXiv: https://arxiv.org/abs/1809.09924 · HTML: https://mlsanity.com/paper/1809.09924/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Sep 2018.

## Built on by (most cited first, 30 of 41)

- [Deep Hierarchical Semantic Segmentation](https://mlsanity.com/paper/2203.14335/) (2022)
- [Semantics for Robotic Mapping, Perception and Interaction: A Survey](https://mlsanity.com/paper/2101.00443/) (2021)
- [On Semantic Similarity in Video Retrieval](https://mlsanity.com/paper/2103.10095/) (2021)
- [Learning Hierarchy Aware Features for Reducing Mistake Severity](https://mlsanity.com/paper/2207.12646/) (2022)
- [No Cost Likelihood Manipulation at Test Time for Making Better Mistakes in Deep Networks](https://mlsanity.com/paper/2104.00795/) (2021)
- [Integrating Language Guidance into Vision-based Deep Metric Learning](https://mlsanity.com/paper/2203.08543/) (2022)
- [Inducing Neural Collapse to a Fixed Hierarchy-Aware Frame for Reducing Mistake Severity](https://mlsanity.com/paper/2303.05689/) (2023)
- [LCA-on-the-Line: Benchmarking Out-of-Distribution Generalization with Class Taxonomies](https://mlsanity.com/paper/2407.16067/) (2024)
- [SHiNe: Semantic Hierarchy Nexus for Open-vocabulary Object Detection](https://mlsanity.com/paper/2405.10053/) (2024)
- [HELM: Hierarchical Encoding for mRNA Language Modeling](https://mlsanity.com/paper/2410.12459/) (2024)
- [Probabilistic Embeddings for Frozen Vision-Language Models: Uncertainty Quantification with Gaussian Process Latent Variable Models](https://mlsanity.com/paper/2505.05163/) (2025)
- [Hierarchical Few-Shot Object Detection: Problem, Benchmark and Method](https://mlsanity.com/paper/2210.03940/) (2022)
- [boldsymbolλ-Orthogonality Regularization for Compatible Representation Learning](https://mlsanity.com/paper/2509.16664/) (2025)
- [HIERMATCH: Leveraging Label Hierarchies for Improving Semi-Supervised Learning](https://mlsanity.com/paper/2111.00164/) (2021)
- [A Survey of Neural Trees](https://mlsanity.com/paper/2209.03415/) (2022)
- [SEAL: Simultaneous Label Hierarchy Exploration And Learning](https://mlsanity.com/paper/2304.13374/) (2023)
- [Scene Graph Embeddings Using Relative Similarity Supervision](https://mlsanity.com/paper/2104.02381/) (2021)
- [MMF: Multi-Task Multi-Structure Fusion for Hierarchical Image Classification](https://mlsanity.com/paper/2107.00808/) (2021)
- [Test-Time Amendment with a Coarse Classifier for Fine-Grained Classification](https://mlsanity.com/paper/2302.00368/) (2023)
- [Geometry-Aware Adaptation for Pretrained Models](https://mlsanity.com/paper/2307.12226/) (2023)
- [Transformers in Action: Weakly Supervised Action Segmentation](https://mlsanity.com/paper/2201.05675/) (2022)
- [United We Learn Better: Harvesting Learning Improvements From Class Hierarchies Across Tasks](https://mlsanity.com/paper/2107.13627/) (2021)
- [Encoding Hierarchical Information in Neural Networks helps in Subpopulation Shift](https://mlsanity.com/paper/2112.10844/) (2021)
- [Integrating Visual and Semantic Similarity Using Hierarchies for Image Retrieval](https://mlsanity.com/paper/2308.08431/) (2023)
- [Learning Structured Representations by Embedding Class Hierarchy with Fast Optimal Transport](https://mlsanity.com/paper/2410.03052/) (2024)
- [SoilNet: A Multimodal Multitask Model for Hierarchical Classification of Soil Horizons](https://mlsanity.com/paper/2508.03785/) (2025)
- [A Hierarchical Assessment of Adversarial Severity](https://mlsanity.com/paper/2108.11785/) (2021)
- [Semantic-Based Few-Shot Learning by Interactive Psychometric Testing](https://mlsanity.com/paper/2112.09201/) (2021)
- [Is Medieval Distant Viewing Possible? : Extending and Enriching Annotation of Legacy Image Collections using Visual Analytics](https://mlsanity.com/paper/2208.09657/) (2022)
- [Inspecting class hierarchies in classification-based metric learning models](https://mlsanity.com/paper/2301.11065/) (2023)

## Abstract (arXiv)

Deep neural networks trained for classification have been found to learn powerful image representations, which are also often used for other tasks such as comparing images w.r.t. their visual similarity. However, visual similarity does not imply semantic similarity. In order to learn semantically discriminative features, we propose to map images onto class embeddings whose pair-wise dot products correspond to a measure of semantic similarity between classes. Such an embedding does not only improve image retrieval results, but could also facilitate integrating semantics for other tasks, e.g., novelty detection or few-shot learning. We introduce a deterministic algorithm for computing the class centroids directly based on prior world-knowledge encoded in a hierarchy of classes such as WordNet. Experiments on CIFAR-100, NABirds, and ImageNet show that our learned semantic image embeddings improve the semantic consistency of image retrieval results by a large margin.