# Learning Visual Hierarchies in Hyperbolic Space for Image Retrieval

arXiv: https://arxiv.org/abs/2411.17490 · HTML: https://mlsanity.com/paper/2411.17490/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 52 references that are in the map.
- First posted on arXiv 26 Nov 2024.

## Builds on (most cited first, 30 of 33)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Improved Baselines with Momentum Contrastive Learning](https://mlsanity.com/paper/2003.04297/) (2020)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [Transformers in Vision: A Survey](https://mlsanity.com/paper/2101.01169/) (2021)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [Deep metric learning using Triplet network](https://mlsanity.com/paper/1412.6622/) (2014)
- [Hyperbolic Neural Networks](https://mlsanity.com/paper/1805.09112/) (2018)
- [Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry](https://mlsanity.com/paper/1806.03417/) (2018)
- [Hyperbolic Graph Neural Networks](https://mlsanity.com/paper/1910.12892/) (2019)
- [Representation Tradeoffs for Hyperbolic Embeddings](https://mlsanity.com/paper/1804.03329/) (2018)
- [Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild](https://mlsanity.com/paper/1906.02569/) (2019)
- [Hyperbolic Entailment Cones for Learning Hierarchical Embeddings](https://mlsanity.com/paper/1804.01882/) (2018)
- [Poincar'e GloVe: Hyperbolic Word Embeddings](https://mlsanity.com/paper/1810.06546/) (2018)
- [Hyperbolic Image-Text Representations](https://mlsanity.com/paper/2304.09172/) (2023)
- [Compositional Entailment Learning for Hyperbolic Vision-Language Models](https://mlsanity.com/paper/2410.06912/) (2024)
- [Probabilistic Embedding of Knowledge Graphs with Box Lattice Measures](https://mlsanity.com/paper/1805.06627/) (2018)
- [Hyperbolic Contrastive Learning for Visual Representations beyond Objects](https://mlsanity.com/paper/2212.00653/) (2022)
- [Clipped Hyperbolic Classifiers Are Super-Hyperbolic Classifiers](https://mlsanity.com/paper/2107.11472/) (2021)
- [Hierarchical Image Classification using Entailment Cone Embeddings](https://mlsanity.com/paper/2004.03459/) (2020)
- [Robust Contrastive Learning Using Negative Samples with Diminished Semantics](https://mlsanity.com/paper/2110.14189/) (2021)
- [HyperText: Endowing FastText with Hyperbolic Geometry](https://mlsanity.com/paper/2010.16143/) (2020)
- [HIER: Metric Learning Beyond Class Labels via Hierarchical Regularization](https://mlsanity.com/paper/2212.14258/) (2022)
- [Capturing implicit hierarchical structure in 3D biomedical images with self-supervised hyperbolic representations](https://mlsanity.com/paper/2012.01644/) (2020)
- [HypLiLoc: Towards Effective LiDAR Pose Regression with Hyperbolic Fusion](https://mlsanity.com/paper/2304.00932/) (2023)
- [Skin Lesion Recognition with Class-Hierarchy Regularized Hyperbolic Embeddings](https://mlsanity.com/paper/2209.05842/) (2022)

## Built on by (most cited first, 8 of 8)

- [Fast and Geometrically Grounded Lorentz Neural Networks](https://mlsanity.com/paper/2601.21529/) (2026)
- [H2em: Learning Hierarchical Hyperbolic Embeddings for Compositional Zero-Shot Learning](https://mlsanity.com/paper/2512.20029/) (2025)
- [A Hyperbolic Perspective on Hierarchical Structure in Object-Centric Scene Representations](https://mlsanity.com/paper/2603.14022/) (2026)
- [Semantic Alignment in Hyperbolic Space for Open-Vocabulary Semantic Segmentation](https://mlsanity.com/paper/2605.08874/) (2026)
- [HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding](https://mlsanity.com/paper/2607.00428/) (2026)
- [HSA: Hierarchical Slot Attention for Multi-granularity Scene-Decomposition](https://mlsanity.com/paper/2607.08249/) (2026)
- [Hierarchical Prompt Learning for Hyperbolic Vision-Language Models](https://mlsanity.com/paper/2609.24276/) (2026)
- [What Drives Hierarchy-Aware Image Retrieval? Taxonomy Alignment, Objective Choice, and Geometry](https://mlsanity.com/paper/2609.25638/) (2026)

## Abstract (arXiv)

Structuring latent representations in a hierarchical manner enables models to learn patterns at multiple levels of abstraction. However, most prevalent image understanding models focus on visual similarity, and learning visual hierarchies is relatively unexplored. In this work, for the first time, we introduce a learning paradigm that can encode user-defined multi-level complex visual hierarchies in hyperbolic space without requiring explicit hierarchical labels. As a concrete example, first, we define a part-based image hierarchy using object-level annotations within and across images. Then, we introduce an approach to enforce the hierarchy using contrastive loss with pairwise entailment metrics. Finally, we discuss new evaluation metrics to effectively measure hierarchical image retrieval. Encoding these complex relationships ensures that the learned representations capture semantic and structural information that transcends mere visual similarity. Experiments in part-based image retrieval show significant improvements in hierarchical retrieval tasks, demonstrating the capability of our model in capturing visual hierarchies.