# What Drives Hierarchy-Aware Image Retrieval? Taxonomy Alignment, Objective Choice, and Geometry

arXiv: https://arxiv.org/abs/2609.25638 · HTML: https://mlsanity.com/paper/2609.25638/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 22 references that are in the map.
- First posted on arXiv 22 Sep 2026.

## Builds on (most cited first, 17 of 17)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [Hyperbolic Neural Networks](https://mlsanity.com/paper/1805.09112/) (2018)
- [Hyperbolic Image Embeddings](https://mlsanity.com/paper/1904.02239/) (2019)
- [Proxy Anchor Loss for Deep Metric Learning](https://mlsanity.com/paper/2003.13911/) (2020)
- [Hyperbolic Vision Transformers: Combining Improvements in Metric Learning](https://mlsanity.com/paper/2203.10833/) (2022)
- [Hyperbolic Deep Learning in Computer Vision: A Survey](https://mlsanity.com/paper/2305.06611/) (2023)
- [Hyperbolic Contrastive Learning for Visual Representations beyond Objects](https://mlsanity.com/paper/2212.00653/) (2022)
- [HIER: Metric Learning Beyond Class Labels via Hierarchical Regularization](https://mlsanity.com/paper/2212.14258/) (2022)
- [Visually Consistent Hierarchical Image Classification](https://mlsanity.com/paper/2406.11608/) (2024)
- [Improving Visual Recognition with Hyperbolical Visual Hierarchy Mapping](https://mlsanity.com/paper/2404.00974/) (2024)
- [Understanding Hyperbolic Metric Learning through Hard Negative Sampling](https://mlsanity.com/paper/2404.15523/) (2024)
- [Hyperbolic Space with Hierarchical Margin Boosts Fine-Grained Learning from Coarse Labels](https://mlsanity.com/paper/2311.11019/) (2023)
- [Learning Visual Hierarchies in Hyperbolic Space for Image Retrieval](https://mlsanity.com/paper/2411.17490/) (2024)
- [Taxonomy-Aware Representation Alignment for Hierarchical Visual Recognition with Large Multimodal Models](https://mlsanity.com/paper/2603.00431/) (2026)
- [Free-Grained Hierarchical Visual Recognition](https://mlsanity.com/paper/2510.14737/) (2025)

## Abstract (arXiv)

Foundation vision models provide strong generic representations, yet high class-level retrieval accuracy does not necessarily imply that an embedding respects a target semantic taxonomy. We study strict explicit-taxonomy image retrieval on frozen DINOv2 features and ask: when hierarchical retrieval improves, how much of the change is associated with the organization of taxonomy-aware supervision, and how much with the Euclidean-hyperbolic geometry choice? We evaluate higher levels with strict cross-class criteria that exclude finer-grained matches, and compare Euclidean and hyperbolic projections trained with taxonomy-distance regression or a taxonomy-aware supervised contrastive objective. A compute-matched 2 x 2 Geometry x Loss factorial uses the same 768-256-32 projector capacity, optimization schedule, batch order, and fixed 100-epoch budget; the Loss axis denotes the Regression-to-Taxonomy-SupCon objective-family contrast. On CUB, the objective-family contrasts in mean hierarchy mAP (strict middle/high average, excluding Class/Leaf) are +0.0487 in Euclidean space and +0.0414 in hyperbolic space, compared with geometry contrasts of +0.0102 and +0.0030. On NABirds Parent-disjoint retrieval, the corresponding objective-family contrasts are +0.0467 and +0.0440, whereas geometry contrasts are +0.0017 and -0.0009. A semantic-alignment control shows that the true taxonomy substantially outperforms a structure-preserving shuffled hierarchy, while a NABirds curvature/radius control does not support stronger negative curvature as the explanation for the observed hierarchy gains. Across the two taxonomies, the Regression-to-Taxonomy-SupCon contrasts are larger in aggregate than the evaluated geometry contrasts; semantic alignment also matters separately, while geometry remains hierarchy-dependent.