# Hierarchical Proxy-based Loss for Deep Metric Learning

arXiv: https://arxiv.org/abs/2103.13538 · HTML: https://mlsanity.com/paper/2103.13538/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 46 references that are in the map.
- First posted on arXiv 25 Mar 2021.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [Deep Metric Learning via Lifted Structured Feature Embedding](https://mlsanity.com/paper/1511.06452/) (2015)
- [Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning](https://mlsanity.com/paper/1904.06627/) (2019)
- [A Metric Learning Reality Check](https://mlsanity.com/paper/2003.08505/) (2020)
- [Sampling Matters in Deep Embedding Learning](https://mlsanity.com/paper/1706.07567/) (2017)
- [No Fuss Distance Metric Learning using Proxies](https://mlsanity.com/paper/1703.07464/) (2017)
- [Proxy Anchor Loss for Deep Metric Learning](https://mlsanity.com/paper/2003.13911/) (2020)
- [Beyond triplet loss: a deep quadruplet network for person re-identification](https://mlsanity.com/paper/1704.01719/) (2017)
- [SoftTriple Loss: Deep Metric Learning Without Triplet Sampling](https://mlsanity.com/paper/1909.05235/) (2019)
- [Cross-Batch Memory for Embedding Learning](https://mlsanity.com/paper/1912.06798/) (2019)
- [Deep Metric Learning with Hierarchical Triplet Loss](https://mlsanity.com/paper/1810.06951/) (2018)
- [Self-supervising Fine-grained Region Similarities for Large-scale Image Localization](https://mlsanity.com/paper/2006.03926/) (2020)
- [ProxyNCA++: Revisiting and Revitalizing Proxy Neighborhood Component Analysis](https://mlsanity.com/paper/2004.01113/) (2020)
- [A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses](https://mlsanity.com/paper/2003.08983/) (2020)
- [Online Deep Clustering for Unsupervised Representation Learning](https://mlsanity.com/paper/2006.10645/) (2020)
- [Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrieval](https://mlsanity.com/paper/2007.12163/) (2020)
- [Batch DropBlock Network for Person Re-identification and Beyond](https://mlsanity.com/paper/1811.07130/) (2018)
- [MIC: Mining Interclass Characteristics for Improved Metric Learning](https://mlsanity.com/paper/1909.11574/) (2019)
- [DiVA: Diverse Visual Feature Aggregation for Deep Metric Learning](https://mlsanity.com/paper/2004.13458/) (2020)
- [Divide and Conquer the Embedding Space for Metric Learning](https://mlsanity.com/paper/1906.05990/) (2019)
- [The Group Loss for Deep Metric Learning](https://mlsanity.com/paper/1912.00385/) (2019)

## Built on by (most cited first, 18 of 18)

- [SelEx: Self-Expertise in Fine-Grained Generalized Category Discovery](https://mlsanity.com/paper/2408.14371/) (2024)
- [Artificial Intelligence in Drug Discovery: Applications and Techniques](https://mlsanity.com/paper/2106.05386/) (2021)
- [Learning Hierarchy Aware Features for Reducing Mistake Severity](https://mlsanity.com/paper/2207.12646/) (2022)
- [HIER: Metric Learning Beyond Class Labels via Hierarchical Regularization](https://mlsanity.com/paper/2212.14258/) (2022)
- [Bringing Multimodality to Amazon Visual Search System](https://mlsanity.com/paper/2412.13364/) (2024)
- [Global Proxy-based Hard Mining for Visual Place Recognition](https://mlsanity.com/paper/2302.14217/) (2023)
- [Learning Semantic Proxies from Visual Prompts for Parameter-Efficient Fine-Tuning in Deep Metric Learning](https://mlsanity.com/paper/2402.02340/) (2024)
- [Multi-modal Reference Learning for Fine-grained Text-to-Image Retrieval](https://mlsanity.com/paper/2504.07718/) (2025)
- [Globally Correlation-Aware Hard Negative Generation](https://mlsanity.com/paper/2411.13145/) (2024)
- [Deep Metric Learning with Chance Constraints](https://mlsanity.com/paper/2209.09060/) (2022)
- [Mean Field Theory in Deep Metric Learning](https://mlsanity.com/paper/2306.15368/) (2023)
- [Learning Structured Representations by Embedding Class Hierarchy with Fast Optimal Transport](https://mlsanity.com/paper/2410.03052/) (2024)
- [ProcSim: Proxy-based Confidence for Robust Similarity Learning](https://mlsanity.com/paper/2311.00668/) (2023)
- [Relational Proxies: Emergent Relationships as Fine-Grained Discriminators](https://mlsanity.com/paper/2210.02149/) (2022)
- [Robust Calibrate Proxy Loss for Deep Metric Learning](https://mlsanity.com/paper/2304.09162/) (2023)
- [Distinguishability Calibration to In-Context Learning](https://mlsanity.com/paper/2302.06198/) (2023)
- [Realigned Softmax Warping for Deep Metric Learning](https://mlsanity.com/paper/2408.15656/) (2024)
- [NearID: Identity Representation Learning via Near-identity Distractors](https://mlsanity.com/paper/2604.01973/) (2026)

## Abstract (arXiv)

Proxy-based metric learning losses are superior to pair-based losses due to their fast convergence and low training complexity. However, existing proxy-based losses focus on learning class-discriminative features while overlooking the commonalities shared across classes which are potentially useful in describing and matching samples. Moreover, they ignore the implicit hierarchy of categories in real-world datasets, where similar subordinate classes can be grouped together. In this paper, we present a framework that leverages this implicit hierarchy by imposing a hierarchical structure on the proxies and can be used with any existing proxy-based loss. This allows our model to capture both class-discriminative features and class-shared characteristics without breaking the implicit data hierarchy. We evaluate our method on five established image retrieval datasets such as In-Shop and SOP. Results demonstrate that our hierarchical proxy-based loss framework improves the performance of existing proxy-based losses, especially on large datasets which exhibit strong hierarchical structure.