# S2SD: Simultaneous Similarity-based Self-Distillation for Deep Metric Learning

arXiv: https://arxiv.org/abs/2009.08348 · HTML: https://mlsanity.com/paper/2009.08348/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2020.

## Built on by (most cited first, 14 of 14)

- [Understanding the Role of the Projector in Knowledge Distillation](https://mlsanity.com/paper/2303.11098/) (2023)
- [TeST: Test-time Self-Training under Distribution Shift](https://mlsanity.com/paper/2209.11459/) (2022)
- [Better Supervisory Signals by Observing Learning Paths](https://mlsanity.com/paper/2203.02485/) (2022)
- [A Non-isotropic Probabilistic Take on Proxy-based Deep Metric Learning](https://mlsanity.com/paper/2207.03784/) (2022)
- [Characterizing Generalization under Out-Of-Distribution Shifts in Deep Metric Learning](https://mlsanity.com/paper/2107.09562/) (2021)
- [Is Fairness Only Metric Deep? Evaluating and Addressing Subgroup Gaps in Deep Metric Learning](https://mlsanity.com/paper/2203.12748/) (2022)
- [Fine-grained Retrieval Prompt Tuning](https://mlsanity.com/paper/2207.14465/) (2022)
- [Learning with Label Noise for Image Retrieval by Selecting Interactions](https://mlsanity.com/paper/2112.10453/) (2021)
- [Supervised Metric Learning to Rank for Retrieval via Contextual Similarity Optimization](https://mlsanity.com/paper/2210.01908/) (2022)
- [Large-to-small Image Resolution Asymmetry in Deep Metric Learning](https://mlsanity.com/paper/2210.05463/) (2022)
- [Deep Metric Learning with Chance Constraints](https://mlsanity.com/paper/2209.09060/) (2022)
- [Generalizable Embeddings with Cross-batch Metric Learning](https://mlsanity.com/paper/2307.07620/) (2023)
- [Generalized Sum Pooling for Metric Learning](https://mlsanity.com/paper/2308.09228/) (2023)
- [Advancing Deep Metric Learning Through Multiple Batch Norms And Multi-Targeted Adversarial Examples](https://mlsanity.com/paper/2211.16253/) (2022)

## Abstract (arXiv)

Deep Metric Learning (DML) provides a crucial tool for visual similarity and zero-shot applications by learning generalizing embedding spaces, although recent work in DML has shown strong performance saturation across training objectives. However, generalization capacity is known to scale with the embedding space dimensionality. Unfortunately, high dimensional embeddings also create higher retrieval cost for downstream applications. To remedy this, we propose emphSimultaneous Similarity-based Self-distillation (S2SD). S2SD extends DML with knowledge distillation from auxiliary, high-dimensional embedding and feature spaces to leverage complementary context during training while retaining test-time cost and with negligible changes to the training time. Experiments and ablations across different objectives and standard benchmarks show S2SD offers notable improvements of up to 7% in Recall@1, while also setting a new state-of-the-art. Code available at https://github.com/MLforHealth/S2SD.