# Improving Generalization of Metric Learning via Listwise Self-distillation

arXiv: https://arxiv.org/abs/2206.08880 · HTML: https://mlsanity.com/paper/2206.08880/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 17 Jun 2022.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 26 of 26)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://mlsanity.com/paper/1503.03832/) (2015)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Deep Metric Learning via Lifted Structured Feature Embedding](https://mlsanity.com/paper/1511.06452/) (2015)
- [Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning](https://mlsanity.com/paper/1904.06627/) (2019)
- [Sampling Matters in Deep Embedding Learning](https://mlsanity.com/paper/1706.07567/) (2017)
- [No Fuss Distance Metric Learning using Proxies](https://mlsanity.com/paper/1703.07464/) (2017)
- [Learning Fine-grained Image Similarity with Deep Ranking](https://mlsanity.com/paper/1404.4661/) (2014)
- [SoftTriple Loss: Deep Metric Learning Without Triplet Sampling](https://mlsanity.com/paper/1909.05235/) (2019)
- [Regularizing Class-wise Predictions via Self-knowledge Distillation](https://mlsanity.com/paper/2003.13964/) (2020)
- [End-to-end Learning of Deep Visual Representations for Image Retrieval](https://mlsanity.com/paper/1610.07940/) (2016)
- [Learning with Symmetric Label Noise: The Importance of Being Unhinged](https://mlsanity.com/paper/1505.07634/) (2015)
- [Self-Knowledge Distillation with Progressive Refinement of Targets](https://mlsanity.com/paper/2006.12000/) (2020)
- [Revisiting Training Strategies and Generalization Performance in Deep Metric Learning](https://mlsanity.com/paper/2002.08473/) (2020)
- [Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrieval](https://mlsanity.com/paper/2007.12163/) (2020)
- [Optimizing Rank-based Metrics with Blackbox Differentiation](https://mlsanity.com/paper/1912.03500/) (2019)
- [Attention-based Ensemble for Deep Metric Learning](https://mlsanity.com/paper/1804.00382/) (2018)
- [MIC: Mining Interclass Characteristics for Improved Metric Learning](https://mlsanity.com/paper/1909.11574/) (2019)
- [Deep Metric Learning with BIER: Boosting Independent Embeddings Robustly](https://mlsanity.com/paper/1801.04815/) (2018)
- [DiVA: Diverse Visual Feature Aggregation for Deep Metric Learning](https://mlsanity.com/paper/2004.13458/) (2020)
- [Self-distillation with Batch Knowledge Ensembling Improves ImageNet Classification](https://mlsanity.com/paper/2104.13298/) (2021)

## Built on by (most cited first, 1 of 1)

- [ProcSim: Proxy-based Confidence for Robust Similarity Learning](https://mlsanity.com/paper/2311.00668/) (2023)

## Abstract (arXiv)

Most deep metric learning (DML) methods employ a strategy that forces all positive samples to be close in the embedding space while keeping them away from negative ones. However, such a strategy ignores the internal relationships of positive (negative) samples and often leads to overfitting, especially in the presence of hard samples and mislabeled samples. In this work, we propose a simple yet effective regularization, namely Listwise Self-Distillation (LSD), which progressively distills a model's own knowledge to adaptively assign a more appropriate distance target to each sample pair in a batch. LSD encourages smoother embeddings and information mining within positive (negative) samples as a way to mitigate overfitting and thus improve generalization. Our LSD can be directly integrated into general DML frameworks. Extensive experiments show that LSD consistently boosts the performance of various metric learning methods on multiple datasets.