# DarkRank: Accelerating Deep Metric Learning via Cross Sample Similarities Transfer

arXiv: https://arxiv.org/abs/1707.01220 · HTML: https://mlsanity.com/paper/1707.01220/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jul 2017.

## Built on by (most cited first, 30 of 56)

- [A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance](https://mlsanity.com/paper/2309.12530/) (2023)
- [Learning Compatible Embeddings](https://mlsanity.com/paper/2108.01958/) (2021)
- [Deep Spatially and Temporally Aware Similarity Computation for Road Network Constrained Trajectories](https://mlsanity.com/paper/2112.09339/) (2021)
- [Content-Aware GAN Compression](https://mlsanity.com/paper/2104.02244/) (2021)
- [Teacher-Student Architecture for Knowledge Distillation: A Survey](https://mlsanity.com/paper/2308.04268/) (2023)
- [Distribution-sensitive Information Retention for Accurate Binary Neural Network](https://mlsanity.com/paper/2109.12338/) (2021)
- [Revisiting Self-Distillation](https://mlsanity.com/paper/2206.08491/) (2022)
- [Teacher-Student Architecture for Knowledge Learning: A Survey](https://mlsanity.com/paper/2210.17332/) (2022)
- [VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer](https://mlsanity.com/paper/2107.02681/) (2021)
- [BiBench: Benchmarking and Analyzing Network Binarization](https://mlsanity.com/paper/2301.11233/) (2023)
- [Embedding Transfer with Label Relaxation for Improved Metric Learning](https://mlsanity.com/paper/2103.14908/) (2021)
- [Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation](https://mlsanity.com/paper/2306.10687/) (2023)
- [Integrating Language Guidance into Vision-based Deep Metric Learning](https://mlsanity.com/paper/2203.08543/) (2022)
- [Evaluation-oriented Knowledge Distillation for Deep Face Recognition](https://mlsanity.com/paper/2206.02325/) (2022)
- [Self-Regulation for Semantic Segmentation](https://mlsanity.com/paper/2108.09702/) (2021)
- [Continual Learning for Visual Search with Backward Consistent Feature Embedding](https://mlsanity.com/paper/2205.13384/) (2022)
- [New Insights for the Stability-Plasticity Dilemma in Online Continual Learning](https://mlsanity.com/paper/2302.08741/) (2023)
- [Semantic Relation Preserving Knowledge Distillation for Image-to-Image Translation](https://mlsanity.com/paper/2104.15082/) (2021)
- [Distillation from Heterogeneous Models for Top-K Recommendation](https://mlsanity.com/paper/2303.01130/) (2023)
- [SERE: Exploring Feature Self-relation for Self-supervised Transformer](https://mlsanity.com/paper/2206.05184/) (2022)
- [Exploiting User Comments for Early Detection of Fake News Prior to Users' Commenting](https://mlsanity.com/paper/2310.10429/) (2023)
- [VGSG: Vision-Guided Semantic-Group Network for Text-based Person Search](https://mlsanity.com/paper/2311.07514/) (2023)
- [Cross Architecture Distillation for Face Recognition](https://mlsanity.com/paper/2306.14662/) (2023)
- [Hot-Refresh Model Upgrades with Regression-Alleviating Compatible Training in Image Retrieval](https://mlsanity.com/paper/2201.09724/) (2022)
- [VLM-KD: Knowledge Distillation from VLM for Long-Tail Visual Recognition](https://mlsanity.com/paper/2408.16930/) (2024)
- [Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-Training of Deep Networks](https://mlsanity.com/paper/2410.02116/) (2024)
- [Grouped Knowledge Distillation for Deep Face Recognition](https://mlsanity.com/paper/2304.04462/) (2023)
- [Simple Unsupervised Knowledge Distillation With Space Similarity](https://mlsanity.com/paper/2409.13939/) (2024)
- [Optimising TinyML with Quantization and Distillation of Transformer and Mamba Models for Indoor Localisation on Edge Devices](https://mlsanity.com/paper/2412.09289/) (2024)
- [Deep Boosting Learning: A Brand-new Cooperative Approach for Image-Text Matching](https://mlsanity.com/paper/2404.18114/) (2024)

## Abstract (arXiv)

We have witnessed rapid evolution of deep neural network architecture design in the past years. These latest progresses greatly facilitate the developments in various areas such as computer vision and natural language processing. However, along with the extraordinary performance, these state-of-the-art models also bring in expensive computational cost. Directly deploying these models into applications with real-time requirement is still infeasible. Recently, Hinton etal. have shown that the dark knowledge within a powerful teacher model can significantly help the training of a smaller and faster student network. These knowledge are vastly beneficial to improve the generalization ability of the student model. Inspired by their work, we introduce a new type of knowledge -- cross sample similarities for model compression and acceleration. This knowledge can be naturally derived from deep metric learning model. To transfer them, we bring the "learning to rank" technique into deep metric learning formulation. We test our proposed DarkRank method on various metric learning tasks including pedestrian re-identification, image retrieval and image clustering. The results are quite encouraging. Our method can improve over the baseline method by a large margin. Moreover, it is fully compatible with other existing methods. When combined, the performance can be further boosted.