# Learning Metrics from Teachers: Compact Networks for Image Embedding

arXiv: https://arxiv.org/abs/1904.03624 · HTML: https://mlsanity.com/paper/1904.03624/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Apr 2019.

## Built on by (most cited first, 30 of 39)

- [Universal Representation Learning from Multiple Domains for Few-shot Classification](https://mlsanity.com/paper/2103.13841/) (2021)
- [A Sentence Speaks a Thousand Images: Domain Generalization through Distilling CLIP with Language Guidance](https://mlsanity.com/paper/2309.12530/) (2023)
- [Distilling Object Detectors with Feature Richness](https://mlsanity.com/paper/2111.00674/) (2021)
- [Universal Representations: A Unified Look at Multiple Task and Domain Learning](https://mlsanity.com/paper/2204.02744/) (2022)
- [Asymmetric Temperature Scaling Makes Larger Networks Teach Well Again](https://mlsanity.com/paper/2210.04427/) (2022)
- [Embedding Transfer with Label Relaxation for Improved Metric Learning](https://mlsanity.com/paper/2103.14908/) (2021)
- [Continual Learning for Visual Search with Backward Consistent Feature Embedding](https://mlsanity.com/paper/2205.13384/) (2022)
- [Utilizing Expert Features for Contrastive Learning of Time-Series Representations](https://mlsanity.com/paper/2206.11517/) (2022)
- [Attention Distillation: self-supervised vision transformer students need more guidance](https://mlsanity.com/paper/2210.00944/) (2022)
- [Generalized Knowledge Distillation via Relationship Matching](https://mlsanity.com/paper/2205.01915/) (2022)
- [AMD: Automatic Multi-step Distillation of Large-scale Vision Models](https://mlsanity.com/paper/2407.04208/) (2024)
- [Dual-Tuning: Joint Prototype Transfer and Structure Regularization for Compatible Feature Learning](https://mlsanity.com/paper/2108.02959/) (2021)
- [Teacher-Student Training and Triplet Loss to Reduce the Effect of Drastic Face Occlusion](https://mlsanity.com/paper/2111.10561/) (2021)
- [Semi-Supervised Domain Generalizable Person Re-Identification](https://mlsanity.com/paper/2108.05045/) (2021)
- [Compacting Deep Neural Networks for Internet of Things: Methods and Applications](https://mlsanity.com/paper/2103.11083/) (2021)
- [Exploiting the Semantic Knowledge of Pre-trained Text-Encoders for Continual Learning](https://mlsanity.com/paper/2408.01076/) (2024)
- [Dataset Distillation via Knowledge Distillation: Towards Efficient Self-Supervised Pre-Training of Deep Networks](https://mlsanity.com/paper/2410.02116/) (2024)
- [ProtoCLIP: Prototypical Contrastive Language Image Pretraining](https://mlsanity.com/paper/2206.10996/) (2022)
- [Audio Embeddings as Teachers for Music Classification](https://mlsanity.com/paper/2306.17424/) (2023)
- [Cross-Modal Fusion Distillation for Fine-Grained Sketch-Based Image Retrieval](https://mlsanity.com/paper/2210.10486/) (2022)
- [Deep Boosting Learning: A Brand-new Cooperative Approach for Image-Text Matching](https://mlsanity.com/paper/2404.18114/) (2024)
- [Exploring Dark Knowledge under Various Teacher Capacities and Addressing Capacity Mismatch](https://mlsanity.com/paper/2405.13078/) (2024)
- [Look One and More: Distilling Hybrid Order Relational Knowledge for Cross-Resolution Image Recognition](https://mlsanity.com/paper/2409.05384/) (2024)
- [Collaborative Teacher-Student Learning via Multiple Knowledge Transfer](https://mlsanity.com/paper/2101.08471/) (2021)
- [Weak Augmentation Guided Relational Self-Supervised Learning](https://mlsanity.com/paper/2203.08717/) (2022)
- [Lightning Fast Video Anomaly Detection via Adversarial Knowledge Distillation](https://mlsanity.com/paper/2211.15597/) (2022)
- [InDistill: Information flow-preserving knowledge distillation for model compression](https://mlsanity.com/paper/2205.10003/) (2022)
- [Learning Using Generated Privileged Information by Text-to-Image Diffusion Models](https://mlsanity.com/paper/2309.15238/) (2023)
- [SnapCap: Efficient Snapshot Compressive Video Captioning](https://mlsanity.com/paper/2401.04903/) (2024)
- [Look Through Masks: Towards Masked Face Recognition with De-Occlusion Distillation](https://mlsanity.com/paper/2409.12385/) (2024)

## Abstract (arXiv)

Metric learning networks are used to compute image embeddings, which are widely used in many applications such as image retrieval and face recognition. In this paper, we propose to use network distillation to efficiently compute image embeddings with small networks. Network distillation has been successfully applied to improve image classification, but has hardly been explored for metric learning. To do so, we propose two new loss functions that model the communication of a deep teacher network to a small student network. We evaluate our system in several datasets, including CUB-200-2011, Cars-196, Stanford Online Products and show that embeddings computed using small student networks perform significantly better than those computed using standard networks of similar size. Results on a very compact network (MobileNet-0.25), which can be used on mobile devices, show that the proposed method can greatly improve Recall@1 results from 27.5% to 44.6%. Furthermore, we investigate various aspects of distillation for embeddings, including hint and attention layers, semi-supervised learning and cross quality distillation. (Code is available at https://github.com/yulu0724/EmbeddingDistillation.)