# Descriptor Distillation: a Teacher-Student-Regularized Framework for Learning Local Descriptors

arXiv: https://arxiv.org/abs/2209.11795 · HTML: https://mlsanity.com/paper/2209.11795/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 26 of its 54 references that are in the map.
- First posted on arXiv 23 Sep 2022.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 26 of 26)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [LoFTR: Detector-Free Local Feature Matching with Transformers](https://mlsanity.com/paper/2104.00680/) (2021)
- [DISK: Learning local features with policy gradient](https://mlsanity.com/paper/2006.13566/) (2020)
- [HPatches: A benchmark and evaluation of handcrafted and learned local descriptors](https://mlsanity.com/paper/1704.05939/) (2017)
- [Large-Scale Image Retrieval with Attentive Deep Local Features](https://mlsanity.com/paper/1612.06321/) (2016)
- [Working hard to know your neighbor's margins: Local descriptor learning loss](https://mlsanity.com/paper/1705.10872/) (2017)
- [SOSNet: Second Order Similarity Regularization for Local Descriptor Learning](https://mlsanity.com/paper/1904.05019/) (2019)
- [ASLFeat: Learning Local Features of Accurate Shape and Localization](https://mlsanity.com/paper/2003.10071/) (2020)
- [Knowledge Adaptation for Efficient Semantic Segmentation](https://mlsanity.com/paper/1903.04688/) (2019)
- [ALIKE: Accurate and Lightweight Keypoint Detection and Descriptor Extraction](https://mlsanity.com/paper/2112.02906/) (2021)
- [Fast Human Pose Estimation](https://mlsanity.com/paper/1811.05419/) (2018)
- [Open-Vocabulary One-Stage Detection with Hierarchical Visual-Language Knowledge Distillation](https://mlsanity.com/paper/2203.10593/) (2022)
- [Inter-Region Affinity Distillation for Road Marking Segmentation](https://mlsanity.com/paper/2004.05304/) (2020)
- [GeoDesc: Learning Local Descriptors by Integrating Geometry Constraints](https://mlsanity.com/paper/1807.06294/) (2018)
- [Comprehensive Attention Self-Distillation for Weakly-Supervised Object Detection](https://mlsanity.com/paper/2010.12023/) (2020)
- [SOLAR: Second-Order Loss and Attention for Image Retrieval](https://mlsanity.com/paper/2001.08972/) (2020)
- [Local Descriptors Optimized for Average Precision](https://mlsanity.com/paper/1804.05312/) (2018)
- [Online Invariance Selection for Local Feature Descriptors](https://mlsanity.com/paper/2007.08988/) (2020)
- [Dynamic Kernel Distillation for Efficient Pose Estimation in Videos](https://mlsanity.com/paper/1908.09216/) (2019)
- [A case for using rotation invariant features in state of the art feature matchers](https://mlsanity.com/paper/2204.10144/) (2022)
- [Evaluation-oriented Knowledge Distillation for Deep Face Recognition](https://mlsanity.com/paper/2206.02325/) (2022)
- [Cross-Descriptor Visual Localization and Mapping](https://mlsanity.com/paper/2012.01377/) (2020)
- [Domain-Size Pooling in Local Descriptors: DSP-SIFT](https://mlsanity.com/paper/1412.8556/) (2014)
- [Structured Binary Neural Networks for Image Recognition](https://mlsanity.com/paper/1909.09934/) (2019)
- [How to Train Triplet Networks with 100K Identities?](https://mlsanity.com/paper/1709.02940/) (2017)
- [Robust image stitching with multiple registrations](https://mlsanity.com/paper/2011.11784/) (2020)

## Built on by (most cited first, 2 of 2)

- [Region Matters: Efficient and Reliable Region-Aware Visual Place Recognition](https://mlsanity.com/paper/2604.22390/) (2026)
- [EdgePoint2: Compact Descriptors for Superior Efficiency and Accuracy](https://mlsanity.com/paper/2504.17280/) (2025)

## Abstract (arXiv)

Learning a fast and discriminative patch descriptor is a challenging topic in computer vision. Recently, many existing works focus on training various descriptor learning networks by minimizing a triplet loss (or its variants), which is expected to decrease the distance between each positive pair and increase the distance between each negative pair. However, such an expectation has to be lowered due to the non-perfect convergence of network optimizer to a local solution. Addressing this problem and the open computational speed problem, we propose a Descriptor Distillation framework for local descriptor learning, called DesDis, where a student model gains knowledge from a pre-trained teacher model, and it is further enhanced via a designed teacher-student regularizer. This teacher-student regularizer is to constrain the difference between the positive (also negative) pair similarity from the teacher model and that from the student model, and we theoretically prove that a more effective student model could be trained by minimizing a weighted combination of the triplet loss and this regularizer, than its teacher which is trained by minimizing the triplet loss singly. Under the proposed DesDis, many existing descriptor networks could be embedded as the teacher model, and accordingly, both equal-weight and light-weight student models could be derived, which outperform their teacher in either accuracy or speed. Experimental results on 3 public datasets demonstrate that the equal-weight student models, derived from the proposed DesDis framework by utilizing three typical descriptor learning networks as teacher models, could achieve significantly better performances than their teachers and several other comparative methods. In addition, the derived light-weight models could achieve 8 times or even faster speeds than the comparative methods under similar patch verification performances