# Matching Guided Distillation

arXiv: https://arxiv.org/abs/2008.09958 · HTML: https://mlsanity.com/paper/2008.09958/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Aug 2020.

## Built on by (most cited first, 18 of 18)

- [NORM: Knowledge Distillation via N-to-One Representation Matching](https://mlsanity.com/paper/2305.13803/) (2023)
- [Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillation](https://mlsanity.com/paper/2206.06067/) (2022)
- [Learning Compatible Embeddings](https://mlsanity.com/paper/2108.01958/) (2021)
- [Function-Consistent Feature Distillation](https://mlsanity.com/paper/2304.11832/) (2023)
- [Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation](https://mlsanity.com/paper/2306.10687/) (2023)
- [Local Dense Logit Relations for Enhanced Knowledge Distillation](https://mlsanity.com/paper/2507.15911/) (2025)
- [Attention-guided Feature Distillation for Semantic Segmentation](https://mlsanity.com/paper/2403.05451/) (2024)
- [Mitigating Catastrophic Forgetting in the Incremental Learning of Medical Images](https://mlsanity.com/paper/2504.20033/) (2025)
- [AICSD: Adaptive Inter-Class Similarity Distillation for Semantic Segmentation](https://mlsanity.com/paper/2308.04243/) (2023)
- [D-YOLO a robust framework for object detection in adverse weather conditions](https://mlsanity.com/paper/2403.09233/) (2024)
- [FARTrack: Fast Autoregressive Visual Tracking with High Performance](https://mlsanity.com/paper/2602.03214/) (2026)
- [Pixel Distillation: A New Knowledge Distillation Scheme for Low-Resolution Image Recognition](https://mlsanity.com/paper/2112.09532/) (2021)
- [Selective Cross-Task Distillation](https://mlsanity.com/paper/2204.11526/) (2022)
- [Online Cross-Layer Knowledge Distillation on Graph Neural Networks with Deep Supervision](https://mlsanity.com/paper/2210.13743/) (2022)
- [Class-aware Information for Logit-based Knowledge Distillation](https://mlsanity.com/paper/2211.14773/) (2022)
- [A Survey on Recent Teacher-student Learning Studies](https://mlsanity.com/paper/2304.04615/) (2023)
- [Adaptive Teaching with Shared Classifier for Knowledge Distillation](https://mlsanity.com/paper/2406.08528/) (2024)
- [Seeing the Whole Picture: Distribution-Guided Data-Free Distillation for Semantic Segmentation](https://mlsanity.com/paper/2512.13175/) (2025)

## Abstract (arXiv)

Feature distillation is an effective way to improve the performance for a smaller student model, which has fewer parameters and lower computation cost compared to the larger teacher model. Unfortunately, there is a common obstacle - the gap in semantic feature structure between the intermediate features of teacher and student. The classic scheme prefers to transform intermediate features by adding the adaptation module, such as naive convolutional, attention-based or more complicated one. However, this introduces two problems: a) The adaptation module brings more parameters into training. b) The adaptation module with random initialization or special transformation isn't friendly for distilling a pre-trained student. In this paper, we present Matching Guided Distillation (MGD) as an efficient and parameter-free manner to solve these problems. The key idea of MGD is to pose matching the teacher channels with students' as an assignment problem. We compare three solutions of the assignment problem to reduce channels from teacher features with partial distillation loss. The overall training takes a coordinate-descent approach between two optimization objects - assignments update and parameters update. Since MGD only contains normalization or pooling operations with negligible computation cost, it is flexible to plug into network with other distillation methods.