# Distilling Knowledge by Mimicking Features

arXiv: https://arxiv.org/abs/2011.01424 · HTML: https://mlsanity.com/paper/2011.01424/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Nov 2020.

## Built on by (most cited first, 30 of 35)

- [EVC: Towards Real-Time Neural Image Compression with Mask Decay](https://mlsanity.com/paper/2302.05071/) (2023)
- [LLaVA-KD: A Framework of Distilling Multimodal Large Language Models](https://mlsanity.com/paper/2410.16236/) (2024)
- [Towards Efficient Post-training Quantization of Pre-trained Language Models](https://mlsanity.com/paper/2109.15082/) (2021)
- [Quantization without Tears](https://mlsanity.com/paper/2411.13918/) (2024)
- [Long-Tailed Continual Learning For Visual Food Recognition](https://mlsanity.com/paper/2307.00183/) (2023)
- [Localization Distillation for Object Detection](https://mlsanity.com/paper/2204.05957/) (2022)
- [Skeleton-based Action Recognition via Adaptive Cross-Form Learning](https://mlsanity.com/paper/2206.15085/) (2022)
- [IDa-Det: An Information Discrepancy-aware Distillation for 1-bit Detectors](https://mlsanity.com/paper/2210.03477/) (2022)
- [Guided Hybrid Quantization for Object detection in Multimodal Remote Sensing Imagery via One-to-one Self-teaching](https://mlsanity.com/paper/2301.00131/) (2022)
- [Practical Network Acceleration with Tiny Sets](https://mlsanity.com/paper/2202.07861/) (2022)
- [GenRecal: Generation after Recalibration from Large to Small Vision-Language Models](https://mlsanity.com/paper/2506.15681/) (2025)
- [Towards Real-Time Neural Video Codec for Cross-Platform Application Using Calibration Information](https://mlsanity.com/paper/2309.11276/) (2023)
- [Compressing Models with Few Samples: Mimicking then Replacing](https://mlsanity.com/paper/2201.02620/) (2022)
- [GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers](https://mlsanity.com/paper/2506.11784/) (2025)
- [All You Need in Knowledge Distillation Is a Tailored Coordinate System](https://mlsanity.com/paper/2412.09388/) (2024)
- [Masking Teacher and Reinforcing Student for Distilling Vision-Language Models](https://mlsanity.com/paper/2512.22238/) (2025)
- [Towards Efficient Task-Driven Model Reprogramming with Foundation Models](https://mlsanity.com/paper/2304.02263/) (2023)
- [Shape-intensity knowledge distillation for robust medical image segmentation](https://mlsanity.com/paper/2409.17503/) (2024)
- [Why does Knowledge Distillation Work? Rethink its Attention and Fidelity Mechanism](https://mlsanity.com/paper/2405.00739/) (2024)
- [Overcoming Uncertain Incompleteness for Robust Multimodal Sequential Diagnosis Prediction via Curriculum Data Erasing Guided Knowledge Distillation](https://mlsanity.com/paper/2407.19540/) (2024)
- [Vision-Based Detection of Uncooperative Targets and Components on Small Satellites](https://mlsanity.com/paper/2408.12084/) (2024)
- [Practical Network Acceleration with Tiny Sets: Hypothesis, Theory, and Algorithm](https://mlsanity.com/paper/2303.00972/) (2023)
- [Switch-KD: Visual-Switch Knowledge Distillation for Vision-Language Models](https://mlsanity.com/paper/2604.14629/) (2026)
- [Optimizing for In-memory Deep Learning with Emerging Memory Technology](https://mlsanity.com/paper/2112.00324/) (2021)
- [Architectural Insights into Knowledge Distillation for Object Detection: A Comprehensive Review](https://mlsanity.com/paper/2508.03317/) (2025)
- [CanKD: Cross-Attention-based Non-local operation for Feature-based Knowledge Distillation](https://mlsanity.com/paper/2511.21503/) (2025)
- [Nonlinear Bipolar Compensation: Handling Outliers in Post-Training Quantization](https://mlsanity.com/paper/2605.16423/) (2026)
- [Ground Reaction Force Estimation via Time-aware Knowledge Distillation](https://mlsanity.com/paper/2506.10265/) (2025)
- [When Big Models Train Small Ones: Label-Free Model Parity Alignment for Efficient Visual Question Answering using Small VLMs](https://mlsanity.com/paper/2509.16633/) (2025)
- [Rethinking Decoupled Knowledge Distillation: A Predictive Distribution Perspective](https://mlsanity.com/paper/2512.04625/) (2025)

## Abstract (arXiv)

Knowledge distillation (KD) is a popular method to train efficient networks ("student") with the help of high-capacity networks ("teacher"). Traditional methods use the teacher's soft logits as extra supervision to train the student network. In this paper, we argue that it is more advantageous to make the student mimic the teacher's features in the penultimate layer. Not only the student can directly learn more effective information from the teacher feature, feature mimicking can also be applied for teachers trained without a softmax layer. Experiments show that it can achieve higher accuracy than traditional KD. To further facilitate feature mimicking, we decompose a feature vector into the magnitude and the direction. We argue that the teacher should give more freedom to the student feature's magnitude, and let the student pay more attention on mimicking the feature direction. To meet this requirement, we propose a loss term based on locality-sensitive hashing (LSH). With the help of this new loss, our method indeed mimics feature directions more accurately, relaxes constraints on feature magnitudes, and achieves state-of-the-art distillation accuracy. We provide theoretical analyses of how LSH facilitates feature direction mimicking, and further extend feature mimicking to multi-label recognition and object detection.