# Student Network Learning via Evolutionary Knowledge Distillation

arXiv: https://arxiv.org/abs/2103.13811 · HTML: https://mlsanity.com/paper/2103.13811/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 53 of its 83 references that are in the map.
- First posted on arXiv 23 Mar 2021.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 53)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [CosFace: Large Margin Cosine Loss for Deep Face Recognition](https://mlsanity.com/paper/1801.09414/) (2018)
- [VGGFace2: A dataset for recognising faces across pose and age](https://mlsanity.com/paper/1710.08092/) (2017)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Similarity-Preserving Knowledge Distillation](https://mlsanity.com/paper/1907.09682/) (2019)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Variational Information Distillation for Knowledge Transfer](https://mlsanity.com/paper/1904.05835/) (2019)
- [Model compression via distillation and quantization](https://mlsanity.com/paper/1802.05668/) (2018)
- [Correlation Congruence for Knowledge Distillation](https://mlsanity.com/paper/1904.01802/) (2019)
- [Structured Knowledge Distillation for Dense Prediction](https://mlsanity.com/paper/1903.04197/) (2019)
- [Paraphrasing Complex Network: Network Compression via Factor Transfer](https://mlsanity.com/paper/1802.04977/) (2018)
- [Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons](https://mlsanity.com/paper/1811.03233/) (2018)
- [Learning Deep Representations with Probabilistic Knowledge Transfer](https://mlsanity.com/paper/1803.10837/) (2018)
- [Knowledge Distillation by On-the-Fly Native Ensemble](https://mlsanity.com/paper/1806.04606/) (2018)
- [Large scale distributed neural network training through online distillation](https://mlsanity.com/paper/1804.03235/) (2018)
- [Regularizing Class-wise Predictions via Self-knowledge Distillation](https://mlsanity.com/paper/2003.13964/) (2020)

## Built on by (most cited first, 30 of 30)

- [Selective-Supervised Contrastive Learning with Noisy Labels](https://mlsanity.com/paper/2203.04181/) (2022)
- [Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition](https://mlsanity.com/paper/2207.11518/) (2022)
- [WebUOT-1M: Advancing Deep Underwater Object Tracking with A Million-Scale Benchmark](https://mlsanity.com/paper/2405.19818/) (2024)
- [Label Semantic Knowledge Distillation for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2208.03763/) (2022)
- [Knowledge Distillation Using Hierarchical Self-Supervision Augmented Distribution](https://mlsanity.com/paper/2109.03075/) (2021)
- [Is Complexity Required for Neural Network Pruning? A Case Study on Global Magnitude Pruning](https://mlsanity.com/paper/2209.14624/) (2022)
- [MDFlow: Unsupervised Optical Flow Learning by Reliable Mutual Knowledge Distillation](https://mlsanity.com/paper/2211.06018/) (2022)
- [Self-Supervised Visual Representation Learning via Residual Momentum](https://mlsanity.com/paper/2211.09861/) (2022)
- [CNN-Transformer Rectified Collaborative Learning for Medical Image Segmentation](https://mlsanity.com/paper/2408.13698/) (2024)
- [Sentinel-Guided Zero-Shot Learning: A Collaborative Paradigm without Real Data Exposure](https://mlsanity.com/paper/2403.09363/) (2024)
- [Augmenting Knowledge Distillation With Peer-To-Peer Mutual Learning For Model Compression](https://mlsanity.com/paper/2110.11023/) (2021)
- [Low-Resolution Object Recognition with Cross-Resolution Relational Contrastive Distillation](https://mlsanity.com/paper/2409.02555/) (2024)
- [DiReDi: Distillation and Reverse Distillation for AIoT Applications](https://mlsanity.com/paper/2409.08308/) (2024)
- [Delving Deep into Semantic Relation Distillation](https://mlsanity.com/paper/2503.21269/) (2025)
- [Dual-Forward Path Teacher Knowledge Distillation: Bridging the Capacity Gap Between Teacher and Student](https://mlsanity.com/paper/2506.18244/) (2025)
- [Instance-aware Model Ensemble With Distillation For Unsupervised Domain Adaptation](https://mlsanity.com/paper/2211.08106/) (2022)
- [Inference-Time Rule Eraser: Fair Recognition via Distilling and Removing Biased Rules](https://mlsanity.com/paper/2404.04814/) (2024)
- [Multi-objective hybrid knowledge distillation for efficient deep learning in smart agriculture](https://mlsanity.com/paper/2512.22239/) (2025)
- [Few-shot Class-Incremental Learning via Generative Co-Memory Regularization](https://mlsanity.com/paper/2601.07117/) (2026)
- [Leveraging Different Learning Styles for Improved Knowledge Distillation in Biomedical Imaging](https://mlsanity.com/paper/2212.02931/) (2022)
- [A Multi-objective Complex Network Pruning Framework Based on Divide-and-conquer and Global Performance Impairment Ranking](https://mlsanity.com/paper/2303.16212/) (2023)
- [SRMAE: Masked Image Modeling for Scale-Invariant Deep Representations](https://mlsanity.com/paper/2308.08884/) (2023)
- [Generative Denoise Distillation: Simple Stochastic Noises Induce Efficient Knowledge Transfer for Dense Prediction](https://mlsanity.com/paper/2401.08332/) (2024)
- [Low-Resolution Face Recognition via Adaptable Instance-Relation Distillation](https://mlsanity.com/paper/2409.02049/) (2024)
- [Distilling Generative-Discriminative Representations for Very Low-Resolution Face Recognition](https://mlsanity.com/paper/2409.06371/) (2024)
- [A Lightweight Target-Driven Network of Stereo Matching for Inland Waterways](https://mlsanity.com/paper/2410.07915/) (2024)
- [Preview-based Category Contrastive Learning for Knowledge Distillation](https://mlsanity.com/paper/2410.14143/) (2024)
- [Rotation Perturbation Robustness in Point Cloud Analysis: A Perspective of Manifold Distillation](https://mlsanity.com/paper/2411.01748/) (2024)
- [Contrastive Representation Distillation via Multi-Scale Feature Decoupling](https://mlsanity.com/paper/2502.05835/) (2025)
- [SciceVPR: Stable Cross-Image Correlation Enhanced Model for Visual Place Recognition](https://mlsanity.com/paper/2502.20676/) (2025)

## Abstract (arXiv)

Knowledge distillation provides an effective way to transfer knowledge via teacher-student learning, where most existing distillation approaches apply a fixed pre-trained model as teacher to supervise the learning of student network. This manner usually brings in a big capability gap between teacher and student networks during learning. Recent researches have observed that a small teacher-student capability gap can facilitate knowledge transfer. Inspired by that, we propose an evolutionary knowledge distillation approach to improve the transfer effectiveness of teacher knowledge. Instead of a fixed pre-trained teacher, an evolutionary teacher is learned online and consistently transfers intermediate knowledge to supervise student network learning on-the-fly. To enhance intermediate knowledge representation and mimicking, several simple guided modules are introduced between corresponding teacher-student blocks. In this way, the student can simultaneously obtain rich internal knowledge and capture its growth process, leading to effective student network learning. Extensive experiments clearly demonstrate the effectiveness of our approach as well as good adaptability in the low-resolution and few-sample visual recognition scenarios.