mlsanityData to 14 Oct 2026

Teacher-Student Architecture for Knowledge Distillation: A Survey

arXiv 2308.04268 ↗First posted 8 Aug 2023cs.LG cs.AIcited by 26

On the map: Knowledge distillation → General knowledge distillation methods

Abstract (arXiv)

Although Deep neural networks (DNNs) have shown a strong capacity to solve large-scale problems in many areas, such DNNs are hard to be deployed in real-world systems due to their voluminous parameters. To tackle this issue, Teacher-Student architectures were proposed, where simple student networks with a few parameters can achieve comparable performance to deep teacher networks with many parameters. Recently, Teacher-Student architectures have been effectively and widely embraced on various knowledge distillation (KD) objectives, including knowledge compression, knowledge expansion, knowledge adaptation, and knowledge enhancement. With the help of Teacher-Student architectures, current studies are able to achieve multiple distillation objectives through lightweight and generalized student networks. Different from existing KD surveys that primarily focus on knowledge compression, this survey first explores Teacher-Student architectures across multiple distillation objectives. This survey presents an introduction to various knowledge representations and their corresponding optimization objectives. Additionally, we provide a systematic overview of Teacher-Student architectures with representative learning algorithms and effective distillation schemes. This survey also summarizes recent applications of Teacher-Student architectures across multiple purposes, including classification, recognition, generation, ranking, and regression. Lastly, potential research directions in KD are investigated, focusing on architecture design, knowledge quality, and theoretical studies of regression-based learning, respectively. Through this comprehensive survey, industry practitioners and the academic community can gain valuable insights and guidelines for effectively designing, learning, and applying Teacher-Student architectures on various distillation objectives.

Key facts. Cited by 26 papers in the map (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 145 of its 197 references that are in the map. First posted on arXiv 8 Aug 2023.

Builds on

145 of 197 references.

Closest ancestors · 10

Same direction, newest first

  1. 2022Knowledge Distillation from A Stronger Teacher21 May
  2. 2022Decoupled Knowledge Distillation16 Mar
  3. 2021Information Theoretic Representation Distillation1 Dec
  4. 2021Estimating and Maximizing Mutual Information for Knowledge Distillation29 Oct
  5. 2021Does Knowledge Distillation Really Work?10 Jun
  6. 2021Knowledge distillation: A good teacher is patient and consistent9 Jun
  7. 2021BERT Learns to Teach: Knowledge Distillation with Meta Learning8 Jun
  8. 2021Complementary Relation Contrastive Distillation29 Mar
  9. 2021Adaptive Multi-Teacher Multi-level Knowledge Distillation6 Mar
  10. 2021Learning Student-Friendly Teacher Networks for Knowledge Distillation12 Feb

Other references · 135

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  9. 2016Deep Learning with Differential Privacycited by 2,728
  10. 2016Learning without Forgettingcited by 2,496
  11. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  12. 2020Knowledge Distillation: A Surveycited by 1,603
38 more
  1. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  2. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  3. 2017CyCADA: Cycle-Consistent Adversarial Domain Adaptationcited by 1,082
  4. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,077
  5. 2019Relational Knowledge Distillationcited by 889
  6. 2016Sequence-Level Knowledge Distillationcited by 801
  7. 2019Contrastive Representation Distillationcited by 748
  8. 2020Self-supervised Learning: Generative or Contrastivecited by 694
  9. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  10. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 678
  11. 2020Image Segmentation Using Deep Learning: A Surveycited by 660
  12. 2017Deep Mutual Learningcited by 628
  13. 2018Learning to Adapt Structured Output Space for Semantic Segmentationcited by 584
  14. 2019Improved Knowledge Distillation via Teacher Assistantcited by 573
  15. 2015Bidirectional LSTM-CRF Models for Sequence Taggingcited by 546
  16. 2018Born Again Neural Networkscited by 546
  17. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 508
  18. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  19. 2017Progressive Neural Architecture Searchcited by 487
  20. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 459
  21. 2016R-FCN: Object Detection via Region-based Fully Convolutional Networkscited by 414
  22. 2014Understanding Deep Image Representations by Inverting Themcited by 403
  23. 2019Uninformed Students: Student-Teacher Anomaly Detection with Discriminative Latent Embeddingscited by 386
  24. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 348
  25. 2019Variational Information Distillation for Knowledge Transfercited by 324
  26. 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 316
  27. 2019Bidirectional Learning for Domain Adaptation of Semantic Segmentationcited by 308
  28. 2019Correlation Congruence for Knowledge Distillationcited by 285
  29. 2020A Simple Semi-Supervised Learning Framework for Object Detectioncited by 282
  30. 2018Paraphrasing Complex Network: Network Compression via Factor Transfercited by 260
  31. 2018Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neuronscited by 238
  32. 2015Bilinear CNNs for Fine-grained Visual Recognitioncited by 228
  33. 2018Learning Deep Representations with Probabilistic Knowledge Transfercited by 213
  34. 2017Teacher-Student Curriculum Learningcited by 210
  35. 2017Incremental Learning of Object Detectors without Catastrophic Forgettingcited by 208
  36. 2017Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learningcited by 192
  37. 2018Knowledge Distillation by On-the-Fly Native Ensemblecited by 186
  38. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176

50 most cited of 135 All 135 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 7; 2025: 12; 2026: 5.

26 papers.

Sort:

By area:

  1. 2024Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Applicationcited by 56
  2. 2023Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Modelscited by 15
  3. 2025TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillationcited by 12
  4. 2025Efficient Switchable Safety Control in LLMs via Magic-Token-Guided Co-Trainingcited by 7
  5. 2024Weak-to-Strong 3D Object Detection with X-Ray Distillationcited by 5
  6. 2024Make a Strong Teacher with Label Assistance: A Novel Knowledge Distillation Approach for Semantic Segmentationcited by 5
  7. 2025Neuroplasticity in Artificial Intelligence -- An Overview and Inspirations on Drop In & Out Learningcited by 3
  8. 2023Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillationcited by 2
  9. 2025Adaptive Temperature Based on Logits Correlation in Knowledge Distillationcited by 2
  10. 2025Fine-grained Image Retrieval via Dual-Vision Adaptationcited by 2
  11. 2024GeoWATCH for Detecting Heavy Construction in Heterogeneous Time Series of Satellite Imagescited by 1
  12. 2024POA: Pre-training Once for Models of All Sizescited by 1
14 more
  1. 2024MobileIQA: Exploiting Mobile-level Diverse Opinion Network For No-Reference Image Quality Assessment Using Knowledge Distillationcited by 1
  2. 2025A Low-Complexity Plug-and-Play Deep Learning Model for Massive MIMO Precoding Across Sitescited by 1
  3. 2025Knowledge Reasoning Language Model: Unifying Knowledge and Language for Inductive Knowledge Graph Reasoningcited by 1
  4. 2026Beyond Student: An Asymmetric Network for Neural Network Inheritancecited by 1
  5. 2026AnyBody: Free-Form Whole-Body Humanoid Control from Arbitrary Keypoint Guidancecited by 1
  6. 2024Online pre-training with long-form videoscited by 0
  7. 2025SciceVPR: Stable Cross-Image Correlation Enhanced Model for Visual Place Recognitioncited by 0
  8. 2025Normative Moral Pluralism for AI: A Framework for Deliberation in Complex Moral Contextscited by 0
  9. 2025Accelerating Reinforcement Learning Algorithms Convergence using Pre-trained Large Language Models as Tutors With Advice Reusingcited by 0
  10. 2025Information-Theoretic Criteria for Knowledge Distillation in Multimodal Learningcited by 0
  11. 2025Osmotic Learning: A Self-Supervised Paradigm for Decentralized Contextual Data Representationcited by 0
  12. 2026KD-MARL: Resource-Aware Knowledge Distillation in Multi-Agent Reinforcement Learningcited by 0
  13. 2026A Multi-Branch Hierarchy-Aware Framework for Heterogeneous Audio Classificationcited by 0
  14. 2026Too much of a good thing -- when knowledge distillation promotes overfitting, and how to avoid itcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Teacher-Student Architecture for Knowledge Learning: A Survey
  2. 2025A Comprehensive Survey on Knowledge Distillation
  3. 2021Fixing the Teacher-Student Knowledge Discrepancy in Distillation
  4. 2023Education distillation:getting student models to learn in shcools
  5. 2023Can a student Large Language Model perform as well as it's teacher?
  6. 2022Collaborative Multi-Teacher Knowledge Distillation for Learning Low Bit-width Deep Neural Networks
  7. 2022Knowledge Distillation with Deep Supervision
  8. 2024Generalizing Teacher Networks for Effective Knowledge Distillation Across Student Architectures