# Teacher-Student Architecture for Knowledge Learning: A Survey

arXiv: https://arxiv.org/abs/2210.17332 · HTML: https://mlsanity.com/paper/2210.17332/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation.
- Builds on 76 of its 98 references that are in the map.
- First posted on arXiv 28 Oct 2022.

## Placed in

- Knowledge distillation / [General knowledge distillation methods](https://mlsanity.com/distill/general-knowledge-distillation-methods/)

## Builds on (most cited first, 30 of 76)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [CyCADA: Cycle-Consistent Adversarial Domain Adaptation](https://mlsanity.com/paper/1711.03213/) (2017)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Relational Knowledge Distillation](https://mlsanity.com/paper/1904.05068/) (2019)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks](https://mlsanity.com/paper/1703.10593/) (2017)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [Deep Mutual Learning](https://mlsanity.com/paper/1706.00384/) (2017)
- [Learning to Adapt Structured Output Space for Semantic Segmentation](https://mlsanity.com/paper/1802.10349/) (2018)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Patient Knowledge Distillation for BERT Model Compression](https://mlsanity.com/paper/1908.09355/) (2019)
- [Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation](https://mlsanity.com/paper/1905.08094/) (2019)
- [Progressive Neural Architecture Search](https://mlsanity.com/paper/1712.00559/) (2017)
- [Uninformed Students: Student-Teacher Anomaly Detection with Discriminative Latent Embeddings](https://mlsanity.com/paper/1911.02357/) (2019)
- [Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data](https://mlsanity.com/paper/1610.05755/) (2016)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [Bidirectional Learning for Domain Adaptation of Semantic Segmentation](https://mlsanity.com/paper/1904.10620/) (2019)
- [Correlation Congruence for Knowledge Distillation](https://mlsanity.com/paper/1904.01802/) (2019)
- [A Simple Semi-Supervised Learning Framework for Object Detection](https://mlsanity.com/paper/2005.04757/) (2020)

## Built on by (most cited first, 23 of 23)

- [Leveraging LLM-Respondents for Item Evaluation: a Psychometric Analysis](https://mlsanity.com/paper/2407.10899/) (2024)
- [Data Augmentation using Large Language Models: Data Perspectives, Learning Paradigms and Challenges](https://mlsanity.com/paper/2403.02990/) (2024)
- [Understanding the Gains from Repeated Self-Distillation](https://mlsanity.com/paper/2407.04600/) (2024)
- [Accelerating Molecular Graph Neural Networks via Knowledge Distillation](https://mlsanity.com/paper/2306.14818/) (2023)
- [Low-resource finetuning of foundation models beats state-of-the-art in histopathology](https://mlsanity.com/paper/2401.04720/) (2024)
- [Unsupervised Text Style Transfer via LLMs and Attention Masking with Multi-way Interactions](https://mlsanity.com/paper/2402.13647/) (2024)
- [Retinal OCT Synthesis with Denoising Diffusion Probabilistic Models for Layer Segmentation](https://mlsanity.com/paper/2311.05479/) (2023)
- [Neural Network Pruning by Gradient Descent](https://mlsanity.com/paper/2311.12526/) (2023)
- [FLINT: Learning-based Flow Estimation and Temporal Interpolation for Scientific Ensemble Visualization](https://mlsanity.com/paper/2409.19178/) (2024)
- [Filter Pruning For CNN With Enhanced Linear Representation Redundancy](https://mlsanity.com/paper/2310.06344/) (2023)
- [SSF3D: Strict Semi-Supervised 3D Object Detection with Switching Filter](https://mlsanity.com/paper/2403.17390/) (2024)
- [Auxiliary Input in Training: Incorporating Catheter Features into Deep Learning Models for ECG-Free Dynamic Coronary Roadmapping](https://mlsanity.com/paper/2408.15947/) (2024)
- [Augmenting Offline RL with Unlabeled Data](https://mlsanity.com/paper/2406.07117/) (2024)
- [Development and Validation of Fully Automatic Deep Learning-Based Algorithms for Immunohistochemistry Reporting of Invasive Breast Ductal Carcinoma](https://mlsanity.com/paper/2406.10893/) (2024)
- [Adaptive Guidance for Local Training in Heterogeneous Federated Learning](https://mlsanity.com/paper/2410.06490/) (2024)
- [MT-NAM: An Efficient and Adaptive Model for Epileptic Seizure Detection](https://mlsanity.com/paper/2503.08251/) (2025)
- [Weather-Aware Object Detection Transformer for Domain Adaptation](https://mlsanity.com/paper/2504.10877/) (2025)
- [When Small Guides Large: Cross-Model Co-Learning for Test-Time Adaptation](https://mlsanity.com/paper/2506.23724/) (2025)
- [CORE-ReID V2: Advancing the Domain Adaptation for Object Re-Identification with Optimized Training and Ensemble Fusion](https://mlsanity.com/paper/2508.04036/) (2025)
- [POEM: Explore Unexplored Reliable Samples to Enhance Test-Time Adaptation](https://mlsanity.com/paper/2510.03258/) (2025)
- [Defense against Unauthorized Distillation in Image Restoration via Feature Space Perturbation](https://mlsanity.com/paper/2510.08925/) (2025)
- [SAMora: Enhancing SAM through Hierarchical Self-Supervised Pre-Training for Medical Images](https://mlsanity.com/paper/2511.08626/) (2025)
- [Single State Update Predictive Coding training for Time Series Forecasting and Anomaly Detection](https://mlsanity.com/paper/2608.24697/) (2026)

## Abstract (arXiv)

Although Deep Neural Networks (DNNs) have shown a strong capacity to solve large-scale problems in many areas, such DNNs with voluminous parameters are hard to be deployed in a real-time system. To tackle this issue, Teacher-Student architectures were first utilized in knowledge distillation, where simple student networks can achieve comparable performance to deep teacher networks. Recently, Teacher-Student architectures have been effectively and widely embraced on various knowledge learning objectives, including knowledge distillation, knowledge expansion, knowledge adaption, and multi-task learning. With the help of Teacher-Student architectures, current studies are able to achieve multiple knowledge-learning objectives through lightweight and effective student networks. Different from the existing knowledge distillation surveys, this survey detailedly discusses Teacher-Student architectures with multiple knowledge learning objectives. In addition, we systematically introduce the knowledge construction and optimization process during the knowledge learning and then analyze various Teacher-Student architectures and effective learning schemes that have been leveraged to learn representative and robust knowledge. This paper also summarizes the latest applications of Teacher-Student architectures based on different purposes (i.e., classification, recognition, and generation). Finally, the potential research directions of knowledge learning are investigated on the Teacher-Student architecture design, the quality of knowledge, and the theoretical studies of regression-based learning, respectively. With this comprehensive survey, both industry practitioners and the academic community can learn insightful guidelines about Teacher-Student architectures on multiple knowledge learning objectives.