# Conditional Teacher-Student Learning

arXiv: https://arxiv.org/abs/1904.12399 · HTML: https://mlsanity.com/paper/1904.12399/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Apr 2019.

## Built on by (most cited first, 30 of 34)

- [Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognition](https://mlsanity.com/paper/2102.01380/) (2021)
- [Teacher-Student Architecture for Knowledge Distillation: A Survey](https://mlsanity.com/paper/2308.04268/) (2023)
- [Balanced Knowledge Distillation for Long-tailed Learning](https://mlsanity.com/paper/2104.10510/) (2021)
- [Minimum Word Error Rate Training with Language Model Fusion for End-to-End Speech Recognition](https://mlsanity.com/paper/2106.02302/) (2021)
- [Internal Language Model Adaptation with Text-Only Data for End-to-End Speech Recognition](https://mlsanity.com/paper/2110.05354/) (2021)
- [Representation learning through cross-modal conditional teacher-student training for speech emotion recognition](https://mlsanity.com/paper/2112.00158/) (2021)
- [Distillation of Diffusion Features for Semantic Correspondence](https://mlsanity.com/paper/2412.03512/) (2024)
- [Black-box Few-shot Knowledge Distillation](https://mlsanity.com/paper/2207.12106/) (2022)
- [Learning to "Segment Anything" in Thermal Infrared Images through Knowledge Distillation with a Large Scale Dataset SATIR](https://mlsanity.com/paper/2304.07969/) (2023)
- [Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models](https://mlsanity.com/paper/2405.14161/) (2024)
- [Learn From the Past: Experience Ensemble Knowledge Distillation](https://mlsanity.com/paper/2202.12488/) (2022)
- [Federated Co-tuning Framework for Large and Small Language Models](https://mlsanity.com/paper/2411.11707/) (2024)
- [Enabling All In-Edge Deep Learning: A Literature Review](https://mlsanity.com/paper/2204.03326/) (2022)
- [ABC-KD: Attention-Based-Compression Knowledge Distillation for Deep Learning-Based Noise Suppression](https://mlsanity.com/paper/2305.16665/) (2023)
- [Bridging the Gap Between Patient-specific and Patient-independent Seizure Prediction via Knowledge Distillation](https://mlsanity.com/paper/2202.12598/) (2022)
- [Aligning Logits Generatively for Principled Black-Box Knowledge Distillation](https://mlsanity.com/paper/2205.10490/) (2022)
- [InDistill: Information flow-preserving knowledge distillation for model compression](https://mlsanity.com/paper/2205.10003/) (2022)
- [MixBCT: Towards Self-Adapting Backward-Compatible Training](https://mlsanity.com/paper/2308.06948/) (2023)
- [Soft Knowledge Distillation with Multi-Dimensional Cross-Net Attention for Image Restoration Models Compression](https://mlsanity.com/paper/2501.09321/) (2025)
- [The Capabilities and Limitations of Weak-to-Strong Generalization: Generalization and Calibration](https://mlsanity.com/paper/2502.01458/) (2025)
- [Self Regulated Learning Mechanism for Data Efficient Knowledge Distillation](https://mlsanity.com/paper/2102.07125/) (2021)
- [Confidence Conditioned Knowledge Distillation](https://mlsanity.com/paper/2107.06993/) (2021)
- [Practical Knowledge Distillation: Using DNNs to Beat DNNs](https://mlsanity.com/paper/2302.12360/) (2023)
- [ERSAM: Neural Architecture Search For Energy-Efficient and Real-Time Social Ambiance Measurement](https://mlsanity.com/paper/2303.10727/) (2023)
- [Knowledge Distillation on Spatial-Temporal Graph Convolutional Network for Traffic Prediction](https://mlsanity.com/paper/2401.11798/) (2024)
- [Customizing Speech Recognition Model with Large Language Model Feedback](https://mlsanity.com/paper/2506.11091/) (2025)
- [Memory-Efficient Transfer Learning with Fading Side Networks via Masked Dual Path Distillation](https://mlsanity.com/paper/2604.09088/) (2026)
- [A Light-weight Deep Human Activity Recognition Algorithm Using Multi-knowledge Distillation](https://mlsanity.com/paper/2107.07331/) (2021)
- [Sequence-level self-learning with multiple hypotheses](https://mlsanity.com/paper/2112.05826/) (2021)
- [Adaptive Prompt Learning with Distilled Connective Knowledge for Implicit Discourse Relation Recognition](https://mlsanity.com/paper/2309.07561/) (2023)

## Abstract (arXiv)

The teacher-student (T/S) learning has been shown to be effective for a variety of problems such as domain adaptation and model compression. One shortcoming of the T/S learning is that a teacher model, not always perfect, sporadically produces wrong guidance in form of posterior probabilities that misleads the student model towards a suboptimal performance. To overcome this problem, we propose a conditional T/S learning scheme, in which a "smart" student model selectively chooses to learn from either the teacher model or the ground truth labels conditioned on whether the teacher can correctly predict the ground truth. Unlike a naive linear combination of the two knowledge sources, the conditional learning is exclusively engaged with the teacher model when the teacher model's prediction is correct, and otherwise backs off to the ground truth. Thus, the student model is able to learn effectively from the teacher and even potentially surpass the teacher. We examine the proposed learning scheme on two tasks: domain adaptation on CHiME-3 dataset and speaker adaptation on Microsoft short message dictation dataset. The proposed method achieves 9.8% and 12.8% relative word error rate reductions, respectively, over T/S learning for environment adaptation and speaker-independent model for speaker adaptation.