# Deep Model Compression: Distilling Knowledge from Noisy Teachers

arXiv: https://arxiv.org/abs/1610.09650 · HTML: https://mlsanity.com/paper/1610.09650/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2016.

## Built on by (most cited first, 30 of 36)

- [Adaptive Multi-Teacher Multi-level Knowledge Distillation](https://mlsanity.com/paper/2103.04062/) (2021)
- [Nearly Optimal VC-Dimension and Pseudo-Dimension Bounds for Deep Neural Network Derivatives](https://mlsanity.com/paper/2305.08466/) (2023)
- [Embracing the Dark Knowledge: Domain Generalization Using Regularized Knowledge Distillation](https://mlsanity.com/paper/2107.02629/) (2021)
- [DiGA: Distil to Generalize and then Adapt for Domain Adaptive Semantic Segmentation](https://mlsanity.com/paper/2304.02222/) (2023)
- [Distillation of Diffusion Features for Semantic Correspondence](https://mlsanity.com/paper/2412.03512/) (2024)
- [Few-Shot Learning with a Strong Teacher](https://mlsanity.com/paper/2107.00197/) (2021)
- [Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss](https://mlsanity.com/paper/2402.00152/) (2024)
- [Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding](https://mlsanity.com/paper/2503.16707/) (2025)
- [Generalized Knowledge Distillation via Relationship Matching](https://mlsanity.com/paper/2205.01915/) (2022)
- [LANA: Latency Aware Network Acceleration](https://mlsanity.com/paper/2107.10624/) (2021)
- [Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation](https://mlsanity.com/paper/2308.09105/) (2023)
- [Knowledge Distillation from Multiple Foundation Models for End-to-End Speech Recognition](https://mlsanity.com/paper/2303.10917/) (2023)
- [Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning](https://mlsanity.com/paper/2410.14633/) (2024)
- [Estimating and Maximizing Mutual Information for Knowledge Distillation](https://mlsanity.com/paper/2110.15946/) (2021)
- [Tailor: Altering Skip Connections for Resource-Efficient Inference](https://mlsanity.com/paper/2301.07247/) (2023)
- [DSD2: Can We Dodge Sparse Double Descent and Compress the Neural Network Worry-Free?](https://mlsanity.com/paper/2303.01213/) (2023)
- [Distilling the Knowledge from Conditional Normalizing Flows](https://mlsanity.com/paper/2106.12699/) (2021)
- [Optimal Deep Neural Network Approximation for Korobov Functions with respect to Sobolev Norms](https://mlsanity.com/paper/2311.04779/) (2023)
- [Foldable SuperNets: Scalable Merging of Transformers with Different Initializations and Tasks](https://mlsanity.com/paper/2410.01483/) (2024)
- [AdaKD: Dynamic Knowledge Distillation of ASR models using Adaptive Loss Weighting](https://mlsanity.com/paper/2405.08019/) (2024)
- [Deadwooding: Robust Global Pruning for Deep Neural Networks](https://mlsanity.com/paper/2202.05226/) (2022)
- [Robust Knowledge Distillation from RNN-T Models With Noisy Training Labels Using Full-Sum Loss](https://mlsanity.com/paper/2303.05958/) (2023)
- [Knowledge Distillation for Anomaly Detection](https://mlsanity.com/paper/2310.06047/) (2023)
- [Contrastive Neighborhood Alignment](https://mlsanity.com/paper/2201.01922/) (2022)
- [Dynamic Rectification Knowledge Distillation](https://mlsanity.com/paper/2201.11319/) (2022)
- [Selective Cross-Task Distillation](https://mlsanity.com/paper/2204.11526/) (2022)
- [A Simple Structure For Building A Robust Model](https://mlsanity.com/paper/2204.11596/) (2022)
- [Distilling Knowledge via Intermediate Classifiers](https://mlsanity.com/paper/2103.00497/) (2021)
- [A Studious Approach to Semi-Supervised Learning](https://mlsanity.com/paper/2109.08924/) (2021)
- [Multi-Modality Distillation via Learning the teacher's modality-level Gram Matrix](https://mlsanity.com/paper/2112.11447/) (2021)

## Abstract (arXiv)

The remarkable successes of deep learning models across various applications have resulted in the design of deeper networks that can solve complex problems. However, the increasing depth of such models also results in a higher storage and runtime complexity, which restricts the deployability of such very deep models on mobile and portable devices, which have limited storage and battery capacity. While many methods have been proposed for deep model compression in recent years, almost all of them have focused on reducing storage complexity. In this work, we extend the teacher-student framework for deep model compression, since it has the potential to address runtime and train time complexity too. We propose a simple methodology to include a noise-based regularizer while training the student from the teacher, which provides a healthy improvement in the performance of the student network. Our experiments on the CIFAR-10, SVHN and MNIST datasets show promising improvement, with the best performance on the CIFAR-10 dataset. We also conduct a comprehensive empirical evaluation of the proposed method under related settings on the CIFAR-10 dataset to show the promise of the proposed approach.