# Online Ensemble Model Compression using Knowledge Distillation

arXiv: https://arxiv.org/abs/2011.07449 · HTML: https://mlsanity.com/paper/2011.07449/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Nov 2020.

## Built on by (most cited first, 16 of 16)

- [Mutual Contrastive Learning for Visual Representation Learning](https://mlsanity.com/paper/2104.12565/) (2021)
- [Monitored Distillation for Positive Congruent Depth Completion](https://mlsanity.com/paper/2203.16034/) (2022)
- [Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation](https://mlsanity.com/paper/2306.10687/) (2023)
- [NCL++: Nested Collaborative Learning for Long-Tailed Visual Recognition](https://mlsanity.com/paper/2306.16709/) (2023)
- [Computation-efficient Deep Learning for Computer Vision: A Survey](https://mlsanity.com/paper/2308.13998/) (2023)
- [Embedded Self-Distillation in Compact Multi-Branch Ensemble Network for Remote Sensing Scene Classification](https://mlsanity.com/paper/2104.00222/) (2021)
- [Partial Network Cloning](https://mlsanity.com/paper/2303.10597/) (2023)
- [Unified and Effective Ensemble Knowledge Distillation](https://mlsanity.com/paper/2204.00548/) (2022)
- [Collaborative Teacher-Student Learning via Multiple Knowledge Transfer](https://mlsanity.com/paper/2101.08471/) (2021)
- [HDKD: Hybrid Data-Efficient Knowledge Distillation Network for Medical Image Classification](https://mlsanity.com/paper/2407.07516/) (2024)
- [Embedded Knowledge Distillation in Depth-Level Dynamic Neural Network](https://mlsanity.com/paper/2103.00793/) (2021)
- [Rotation Invariant Quantization for Model Compression](https://mlsanity.com/paper/2303.03106/) (2023)
- [Edge AI for Internet of Energy: Challenges and Perspectives](https://mlsanity.com/paper/2311.16851/) (2023)
- [Student Helping Teacher: Teacher Evolution via Self-Knowledge Distillation](https://mlsanity.com/paper/2110.00329/) (2021)
- [The Road to On-board Change Detection: A Lightweight Patch-Level Change Detection Network via Exploring the Potential of Pruning and Pooling](https://mlsanity.com/paper/2310.10166/) (2023)
- [HSI Image Enhancement Classification Based on Knowledge Distillation: A Study on Forgetting](https://mlsanity.com/paper/2603.20292/) (2026)

## Abstract (arXiv)

This paper presents a novel knowledge distillation based model compression framework consisting of a student ensemble. It enables distillation of simultaneously learnt ensemble knowledge onto each of the compressed student models. Each model learns unique representations from the data distribution due to its distinct architecture. This helps the ensemble generalize better by combining every model's knowledge. The distilled students and ensemble teacher are trained simultaneously without requiring any pretrained weights. Moreover, our proposed method can deliver multi-compressed students with single training, which is efficient and flexible for different scenarios. We provide comprehensive experiments using state-of-the-art classification models to validate our framework's effectiveness. Notably, using our framework a 97% compressed ResNet110 student model managed to produce a 10.64% relative accuracy gain over its individual baseline training on CIFAR100 dataset. Similarly a 95% compressed DenseNet-BC(k=12) model managed a 8.17% relative accuracy gain.