# Ensemble Knowledge Distillation for Learning Improved and Efficient Networks

arXiv: https://arxiv.org/abs/1909.08097 · HTML: https://mlsanity.com/paper/1909.08097/

## Key facts

- Cited by 46 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 17 Sep 2019.

## Built on by (most cited first, 30 of 46)

- [AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One](https://mlsanity.com/paper/2312.06709/) (2023)
- [A Comprehensive Survey on Hardware-Aware Neural Architecture Search](https://mlsanity.com/paper/2101.09336/) (2021)
- [Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillation](https://mlsanity.com/paper/2209.04599/) (2022)
- [Multimodal Distillation for Egocentric Action Recognition](https://mlsanity.com/paper/2307.07483/) (2023)
- [UNIC: Universal Classification Models via Multi-teacher Distillation](https://mlsanity.com/paper/2408.05088/) (2024)
- [PHI-S: Distribution Balancing for Label-Free Multi-Teacher Distillation](https://mlsanity.com/paper/2410.01680/) (2024)
- [CDFKD-MFS: Collaborative Data-free Knowledge Distillation via Multi-level Feature Sharing](https://mlsanity.com/paper/2205.11845/) (2022)
- [A predictive machine learning force field framework for liquid electrolyte development](https://mlsanity.com/paper/2404.07181/) (2024)
- [Enhancing Heterogeneous Federated Learning with Knowledge Extraction and Multi-Model Fusion](https://mlsanity.com/paper/2208.07978/) (2022)
- [FusionBench: A Unified Library and Comprehensive Benchmark for Deep Model Fusion](https://mlsanity.com/paper/2406.03280/) (2024)
- [A Comprehensive Survey on Imbalanced Data Learning](https://mlsanity.com/paper/2502.08960/) (2025)
- [ELODI: Ensemble Logit Difference Inhibition for Positive-Congruent Training](https://mlsanity.com/paper/2205.06265/) (2022)
- [Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation](https://mlsanity.com/paper/2305.13067/) (2023)
- [Federated Learning with Privacy-Preserving Ensemble Attention Distillation](https://mlsanity.com/paper/2210.08464/) (2022)
- [DeNetDM: Debiasing by Network Depth Modulation](https://mlsanity.com/paper/2403.19863/) (2024)
- [Students are the Best Teacher: Exit-Ensemble Distillation with Multi-Exits](https://mlsanity.com/paper/2104.00299/) (2021)
- [Competitive Distillation: A Simple Learning Strategy for Improving Visual Classification](https://mlsanity.com/paper/2506.23285/) (2025)
- [Label driven Knowledge Distillation for Federated Learning with non-IID Data](https://mlsanity.com/paper/2209.14520/) (2022)
- [Reduce, Reuse, Recycle: Improving Training Efficiency with Distillation](https://mlsanity.com/paper/2211.00683/) (2022)
- [Knowledge Distillation for Efficient Sequences of Training Runs](https://mlsanity.com/paper/2303.06480/) (2023)
- [Why does Knowledge Distillation Work? Rethink its Attention and Fidelity Mechanism](https://mlsanity.com/paper/2405.00739/) (2024)
- [Mitigating Negative Flips via Margin Preserving Training](https://mlsanity.com/paper/2511.08322/) (2025)
- [Embedded Knowledge Distillation in Depth-Level Dynamic Neural Network](https://mlsanity.com/paper/2103.00793/) (2021)
- [FSNet: Redesign Self-Supervised MonoDepth for Full-Scale Depth Prediction for Autonomous Driving](https://mlsanity.com/paper/2304.10719/) (2023)
- [MIND: Modality-Informed Knowledge Distillation Framework for Multimodal Clinical Prediction Tasks](https://mlsanity.com/paper/2502.01158/) (2025)
- [Corrected with the Latest Version: Make Robust Asynchronous Federated Learning Possible](https://mlsanity.com/paper/2504.04081/) (2025)
- [Knowledge Distillation via Weighted Ensemble of Teaching Assistants](https://mlsanity.com/paper/2206.12005/) (2022)
- [Instance-aware Model Ensemble With Distillation For Unsupervised Domain Adaptation](https://mlsanity.com/paper/2211.08106/) (2022)
- [Knowledge Distillation on Spatial-Temporal Graph Convolutional Network for Traffic Prediction](https://mlsanity.com/paper/2401.11798/) (2024)
- [Wisdom of Committee: Diverse Distillation from Large Foundation Models and Domain Experts](https://mlsanity.com/paper/2402.14035/) (2024)

## Abstract (arXiv)

Ensemble models comprising of deep Convolutional Neural Networks (CNN) have shown significant improvements in model generalization but at the cost of large computation and memory requirements. In this paper, we present a framework for learning compact CNN models with improved classification performance and model generalization. For this, we propose a CNN architecture of a compact student model with parallel branches which are trained using ground truth labels and information from high capacity teacher networks in an ensemble learning fashion. Our framework provides two main benefits: i) Distilling knowledge from different teachers into the student network promotes heterogeneity in feature learning at different branches of the student network and enables the network to learn diverse solutions to the target problem. ii) Coupling the branches of the student network through ensembling encourages collaboration and improves the quality of the final predictions by reducing variance in the network outputs. Experiments on the well established CIFAR-10 and CIFAR-100 datasets show that our Ensemble Knowledge Distillation (EKD) improves classification accuracy and model generalization especially in situations with limited training data. Experiments also show that our EKD based compact networks outperform in terms of mean accuracy on the test datasets compared to state-of-the-art knowledge distillation based methods.