# Efficient Diversity-Driven Ensemble for Deep Neural Networks

arXiv: https://arxiv.org/abs/2112.13316 · HTML: https://mlsanity.com/paper/2112.13316/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 44 references that are in the map.
- First posted on arXiv 26 Dec 2021.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [Snapshot Ensembles: Train 1, get M for free](https://mlsanity.com/paper/1704.00109/) (2017)
- [Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks](https://mlsanity.com/paper/1511.06314/) (2015)
- [The Relative Performance of Ensemble Methods with Deep Convolutional Neural Networks for Image Classification](https://mlsanity.com/paper/1704.01664/) (2017)
- [Deep Incremental Boosting](https://mlsanity.com/paper/1708.03704/) (2017)
- [Deep Super Learner: A Deep Ensemble for Classification Problems](https://mlsanity.com/paper/1803.02323/) (2018)

## Built on by (most cited first, 11 of 11)

- [Similarity of Neural Network Models: A Survey of Functional and Representational Measures](https://mlsanity.com/paper/2305.06329/) (2023)
- [LightTS: Lightweight Time Series Classification with Adaptive Ensemble Distillation -- Extended Version](https://mlsanity.com/paper/2302.12721/) (2023)
- [DivBO: Diversity-aware CASH for Ensemble Learning](https://mlsanity.com/paper/2302.03255/) (2023)
- [Representational Similarity via Interpretable Visual Concepts](https://mlsanity.com/paper/2503.15699/) (2025)
- [Deep Combinatorial Aggregation](https://mlsanity.com/paper/2210.06436/) (2022)
- [Tree-Structured Synergy of Large Language Models and Bayesian Optimization for Efficient CASH](https://mlsanity.com/paper/2601.12355/) (2026)
- [Transfer learning for ensembles: reducing computation time and keeping the diversity](https://mlsanity.com/paper/2206.13116/) (2022)
- [Deep Negative Correlation Classification](https://mlsanity.com/paper/2212.07070/) (2022)
- [Boosting Deep Ensembles with Learning Rate Tuning](https://mlsanity.com/paper/2410.07564/) (2024)
- [A Two-Phase Recall-and-Select Framework for Fast Model Selection](https://mlsanity.com/paper/2404.00069/) (2024)
- [Representational Difference Explanations](https://mlsanity.com/paper/2505.23917/) (2025)

## Abstract (arXiv)

The ensemble of deep neural networks has been shown, both theoretically and empirically, to improve generalization accuracy on the unseen test set. However, the high training cost hinders its efficiency since we need a sufficient number of base models and each one in the ensemble has to be separately trained. Lots of methods are proposed to tackle this problem, and most of them are based on the feature that a pre-trained network can transfer its knowledge to the next base model and then accelerate the training process. However, these methods suffer a severe problem that all of them transfer knowledge without selection and thus lead to low diversity. As the effect of ensemble learning is more pronounced if ensemble members are accurate and diverse, we propose a method named Efficient Diversity-Driven Ensemble (EDDE) to address both the diversity and the efficiency of an ensemble. To accelerate the training process, we propose a novel knowledge transfer method which can selectively transfer the previous generic knowledge. To enhance diversity, we first propose a new diversity measure, then use it to define a diversity-driven loss function for optimization. At last, we adopt a Boosting-based framework to combine the above operations, such a method can also further improve diversity. We evaluate EDDE on Computer Vision (CV) and Natural Language Processing (NLP) tasks. Compared with other well-known ensemble methods, EDDE can get highest ensemble accuracy with the lowest training cost, which means it is efficient in the ensemble of neural networks.