# Deep Neural Network Ensembles

arXiv: https://arxiv.org/abs/1904.05488 · HTML: https://mlsanity.com/paper/1904.05488/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Apr 2019.

## Built on by (most cited first, 10 of 10)

- [Temperature as Uncertainty in Contrastive Learning](https://mlsanity.com/paper/2110.04403/) (2021)
- [Verification-Aided Deep Ensemble Selection](https://mlsanity.com/paper/2202.03898/) (2022)
- [Adaptive Multi-head Contrastive Learning](https://mlsanity.com/paper/2310.05615/) (2023)
- [Coin Flipping Neural Networks](https://mlsanity.com/paper/2206.09182/) (2022)
- [Improving Compositionality of Neural Networks by Decoding Representations to Inputs](https://mlsanity.com/paper/2106.00769/) (2021)
- [FabricNet: A Fiber Recognition Architecture Using Ensemble ConvNets](https://mlsanity.com/paper/2101.05564/) (2021)
- [Open-set Face Recognition with Neural Ensemble, Maximal Entropy Loss and Feature Augmentation](https://mlsanity.com/paper/2308.12371/) (2023)
- [Faster, Lighter, More Accurate: A Deep Learning Ensemble for Content Moderation](https://mlsanity.com/paper/2309.05150/) (2023)
- [The Case for High-Accuracy Classification: Think Small, Think Many!](https://mlsanity.com/paper/2103.10350/) (2021)
- [LeDNet: Localization-enabled Deep Neural Network for Multi-Label Radiography Image Classification](https://mlsanity.com/paper/2407.03931/) (2024)

## Abstract (arXiv)

Current deep neural networks suffer from two problems; first, they are hard to interpret, and second, they suffer from overfitting. There have been many attempts to define interpretability in neural networks, but they typically lack causality or generality. A myriad of regularization techniques have been developed to prevent overfitting, and this has driven deep learning to become the hot topic it is today; however, while most regularization techniques are justified empirically and even intuitively, there is not much underlying theory. This paper argues that to extract the features used in neural networks to make decisions, it's important to look at the paths between clusters existing in the hidden spaces of neural networks. These features are of particular interest because they reflect the true decision making process of the neural network. This analysis is then furthered to present an ensemble algorithm for arbitrary neural networks which has guarantees for test accuracy. Finally, a discussion detailing the aforementioned guarantees is introduced and the implications to neural networks, including an intuitive explanation for all current regularization methods, are presented. The ensemble algorithm has generated state-of-the-art results for Wide-ResNets on CIFAR-10 (top 5 for all models) and has improved test accuracy for all models it has been applied to.