# Ensemble deep learning: A review

arXiv: https://arxiv.org/abs/2104.02395 · HTML: https://mlsanity.com/paper/2104.02395/

## Key facts

- Cited by 284 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 249 references that are in the map.
- First posted on arXiv 6 Apr 2021.

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [DeepWalk: Online Learning of Social Representations](https://mlsanity.com/paper/1403.6652/) (2014)
- [Temporal Ensembling for Semi-Supervised Learning](https://mlsanity.com/paper/1610.02242/) (2016)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [Snapshot Ensembles: Train 1, get M for free](https://mlsanity.com/paper/1704.00109/) (2017)
- [The Loss Surfaces of Multilayer Networks](https://mlsanity.com/paper/1412.0233/) (2014)
- [Deep Stacked Hierarchical Multi-patch Network for Image Deblurring](https://mlsanity.com/paper/1904.03468/) (2019)
- [Training Very Deep Networks](https://mlsanity.com/paper/1507.06228/) (2015)
- [AdaNet: Adaptive Structural Learning of Artificial Neural Networks](https://mlsanity.com/paper/1607.01097/) (2016)
- [The Relative Performance of Ensemble Methods with Deep Convolutional Neural Networks for Image Classification](https://mlsanity.com/paper/1704.01664/) (2017)
- [Deep Forest](https://mlsanity.com/paper/1702.08835/) (2017)
- [Learning Deep ResNet Blocks Sequentially using Boosting Theory](https://mlsanity.com/paper/1706.04964/) (2017)
- [Horizontal and Vertical Ensemble with Deep Representation for Classification](https://mlsanity.com/paper/1306.2759/) (2013)
- [Locally Weighted Ensemble Clustering](https://mlsanity.com/paper/1605.05011/) (2016)
- [Swapout: Learning an ensemble of deep architectures](https://mlsanity.com/paper/1605.06465/) (2016)
- [Generalized Negative Correlation Learning for Deep Ensembling](https://mlsanity.com/paper/2011.02952/) (2020)
- [Incremental Boosting Convolutional Neural Network for Facial Action Unit Recognition](https://mlsanity.com/paper/1707.05395/) (2017)
- [Online Gradient Boosting](https://mlsanity.com/paper/1506.04820/) (2015)
- [A Deep Learning Approach to Unsupervised Ensemble Learning](https://mlsanity.com/paper/1602.02285/) (2016)
- [Ensemble-Based Deep Reinforcement Learning for Chatbots](https://mlsanity.com/paper/1908.10422/) (2019)
- [EnAET: A Self-Trained framework for Semi-Supervised and Supervised Learning with Ensemble Transformations](https://mlsanity.com/paper/1911.09265/) (2019)
- [Gradual DropIn of Layers to Train Very Deep Neural Networks](https://mlsanity.com/paper/1511.06951/) (2015)
- [Particle Swarm Optimisation for Evolving Deep Neural Networks for Image Classification by Evolving and Stacking Transferable Blocks](https://mlsanity.com/paper/1907.12659/) (2019)
- [Online Learning for Offloading and Autoscaling in Energy Harvesting Mobile Edge Computing](https://mlsanity.com/paper/1703.06060/) (2017)
- [Deep Incremental Boosting](https://mlsanity.com/paper/1708.03704/) (2017)

## Built on by (most cited first, 30 of 284)

- [SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification](https://mlsanity.com/paper/2305.09781/) (2023)
- [A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future](https://mlsanity.com/paper/2504.12328/) (2025)
- [Fusing Models with Complementary Expertise](https://mlsanity.com/paper/2310.01542/) (2023)
- [A review of uncertainty quantification in medical image analysis: probabilistic and non-probabilistic methods](https://mlsanity.com/paper/2310.06873/) (2023)
- [Boosted Prompt Ensembles for Large Language Models](https://mlsanity.com/paper/2304.05970/) (2023)
- [Forecast combinations: an over 50-year review](https://mlsanity.com/paper/2205.04216/) (2022)
- [Random vector functional link network: recent developments, applications, and future directions](https://mlsanity.com/paper/2203.11316/) (2022)
- [LidarCLIP or: How I Learned to Talk to Point Clouds](https://mlsanity.com/paper/2212.06858/) (2022)
- [TIP: Tabular-Image Pre-training for Multimodal Classification with Incomplete Data](https://mlsanity.com/paper/2407.07582/) (2024)
- [Co-Supervised Learning: Improving Weak-to-Strong Generalization with Hierarchical Mixture of Experts](https://mlsanity.com/paper/2402.15505/) (2024)
- [Assessing generalisability of deep learning-based polyp detection and segmentation methods through a computer vision challenge](https://mlsanity.com/paper/2202.12031/) (2022)
- [A review of ensemble learning and data augmentation models for class imbalanced problems: combination, implementation and evaluation](https://mlsanity.com/paper/2304.02858/) (2023)
- [Attention Prompting on Image for Large Vision-Language Models](https://mlsanity.com/paper/2409.17143/) (2024)
- [Long-Tail Prediction Uncertainty Aware Trajectory Planning for Self-driving Vehicles](https://mlsanity.com/paper/2207.00788/) (2022)
- [Towards long-tailed, multi-label disease classification from chest X-ray: Overview of the CXR-LT challenge](https://mlsanity.com/paper/2310.16112/) (2023)
- [Dipper: Diversity in Prompts for Producing Large Language Model Ensembles in Reasoning tasks](https://mlsanity.com/paper/2412.15238/) (2024)
- [Sparse Model Soups: A Recipe for Improved Pruning via Model Averaging](https://mlsanity.com/paper/2306.16788/) (2023)
- [STAP: Sequencing Task-Agnostic Policies](https://mlsanity.com/paper/2210.12250/) (2022)
- [Emergent Equivariance in Deep Ensembles](https://mlsanity.com/paper/2403.03103/) (2024)
- [Comprehensive Review On Twin Support Vector Machines](https://mlsanity.com/paper/2105.00336/) (2021)
- [Ensemble Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.02618/) (2023)
- [MixedNUTS: Training-Free Accuracy-Robustness Balance via Nonlinearly Mixed Classifiers](https://mlsanity.com/paper/2402.02263/) (2024)
- [Estimating Canopy Height at Scale](https://mlsanity.com/paper/2406.01076/) (2024)
- [PLeaS -- Merging Models with Permutations and Least Squares](https://mlsanity.com/paper/2407.02447/) (2024)
- [Learning useful representations for shifting tasks and distributions](https://mlsanity.com/paper/2212.07346/) (2022)
- [Self-Ensembling Gaussian Splatting for Few-Shot Novel View Synthesis](https://mlsanity.com/paper/2411.00144/) (2024)
- [Deep Learning for Brain Age Estimation: A Systematic Review](https://mlsanity.com/paper/2212.03868/) (2022)
- [FREE-Merging: Fourier Transform for Efficient Model Merging](https://mlsanity.com/paper/2411.16815/) (2024)
- [PSO-PINN: Physics-Informed Neural Networks Trained with Particle Swarm Optimization](https://mlsanity.com/paper/2202.01943/) (2022)
- [Machine Learning and Deep Learning -- A review for Ecologists](https://mlsanity.com/paper/2204.05023/) (2022)

## Abstract (arXiv)

Ensemble learning combines several individual models to obtain better generalization performance. Currently, deep learning architectures are showing better performance compared to the shallow or traditional models. Deep ensemble learning models combine the advantages of both the deep learning models as well as the ensemble learning such that the final model has better generalization performance. This paper reviews the state-of-art deep ensemble models and hence serves as an extensive summary for the researchers. The ensemble models are broadly categorised into bagging, boosting, stacking, negative correlation based deep ensemble models, explicit/implicit ensembles, homogeneous/heterogeneous ensemble, decision fusion strategies based deep ensemble models. Applications of deep ensemble models in different domains are also briefly discussed. Finally, we conclude this paper with some potential future research directions.