mlsanityData to 7 Oct 2026

Diversity and Generalization in Neural Network Ensembles

arXiv 2110.13786 ↗First posted 26 Oct 2021cs.LG math.ST stat.THcited by 32

Not on the map. Why?

Abstract (arXiv)

Ensembles are widely used in machine learning and, usually, provide state-of-the-art performance in many prediction tasks. From the very beginning, the diversity of an ensemble has been identified as a key factor for the superior performance of these models. But the exact role that diversity plays in ensemble models is poorly understood, specially in the context of neural networks. In this work, we combine and expand previously published results in a theoretically sound framework that describes the relationship between diversity and ensemble performance for a wide range of ensemble methods. More precisely, we provide sound answers to the following questions: how to measure diversity, how diversity relates to the generalization error of an ensemble, and how diversity is promoted by neural network ensemble algorithms. This analysis covers three widely used loss functions, namely, the squared loss, the cross-entropy loss, and the 0-1 loss; and two widely used model combination strategies, namely, model averaging and weighted majority vote. We empirically validate this theoretical analysis with neural network ensembles.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 18 of its 58 references that are in the map. First posted on arXiv 26 Oct 2021.

Builds on

18 of 58 references.

References · 18

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  3. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,160
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  5. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  6. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  7. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
  8. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  9. 2019Improving Adversarial Robustness via Promoting Ensemble Diversitycited by 213
  10. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  11. 2020Hyperparameter Ensembles for Robustness and Uncertainty Quantificationcited by 149
  12. 2016PAC-Bayesian Theory Meets Bayesian Inferencecited by 87
6 more
  1. 2016Stochastic Multiple Choice Learning for Training Diverse Deep Ensemblescited by 77
  2. 2015Risk Bounds for the Majority Vote: From a PAC-Bayesian Analysis to a Learning Algorithmcited by 50
  3. 2020Second Order PAC-Bayesian Bounds for the Weighted Majority Votecited by 34
  4. 2019Maximizing Overall Diversity for Improved Uncertainty Estimates in Deep Ensemblescited by 30
  5. 2020Generalized Negative Correlation Learning for Deep Ensemblingcited by 16
  6. 2014Kaggle LSHTC4 Winning Solutioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 12; 2024: 9; 2025: 6; 2026: 3.

32 papers.

Sort:

By area:

  1. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  2. 2023A Unified Theory of Diversity in Ensemble Learningcited by 71
  3. 2022Deep Ensembles Work, But Are They Necessary?cited by 63
  4. 2023Joint Training of Deep Ensembles Fails Due to Learner Collusioncited by 23
  5. 2023When are ensembles really effective?cited by 21
  6. 2023Pathologies of Predictive Diversity in Deep Ensemblescited by 16
  7. 2023A Symmetry-Aware Exploration of Bayesian Neural Network Posteriorscited by 12
  8. 2023Verifying Generalization in Deep Learningcited by 11
  9. 2024PAC-Bayes-Chernoff bounds for unbounded lossescited by 9
  10. 2023Leveraging Ensemble Diversity for Robust Self-Training in the Presence of Sample Selection Biascited by 7
  11. 2022Ensembling over Classifiers: a Bias-Variance Perspectivecited by 6
  12. 2023Are Ensembles Getting Better all the Time?cited by 6
20 more
  1. 2024Understanding Model Ensemble in Transferable Adversarial Attackcited by 6
  2. 2024Continual learning with the neural tangent ensemblecited by 4
  3. 2024Sharpness-diversity tradeoff: improving flat ensembles with SharpBalancecited by 3
  4. 2023Diversified Ensemble of Independent Sub-Networks for Robust Self-Supervised Representation Learningcited by 2
  5. 2025Parameter Expanded Stochastic Gradient Markov Chain Monte Carlocited by 2
  6. 2025DuetGraph: Coarse-to-Fine Knowledge Graph Reasoning with Dual-Pathway Global-Local Fusioncited by 2
  7. 2025Neural Diversity Regularizes Hallucinations in Language Modelscited by 2
  8. 2024Diversity-Aware Agnostic Ensemble of Sharpness Minimizerscited by 1
  9. 2024Ex Uno Pluria: Insights on Ensembling in Low Precision Number Systemscited by 1
  10. 2025Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversitycited by 1
  11. 2023EnSolver: Uncertainty-Aware Ensemble CAPTCHA Solvers with Theoretical Guaranteescited by 0
  12. 2023Something for (almost) nothing: Improving deep ensemble calibration using unlabeled datacited by 0
  13. 2024TemporalAugmenter: An Ensemble Recurrent Based Deep Learning Approach for Signal Classificationcited by 0
  14. 2024On Uniform, Bayesian, and PAC-Bayesian Deep Ensemblescited by 0
  15. 2024Understanding the Role of Functional Diversity in Weight-Ensembling with Ingredient Selection and Multidimensional Scalingcited by 0
  16. 2025TreeNet: Layered Decision Ensemblescited by 0
  17. 2025Value of Information-Enhanced Exploration in Bootstrapped DQNcited by 0
  18. 2026Measuring Dataset Diversity from a Geometric Perspectivecited by 0
  19. 2026Meta-Ensemble Learning with Diverse Data Splits for Improved Respiratory Sound Classificationcited by 0
  20. 2026Tabular Deep Learning for Algorithmic Trading: Cross-Regime Bayesian Optimisation for Equity Signal Generationcited by 0