mlsanityData to 14 Oct 2026

A Unified Theory of Diversity in Ensemble Learning

arXiv 2301.03962 ↗First posted 10 Jan 2023cs.LG cs.AI stat.MLcited by 71

Not on the map. Why?

Abstract (arXiv)

We present a theory of ensemble diversity, explaining the nature of diversity for a wide range of supervised learning scenarios. This challenge has been referred to as the holy grail of ensemble learning, an open research issue for over 30 years. Our framework reveals that diversity is in fact a hidden dimension in the bias-variance decomposition of the ensemble loss. We prove a family of exact bias-variance-diversity decompositions, for a wide range of losses in both regression and classification, e.g., squared, cross-entropy, and Poisson losses. For losses where an additive bias-variance decomposition is not available (e.g., 0/1 loss) we present an alternative approach: quantifying the effects of diversity, which turn out to be dependent on the label distribution. Overall, we argue that diversity is a measure of model fit, in precisely the same sense as bias and variance, but accounting for statistical dependencies between ensemble members. Thus, we should not be maximising diversity as so many works aim to do -- instead, we have a bias/variance/diversity trade-off to manage.

Key facts. Cited by 71 papers in the map (data through 14 Oct 2026). Builds on 13 of its 64 references that are in the map. First posted on arXiv 10 Jan 2023.

Builds on

13 of 64 references.

References · 13

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  3. 2019Improving Adversarial Robustness via Promoting Ensemble Diversitycited by 213
  4. 2020Rethinking Bias-Variance Trade-off for Generalization of Neural Networkscited by 107
  5. 2021DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimationcited by 48
  6. 2021Diversity and Generalization in Neural Network Ensemblescited by 32
  7. 2019When does Diversity Help Generalization in Classification Ensembles?cited by 27
  8. 2020Diversity inducing Information Bottleneck in Model Ensemblescited by 27
  9. 2020When Ensembling Smaller Models is More Efficient than Single Large Modelscited by 20
  10. 2020Generalized Negative Correlation Learning for Deep Ensemblingcited by 17
  11. 2013Generalized Ambiguity Decomposition for Understanding Ensemble Diversitycited by 2
  12. 2022Bias-Variance Decompositions for Margin Lossescited by 2
1 more
  1. 2016An ensemble diversity approach to supervised binary hashingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 6; 2024: 14; 2025: 20; 2026: 31.

71 papers.

Sort:

By area:

  1. 2023Pathologies of Predictive Diversity in Deep Ensemblescited by 16
  2. 2025GaussMarker: Robust Dual-Domain Watermark for Diffusion Modelscited by 16
  3. 2024Using Uncertainty Quantification to Characterize and Improve Out-of-Domain Learning for PDEscited by 15
  4. 2023Understanding and Improving Ensemble Adversarial Defensecited by 14
  5. 2026When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Modelscited by 13
  6. 2025Uncertainty-Aware Decoding with Minimum Bayes Riskcited by 11
  7. 2024Diversity Explains Inference Scaling Laws: Through a Case Study of Minimum Bayes Risk Decodingcited by 9
  8. 2026Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensemblescited by 9
  9. 2023Q(D)O-ES: Population-based Quality (Diversity) Optimisation for Post Hoc Ensemble Selection in AutoMLcited by 8
  10. 2025Credal Prediction based on Relative Likelihoodcited by 8
  11. 2024Understanding Model Ensemble in Transferable Adversarial Attackcited by 7
  12. 2025Bias-variance decompositions: the exclusive privilege of Bregman divergencescited by 6
38 more
  1. 2025Wisdom from Diversity: Bias Mitigation Through Hybrid Human-LLM Crowdscited by 6
  2. 2025DUAL: Learning Diverse Kernels for Aggregated Two-sample and Independence Testingcited by 6
  3. 2025CrowdLLM: Building LLM-Based Digital Populations Augmented with Generative Modelscited by 5
  4. 2026Quantifying Diversity of Thought: A Predictive Law of Weighted LLM Ensemble Liftcited by 5
  5. 2024Achieving More with Less: A Tensor-Optimization-Powered Ensemble Methodcited by 4
  6. 2024Learning to Explore for Stochastic Gradient MCMCcited by 4
  7. 2023Robust Collaborative Inference with Vertically Split Data Over Dynamic Device Environmentscited by 2
  8. 2024A Bias-Variance Decomposition for Ensembles over Multiple Synthetic Datasetscited by 2
  9. 2024Out-of-Distribution Detection with Attention Head Masking for Multimodal Document Classificationcited by 2
  10. 2025Parameter Expanded Stochastic Gradient Markov Chain Monte Carlocited by 2
  11. 2025Diversity of Transformer Layers: One Aspect of Parameter Scaling Lawscited by 2
  12. 2025Test-Time Efficient Pretrained Model Portfolios for Time Series Forecastingcited by 2
  13. 2023Bridging Generative and Discriminative Noisy-Label Learning via Direction-Agnostic EM Formulationcited by 1
  14. 2023Structured Radial Basis Function Network: Modelling Diversity for Multiple Hypotheses Predictioncited by 1
  15. 2024Towards a Systematic Approach to Design New Ensemble Learning Algorithmscited by 1
  16. 2024Measuring training variability from stochastic optimization using robust nonparametric testingcited by 1
  17. 2024EnsLoss: Stochastic Calibrated Loss Ensembles for Preventing Overfitting in Classificationcited by 1
  18. 2024Regularized Neural Ensemblerscited by 1
  19. 2024Ex Uno Pluria: Insights on Ensembling in Low Precision Number Systemscited by 1
  20. 2025Mixture of Group Experts for Learning Invariant Representationscited by 1
  21. 2025Meta Fusion: A Unified Framework For Multimodality Fusion with Mutual Learningcited by 1
  22. 2025Structured Basis Function Networks: Loss-Centric Multi-Hypothesis Ensembles with Controllable Diversitycited by 1
  23. 2025iLTM: Integrated Large Tabular Modelcited by 1
  24. 2025Confidence-Credibility Aware Weighted Ensembles of Small LLMs Outperform Large LLMs in Emotion Detectioncited by 1
  25. 2026When Are Two Scores Better Than One? Investigating Ensembles of Diffusion Modelscited by 1
  26. 2026Model Agreement via Anchoringcited by 1
  27. 2026Conflict-Resilient Multi-Agent Reasoning via Signed Graph Modelingcited by 1
  28. 2026Subjective Risk Decomposition: A New View for Uncertainty Quantificationcited by 1
  29. 2026Harnessing Disagreement: Detecting Correlated Agreement Blindness in Multi-Agent Triagecited by 1
  30. 2024Logifold: A Geometrical Foundation of Ensemble Machine Learningcited by 0
  31. 2024Understanding the Role of Functional Diversity in Weight-Ensembling with Ingredient Selection and Multidimensional Scalingcited by 0
  32. 2025Quantifying Correlations of Machine Learning Modelscited by 0
  33. 2025Imbalanced malware classification: an approach based on dynamic classifier selectioncited by 0
  34. 2025A Ridge Too Far: Correcting Over-Shrinkage via Negative Regularizationcited by 0
  35. 2025TreeNet: Layered Decision Ensemblescited by 0
  36. 2025On Joint Regularization and Calibration in Deep Ensemblescited by 0
  37. 2026Logarithmic Scores, Power-Law Discoveries: Disentangling Measurement from Coverage in Agent-Based Evaluationcited by 0
  38. 2026Meta-Ensemble Learning with Diverse Data Splits for Improved Respiratory Sound Classificationcited by 0

50 most cited of 71 All 71 (JSON)