# A Unified Theory of Diversity in Ensemble Learning

arXiv: https://arxiv.org/abs/2301.03962 · HTML: https://mlsanity.com/paper/2301.03962/

## Key facts

- Cited by 71 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 64 references that are in the map.
- First posted on arXiv 10 Jan 2023.

## Builds on (most cited first, 13 of 13)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Improving Adversarial Robustness via Promoting Ensemble Diversity](https://mlsanity.com/paper/1901.08846/) (2019)
- [Rethinking Bias-Variance Trade-off for Generalization of Neural Networks](https://mlsanity.com/paper/2002.11328/) (2020)
- [DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation](https://mlsanity.com/paper/2101.05544/) (2021)
- [Diversity and Generalization in Neural Network Ensembles](https://mlsanity.com/paper/2110.13786/) (2021)
- [When does Diversity Help Generalization in Classification Ensembles?](https://mlsanity.com/paper/1910.13631/) (2019)
- [Diversity inducing Information Bottleneck in Model Ensembles](https://mlsanity.com/paper/2003.04514/) (2020)
- [When Ensembling Smaller Models is More Efficient than Single Large Models](https://mlsanity.com/paper/2005.00570/) (2020)
- [Generalized Negative Correlation Learning for Deep Ensembling](https://mlsanity.com/paper/2011.02952/) (2020)
- [Generalized Ambiguity Decomposition for Understanding Ensemble Diversity](https://mlsanity.com/paper/1312.7463/) (2013)
- [Bias-Variance Decompositions for Margin Losses](https://mlsanity.com/paper/2204.12155/) (2022)
- [An ensemble diversity approach to supervised binary hashing](https://mlsanity.com/paper/1602.01557/) (2016)

## Built on by (most cited first, 30 of 71)

- [Pathologies of Predictive Diversity in Deep Ensembles](https://mlsanity.com/paper/2302.00704/) (2023)
- [GaussMarker: Robust Dual-Domain Watermark for Diffusion Models](https://mlsanity.com/paper/2506.11444/) (2025)
- [Using Uncertainty Quantification to Characterize and Improve Out-of-Domain Learning for PDEs](https://mlsanity.com/paper/2403.10642/) (2024)
- [Understanding and Improving Ensemble Adversarial Defense](https://mlsanity.com/paper/2310.18477/) (2023)
- [When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models](https://mlsanity.com/paper/2606.27288/) (2026)
- [Uncertainty-Aware Decoding with Minimum Bayes Risk](https://mlsanity.com/paper/2503.05318/) (2025)
- [Diversity Explains Inference Scaling Laws: Through a Case Study of Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2410.15021/) (2024)
- [Are Diversity Metrics Measuring Diversity? A Capability-Controlled Audit of Majority-Vote Gain in LLM Ensembles](https://mlsanity.com/paper/2607.20768/) (2026)
- [Q(D)O-ES: Population-based Quality (Diversity) Optimisation for Post Hoc Ensemble Selection in AutoML](https://mlsanity.com/paper/2307.08364/) (2023)
- [Credal Prediction based on Relative Likelihood](https://mlsanity.com/paper/2505.22332/) (2025)
- [Understanding Model Ensemble in Transferable Adversarial Attack](https://mlsanity.com/paper/2410.06851/) (2024)
- [Bias-variance decompositions: the exclusive privilege of Bregman divergences](https://mlsanity.com/paper/2501.18581/) (2025)
- [Wisdom from Diversity: Bias Mitigation Through Hybrid Human-LLM Crowds](https://mlsanity.com/paper/2505.12349/) (2025)
- [DUAL: Learning Diverse Kernels for Aggregated Two-sample and Independence Testing](https://mlsanity.com/paper/2510.11140/) (2025)
- [CrowdLLM: Building LLM-Based Digital Populations Augmented with Generative Models](https://mlsanity.com/paper/2512.07890/) (2025)
- [Quantifying Diversity of Thought: A Predictive Law of Weighted LLM Ensemble Lift](https://mlsanity.com/paper/2607.17384/) (2026)
- [Achieving More with Less: A Tensor-Optimization-Powered Ensemble Method](https://mlsanity.com/paper/2408.02936/) (2024)
- [Learning to Explore for Stochastic Gradient MCMC](https://mlsanity.com/paper/2408.09140/) (2024)
- [Robust Collaborative Inference with Vertically Split Data Over Dynamic Device Environments](https://mlsanity.com/paper/2312.16638/) (2023)
- [A Bias-Variance Decomposition for Ensembles over Multiple Synthetic Datasets](https://mlsanity.com/paper/2402.03985/) (2024)
- [Out-of-Distribution Detection with Attention Head Masking for Multimodal Document Classification](https://mlsanity.com/paper/2408.11237/) (2024)
- [Parameter Expanded Stochastic Gradient Markov Chain Monte Carlo](https://mlsanity.com/paper/2503.00699/) (2025)
- [Diversity of Transformer Layers: One Aspect of Parameter Scaling Laws](https://mlsanity.com/paper/2505.24009/) (2025)
- [Test-Time Efficient Pretrained Model Portfolios for Time Series Forecasting](https://mlsanity.com/paper/2510.06419/) (2025)
- [Bridging Generative and Discriminative Noisy-Label Learning via Direction-Agnostic EM Formulation](https://mlsanity.com/paper/2308.01184/) (2023)
- [Structured Radial Basis Function Network: Modelling Diversity for Multiple Hypotheses Prediction](https://mlsanity.com/paper/2309.00781/) (2023)
- [Towards a Systematic Approach to Design New Ensemble Learning Algorithms](https://mlsanity.com/paper/2402.06818/) (2024)
- [Measuring training variability from stochastic optimization using robust nonparametric testing](https://mlsanity.com/paper/2406.08307/) (2024)
- [EnsLoss: Stochastic Calibrated Loss Ensembles for Preventing Overfitting in Classification](https://mlsanity.com/paper/2409.00908/) (2024)
- [Regularized Neural Ensemblers](https://mlsanity.com/paper/2410.04520/) (2024)

## Abstract (arXiv)

We present a theory of ensemble diversity, explaining the nature of diversity for a wide range of supervised learning scenarios. This challenge has been referred to as the holy grail of ensemble learning, an open research issue for over 30 years. Our framework reveals that diversity is in fact a hidden dimension in the bias-variance decomposition of the ensemble loss. We prove a family of exact bias-variance-diversity decompositions, for a wide range of losses in both regression and classification, e.g., squared, cross-entropy, and Poisson losses. For losses where an additive bias-variance decomposition is not available (e.g., 0/1 loss) we present an alternative approach: quantifying the effects of diversity, which turn out to be dependent on the label distribution. Overall, we argue that diversity is a measure of model fit, in precisely the same sense as bias and variance, but accounting for statistical dependencies between ensemble members. Thus, we should not be maximising diversity as so many works aim to do -- instead, we have a bias/variance/diversity trade-off to manage.