# When does Diversity Help Generalization in Classification Ensembles?

arXiv: https://arxiv.org/abs/1910.13631 · HTML: https://mlsanity.com/paper/1910.13631/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2019.

## Built on by (most cited first, 27 of 27)

- [Robust fine-tuning of zero-shot models](https://mlsanity.com/paper/2109.01903/) (2021)
- [A Unified Theory of Diversity in Ensemble Learning](https://mlsanity.com/paper/2301.03962/) (2023)
- [A Survey on Epistemic (Model) Uncertainty in Supervised Learning: Recent Advances and Applications](https://mlsanity.com/paper/2111.01968/) (2021)
- [Transferability Metrics for Selecting Source Model Ensembles](https://mlsanity.com/paper/2111.13011/) (2021)
- [Revisiting the Importance of Amplifying Bias for Debiasing](https://mlsanity.com/paper/2205.14594/) (2022)
- [Towards Understanding Fairness and its Composition in Ensemble Machine Learning](https://mlsanity.com/paper/2212.04593/) (2022)
- [Exploring the Transferability of Visual Prompting for Multimodal Large Language Models](https://mlsanity.com/paper/2404.11207/) (2024)
- [Hierarchically Structured Task-Agnostic Continual Learning](https://mlsanity.com/paper/2211.07725/) (2022)
- [Mixture-of-Variational-Experts for Continual Learning](https://mlsanity.com/paper/2110.12667/) (2021)
- [Generalized Incremental Learning under Concept Drift across Evolving Data Streams](https://mlsanity.com/paper/2506.05736/) (2025)
- [DivBO: Diversity-aware CASH for Ensemble Learning](https://mlsanity.com/paper/2302.03255/) (2023)
- [Motor Imagery Decoding Using Ensemble Curriculum Learning and Collaborative Training](https://mlsanity.com/paper/2211.11460/) (2022)
- [Generalization Gap in Data Augmentation: Insights from Illumination](https://mlsanity.com/paper/2404.07514/) (2024)
- [LENAS: Learning-based Neural Architecture Search and Ensemble for 3D Radiotherapy Dose Prediction](https://mlsanity.com/paper/2106.06733/) (2021)
- [Rover: An online Spark SQL tuning service via generalized transfer learning](https://mlsanity.com/paper/2302.04046/) (2023)
- [Tree-Structured Synergy of Large Language Models and Bayesian Optimization for Efficient CASH](https://mlsanity.com/paper/2601.12355/) (2026)
- [Deep Negative Correlation Classification](https://mlsanity.com/paper/2212.07070/) (2022)
- [A hybrid ensemble method with negative correlation learning for regression](https://mlsanity.com/paper/2104.02317/) (2021)
- [Dynamic Neural Diversification: Path to Computationally Sustainable Neural Networks](https://mlsanity.com/paper/2109.09612/) (2021)
- [Incentive Mechanism Design for Distributed Ensemble Learning](https://mlsanity.com/paper/2310.08792/) (2023)
- [Grammar-based evolutionary approach for automated workflow composition with domain-specific operators and ensemble diversity](https://mlsanity.com/paper/2402.02124/) (2024)
- [On the Effectiveness of Heterogeneous Ensemble Methods for Re-identification](https://mlsanity.com/paper/2403.12606/) (2024)
- [Reducing Diversity to Generate Hierarchical Archetypes](https://mlsanity.com/paper/2409.18633/) (2024)
- [TriagerX: Dual Transformers for Bug Triaging Tasks with Content and Interaction Based Rankings](https://mlsanity.com/paper/2508.16860/) (2025)
- [Measuring Dataset Diversity from a Geometric Perspective](https://mlsanity.com/paper/2602.09340/) (2026)
- [NCAA Bracket Prediction Using Machine Learning and Combinatorial Fusion Analysis](https://mlsanity.com/paper/2603.10916/) (2026)
- [Which Models Work Well Together? Measuring Heterogeneity for LLM Team Selection](https://mlsanity.com/paper/2609.38274/) (2026)

## Abstract (arXiv)

Ensembles, as a widely used and effective technique in the machine learning community, succeed within a key element -- "diversity." The relationship between diversity and generalization, unfortunately, is not entirely understood and remains an open research issue. To reveal the effect of diversity on the generalization of classification ensembles, we investigate three issues on diversity, i.e., the measurement of diversity, the relationship between the proposed diversity and the generalization error, and the utilization of this relationship for ensemble pruning. In the diversity measurement, we measure diversity by error decomposition inspired by regression ensembles, which decomposes the error of classification ensembles into accuracy and diversity. Then we formulate the relationship between the measured diversity and ensemble performance through the theorem of margin and generalization and observe that the generalization error is reduced effectively only when the measured diversity is increased in a few specific ranges, while in other ranges larger diversity is less beneficial to increasing the generalization of an ensemble. Besides, we propose two pruning methods based on diversity management to utilize this relationship, which could increase diversity appropriately and shrink the size of the ensemble without much-decreasing performance. Empirical results validate the reasonableness of the proposed relationship between diversity and ensemble generalization error and the effectiveness of the proposed pruning methods.