# Hierarchical Pruning of Deep Ensembles with Focal Diversity

arXiv: https://arxiv.org/abs/2311.10293 · HTML: https://mlsanity.com/paper/2311.10293/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 42 references that are in the map.
- First posted on arXiv 17 Nov 2023.

## Builds on (most cited first, 12 of 12)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Deep Ensembles: A Loss Landscape Perspective](https://mlsanity.com/paper/1912.02757/) (2019)
- [The Relative Performance of Ensemble Methods with Deep Convolutional Neural Networks for Image Classification](https://mlsanity.com/paper/1704.01664/) (2017)
- [HOLMES: Health OnLine Model Ensemble Serving for Deep Learning Models in Intensive Care Units](https://mlsanity.com/paper/2008.04063/) (2020)
- [Deep Neural Network Ensembles against Deception: Ensemble Diversity, Accuracy and Robustness](https://mlsanity.com/paper/1908.11091/) (2019)
- [Ensemble Pruning based on Objection Maximization with a General Distributed Framework](https://mlsanity.com/paper/1806.04899/) (2018)
- [Robust Deep Learning Ensemble against Deception](https://mlsanity.com/paper/2009.06589/) (2020)
- [Cross-Layer Strategic Ensemble Defense Against Adversarial Examples](https://mlsanity.com/paper/1910.01742/) (2019)
- [Exploring Model Learning Heterogeneity for Boosting Ensemble Robustness](https://mlsanity.com/paper/2310.02237/) (2023)
- [Denoising and Verification Cross-Layer Ensemble Against Black-box Adversarial Attacks](https://mlsanity.com/paper/1908.07667/) (2019)
- [Promoting High Diversity Ensemble Learning with EnsembleBench](https://mlsanity.com/paper/2010.10623/) (2020)
- [Learning to Diversify via Weighted Kernels for Classifier Ensemble](https://mlsanity.com/paper/1406.1167/) (2014)

## Built on by (most cited first, 4 of 4)

- [Protego: User-Centric Pose-Invariant Privacy Protection Against Face Recognition-Induced Digital Footprint Exposure](https://mlsanity.com/paper/2508.02034/) (2025)
- [Boosting Deep Ensembles with Learning Rate Tuning](https://mlsanity.com/paper/2410.07564/) (2024)
- [RocketStack: Level-aware Deep Recursive Ensemble Learning Architecture](https://mlsanity.com/paper/2506.16965/) (2025)
- [A Diversity-optimized Deep Ensemble Approach for Accurate Plant Leaf Disease Detection](https://mlsanity.com/paper/2511.16982/) (2025)

## Abstract (arXiv)

Deep neural network ensembles combine the wisdom of multiple deep neural networks to improve the generalizability and robustness over individual networks. It has gained increasing popularity to study deep ensemble techniques in the deep learning community. Some mission-critical applications utilize a large number of deep neural networks to form deep ensembles to achieve desired accuracy and resilience, which introduces high time and space costs for ensemble execution. However, it still remains a critical challenge whether a small subset of the entire deep ensemble can achieve the same or better generalizability and how to effectively identify these small deep ensembles for improving the space and time efficiency of ensemble execution. This paper presents a novel deep ensemble pruning approach, which can efficiently identify smaller deep ensembles and provide higher ensemble accuracy than the entire deep ensemble of a large number of member networks. Our hierarchical ensemble pruning approach (HQ) leverages three novel ensemble pruning techniques. First, we show that the focal diversity metrics can accurately capture the complementary capacity of the member networks of an ensemble, which can guide ensemble pruning. Second, we design a focal diversity based hierarchical pruning approach, which will iteratively find high quality deep ensembles with low cost and high accuracy. Third, we develop a focal diversity consensus method to integrate multiple focal diversity metrics to refine ensemble pruning results, where smaller deep ensembles can be effectively identified to offer high accuracy, high robustness and high efficiency. Evaluated using popular benchmark datasets, we demonstrate that the proposed hierarchical ensemble pruning approach can effectively identify high quality deep ensembles with better generalizability while being more time and space efficient in ensemble decision making.