mlsanityData to 14 Oct 2026

Multi-headed Neural Ensemble Search

arXiv 2107.04369 ↗First posted 9 Jul 2021cs.LG stat.MLcited by 4

Not on the map. Why?

Abstract (arXiv)

Ensembles of CNN models trained with different seeds (also known as Deep Ensembles) are known to achieve superior performance over a single copy of the CNN. Neural Ensemble Search (NES) can further boost performance by adding architectural diversity. However, the scope of NES remains prohibitive under limited computational resources. In this work, we extend NES to multi-headed ensembles, which consist of a shared backbone attached to multiple prediction heads. Unlike Deep Ensembles, these multi-headed ensembles can be trained end to end, which enables us to leverage one-shot NAS methods to optimize an ensemble objective. With extensive empirical evaluations, we demonstrate that multi-headed ensemble search finds robust ensembles 3 times faster, while having comparable performance to other ensemble search methods, in both predictive performance and uncertainty calibration.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 24 of its 34 references that are in the map. First posted on arXiv 9 Jul 2021.

Builds on

24 of 34 references.

References · 24

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  5. 2016Wide Residual Networkscited by 3,070
  6. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  7. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  8. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,287
  9. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 909
  10. 2018ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardwarecited by 673
  11. 2018BOHB: Robust and Efficient Hyperparameter Optimization at Scalecited by 470
  12. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 443
12 more
  1. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 371
  2. 2017Snapshot Ensembles: Train 1, get M for freecited by 350
  3. 2019PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Searchcited by 265
  4. 2019Random Search and Reproducibility for Neural Architecture Searchcited by 254
  5. 2019Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Visioncited by 182
  6. 2019Understanding and Robustifying Differentiable Architecture Searchcited by 178
  7. 2020Hyperparameter Ensembles for Robustness and Uncertainty Quantificationcited by 150
  8. 2015Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networkscited by 139
  9. 2019Diversity with Cooperation: Ensemble Methods for Few-Shot Classificationcited by 70
  10. 2019BayesNAS: A Bayesian Approach for Neural Architecture Searchcited by 68
  11. 2017Simple And Efficient Architecture Search for Convolutional Neural Networkscited by 40
  12. 2019Improving Neural Architecture Search Image Classifiers via Ensemble Learningcited by 2

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 2.

4 papers.

  1. 2023Neural Architecture Search: Insights from 1000 Paperscited by 139
  2. 2022Probing Contextual Diversity for Dense Out-of-Distribution Detectioncited by 3
  3. 2023A Multi-Head Ensemble Multi-Task Learning Approach for Dynamical Computation Offloadingcited by 2
  4. 2022Evolutionary Neural Cascade Search across Supernetworkscited by 0