mlsanityData to 14 Oct 2026

Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks

arXiv 1511.06314 ↗First posted 19 Nov 2015cs.CV cs.LG cs.NEcited by 139

Not on the map. Why?

Abstract (arXiv)

Convolutional Neural Networks have achieved state-of-the-art performance on a wide range of tasks. Most benchmarks are led by ensembles of these powerful learners, but ensembling is typically treated as a post-hoc procedure implemented by averaging independently trained models with model variation induced by bagging or random initialization. In this paper, we rigorously treat ensembling as a first-class problem to explicitly address the question: what are the best strategies to create an ensemble? We first compare a large number of ensembling strategies, and then propose and evaluate novel strategies, such as parameter sharing (through a new family of models we call TreeNets) as well as training under ensemble-aware and diversity-encouraging losses. We demonstrate that TreeNets can improve ensemble performance and that diverse ensembles can be trained end-to-end under a unified loss, achieving significantly higher "oracle" accuracies than classical ensembles.

Key facts. Cited by 139 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Nov 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 31; 2022: 31; 2023: 24; 2024: 24; 2025: 15; 2026: 14.

139 papers.

Sort:

By area:

  1. 2021A Survey of Uncertainty in Deep Neural Networkscited by 680
  2. 2021DyTox: Transformers for Continual Learning with DYnamic TOken eXpansioncited by 291
  3. 2021Objects are Different: Flexible Monocular 3D Object Detectioncited by 185
  4. 2024TabM: Advancing Tabular Deep Learning with Parameter-Efficient Ensemblingcited by 113
  5. 2024Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Taskscited by 78
  6. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 76
  7. 2023LoRA ensembles for large language model fine-tuningcited by 67
  8. 2021Representation Topology Divergence: A Method for Comparing Neural Network Representationscited by 66
  9. 2022Deep Ensembles Work, But Are They Necessary?cited by 63
  10. 2022Packed-Ensembles for Efficient Uncertainty Estimationcited by 49
  11. 2021DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimationcited by 48
  12. 2021Quantifying Uncertainty in Deep Spatiotemporal Forecastingcited by 48
38 more
  1. 2021Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsitycited by 44
  2. 2022Transframer: Arbitrary Frame Prediction with Generative Modelscited by 37
  3. 2021MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworkscited by 32
  4. 2021Valid prediction intervals for regression problemscited by 29
  5. 2021Uncertainty Estimation via Response Scaling for Pseudo-mask Noise Mitigation in Weakly-supervised Semantic Segmentationcited by 29
  6. 2021Coarse to Fine Multi-Resolution Temporal Convolutional Networkcited by 28
  7. 2021Transferability Metrics for Selecting Source Model Ensemblescited by 24
  8. 2023Joint Training of Deep Ensembles Fails Due to Learner Collusioncited by 23
  9. 2021Robust Semantic Segmentation with Superpixel-Mixcited by 19
  10. 2021Sparse MoEs meet Efficient Ensemblescited by 19
  11. 2022Satellite Image Based Cross-view Localization for Autonomous Vehiclecited by 19
  12. 2022Weighted Ensemble Self-Supervised Learningcited by 19
  13. 2021Using Soft Labels to Model Uncertainty in Medical Image Segmentationcited by 18
  14. 2022Verification-Aided Deep Ensemble Selectioncited by 18
  15. 2023URL: A Representation Learning Benchmark for Transferable Uncertainty Estimatescited by 17
  16. 2021Efficient Self-Ensemble for Semantic Segmentationcited by 16
  17. 2023Pathologies of Predictive Diversity in Deep Ensemblescited by 16
  18. 2022Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computationcited by 14
  19. 2024Self-Ensembling Gaussian Splatting for Few-Shot Novel View Synthesiscited by 14
  20. 2022Prune and Tune Ensembles: Low-Cost Ensemble Learning With Sparse Independent Subnetworkscited by 13
  21. 2022Deep Learning for Brain Age Estimation: A Systematic Reviewcited by 13
  22. 2023Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learningcited by 12
  23. 2023A Symmetry-Aware Exploration of Bayesian Neural Network Posteriorscited by 12
  24. 2023Resilient Multiple Choice Learning: A learned scoring scheme with application to audio scene analysiscited by 12
  25. 2021Uncertainty Quantification by Ensemble Learning for Computational Optical Form Measurementscited by 11
  26. 2021Efficient Diversity-Driven Ensemble for Deep Neural Networkscited by 11
  27. 2023Towards Inference Efficient Deep Ensemble Learningcited by 11
  28. 2023FAIR-Ensemble: When Fairness Naturally Emerges From Deep Ensemblingcited by 11
  29. 2021Accuracy-Privacy Trade-off in Deep Ensemble: A Membership Inference Perspectivecited by 10
  30. 2021JUWELS Booster -- A Supercomputer for Large-Scale AI Researchcited by 10
  31. 2021Quantifying Epistemic Uncertainty in Deep Learningcited by 10
  32. 2023Efficient Uncertainty Quantification and Reduction for Over-Parameterized Neural Networkscited by 10
  33. 2021Generalisation effects of predictive uncertainty estimation in deep learning for digital pathologycited by 9
  34. 2022Highlighting Object Category Immunity for the Generalization of Human-Object Interaction Detectioncited by 9
  35. 2022Improving robustness and calibration in ensembles with diversity regularizationcited by 8
  36. 2023Multi-Head Multi-Loss Model Calibrationcited by 8
  37. 2024Jointly Modeling Inter- & Intra-Modality Dependencies for Multi-modal Learningcited by 8
  38. 2024Function Space Diversity for Uncertainty Prediction via Repulsive Last-Layer Ensemblescited by 8

50 most cited of 139 All 139 (JSON)