mlsanityData to 7 Oct 2026

Hyperparameter Ensembles for Robustness and Uncertainty Quantification

arXiv 2006.13570 ↗First posted 24 Jun 2020cs.LG stat.MLcited by 149

Not on the map. Why?

Abstract (arXiv)

Ensembles over neural network weights trained from different random initialization, known as deep ensembles, achieve state-of-the-art accuracy and calibration. The recently introduced batch ensembles provide a drop-in replacement that is more parameter efficient. In this paper, we design ensembles not only over weights, but over hyperparameters to improve the state of the art in both settings. For best performance independent of budget, we propose hyper-deep ensembles, a simple procedure that involves a random search over different hyperparameters, themselves stratified across multiple random initializations. Its strong performance highlights the benefit of combining models with both weight and hyperparameter diversity. We further propose a parameter efficient version, hyper-batch ensembles, which builds on the layer structure of batch ensembles and self-tuning networks. The computational and memory costs of our method are notably lower than typical ensembles. On image classification tasks, with MLP, LeNet, ResNet 20 and Wide ResNet 28-10 architectures, we improve upon both deep and batch ensembles.

Key facts. Cited by 149 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 43; 2022: 28; 2023: 35; 2024: 17; 2025: 17; 2026: 9.

149 papers.

Sort:

By area:

  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  2. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 643
  3. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  4. 2021Laplace Redux -- Effortless Bayesian Deep Learningcited by 359
  5. 2021Well-tuned Simple Nets Excel on Tabular Datasetscited by 151
  6. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  7. 2021Deep Learning Through the Lens of Example Difficultycited by 139
  8. 2021Bayesian Neural Network Priors Revisitedcited by 121
  9. 2021Predicting with Confidence on Unseen Distributionscited by 116
  10. 2021Repulsive Deep Ensembles are Bayesiancited by 103
  11. 2021Priors in Bayesian Deep Learning: A Reviewcited by 101
  12. 2021The Devil is in the Margin: Margin-based Label Smoothing for Network Calibrationcited by 91
38 more
  1. 2022Deep Isolation Forest for Anomaly Detectioncited by 90
  2. 2021Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learningcited by 81
  3. 2021Graph Posterior Network: Bayesian Predictive Uncertainty for Node Classificationcited by 79
  4. 2023A Survey on Uncertainty Quantification Methods for Deep Learningcited by 78
  5. 2021Does Your Dermatology Classifier Know What It Doesn't Know? Detecting the Long-Tail of Unseen Conditionscited by 68
  6. 2023LoRA ensembles for large language model fine-tuningcited by 65
  7. 2021No One Representation to Rule Them All: Overlapping Features of Training Methodscited by 57
  8. 2022Assaying Out-Of-Distribution Generalization in Transfer Learningcited by 53
  9. 2021Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsitycited by 44
  10. 2022Uncertainty-Informed Deep Learning Models Enable High-Confidence Predictions for Digital Histopathologycited by 37
  11. 2021MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworkscited by 32
  12. 2021Diversity and Generalization in Neural Network Ensemblescited by 32
  13. 2021AutoDEUQ: Automated Deep Ensemble with Uncertainty Quantificationcited by 31
  14. 2023Synthetic data, real errors: how (not) to publish and use synthetic datacited by 29
  15. 2021Deep Ensembles from a Bayesian Perspectivecited by 27
  16. 2022Disentangling Epistemic and Aleatoric Uncertainty in Reinforcement Learningcited by 27
  17. 2021BORE: Bayesian Optimization by Density-Ratio Estimationcited by 26
  18. 2022Hyperparameter Sensitivity in Deep Outlier Detection: Analysis and a Scalable Hyper-Ensemble Solutioncited by 24
  19. 2021Robustness via Cross-Domain Ensemblescited by 23
  20. 2021On Stein Variational Neural Network Ensemblescited by 21
  21. 2023Sensitivity-Aware Amortized Bayesian Inferencecited by 20
  22. 2024LoRA-Ensemble: Efficient Uncertainty Modelling for Self-Attention Networkscited by 20
  23. 2021Sparse MoEs meet Efficient Ensemblescited by 19
  24. 2023Uncertainty-inspired Open Set Learning for Retinal Anomaly Identificationcited by 18
  25. 2021NOMU: Neural Optimization-based Model Uncertaintycited by 16
  26. 2021Orthogonal Ensemble Networks for Biomedical Image Segmentationcited by 16
  27. 2021Efficient Self-Ensemble for Semantic Segmentationcited by 16
  28. 2022Can we achieve robustness from data alone?cited by 16
  29. 2023Pathologies of Predictive Diversity in Deep Ensemblescited by 16
  30. 2023Multiclass Confidence and Localization Calibration for Object Detectioncited by 16
  31. 2021Agree to Disagree: When Deep Learning Models With Identical Architectures Produce Distinct Explanationscited by 15
  32. 2022ADMoE: Anomaly Detection with Mixture-of-Experts from Noisy Labelscited by 15
  33. 2023Image retrieval outperforms diffusion models on data augmentationcited by 14
  34. 2024Ctrl-U: Robust Conditional Image Generation via Uncertainty-aware Reward Modelingcited by 14
  35. 2021Natural Posterior Network: Deep Bayesian Uncertainty for Exponential Family Distributionscited by 13
  36. 2021Maximum Entropy on Erroneous Predictions (MEEP): Improving model calibration for medical image segmentationcited by 13
  37. 2022Prune and Tune Ensembles: Low-Cost Ensemble Learning With Sparse Independent Subnetworkscited by 13
  38. 2021Combining Diverse Feature Priorscited by 12

50 most cited of 149 All 149 (JSON)