mlsanityData to 7 Oct 2026

How Good is a Single Basin?

arXiv 2402.03187 ↗First posted 5 Feb 2024cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

The multi-modal nature of neural loss landscapes is often considered to be the main driver behind the empirical success of deep ensembles. In this work, we probe this belief by constructing various "connected" ensembles which are restricted to lie in the same basin. Through our experiments, we demonstrate that increased connectivity indeed negatively impacts performance. However, when incorporating the knowledge from other basins implicitly through distillation, we show that the gap in performance can be mitigated by re-discovering (multi-basin) deep ensembles within a single basin. Thus, we conjecture that while the extra-basin knowledge is at least partially present in any given basin, it cannot be easily harnessed without learning it from other basins.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 31 of its 52 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

31 of 52 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  6. 2016Layer Normalizationcited by 5,579
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  8. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  9. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  10. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  11. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  12. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
19 more
  1. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  2. 2018Born Again Neural Networkscited by 539
  3. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
  4. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  5. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
  6. 2020What is being transferred in transfer learning?cited by 365
  7. 2017Snapshot Ensembles: Train 1, get M for freecited by 349
  8. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  9. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  10. 2014The Loss Surfaces of Multilayer Networkscited by 278
  11. 2019Model Fusion via Optimal Transportcited by 217
  12. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  13. 2014Qualitatively characterizing neural network optimization problemscited by 186
  14. 2020Hyperparameter Ensembles for Robustness and Uncertainty Quantificationcited by 149
  15. 2021Repulsive Deep Ensembles are Bayesiancited by 103
  16. 2021Learning Neural Network Subspacescited by 82
  17. 2021Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensemblingcited by 75
  18. 2022Linear Connectivity Reveals Generalization Strategiescited by 55
  19. 2023Pathologies of Predictive Diversity in Deep Ensemblescited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2025Uncertainty-Aware Decoding with Minimum Bayes Riskcited by 10
  2. 2025The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditionscited by 9
  3. 2024Ex Uno Pluria: Insights on Ensembling in Low Precision Number Systemscited by 1
  4. 2026Misalignment of Low-Loss Regions Causes Grokkingcited by 0