mlsanityData to 7 Oct 2026

To Stay or Not to Stay in the Pre-train Basin: Insights on Ensembling in Transfer Learning

arXiv 2303.03374 ↗First posted 6 Mar 2023cs.LG stat.MLcited by 6

Not on the map. Why?

Abstract (arXiv)

Transfer learning and ensembling are two popular techniques for improving the performance and robustness of neural networks. Due to the high cost of pre-training, ensembles of models fine-tuned from a single pre-trained checkpoint are often used in practice. Such models end up in the same basin of the loss landscape, which we call the pre-train basin, and thus have limited diversity. In this work, we show that ensembles trained from a single pre-trained checkpoint may be improved by better exploring the pre-train basin, however, leaving the basin results in losing the benefits of transfer learning and in degradation of the ensemble quality. Based on the analysis of existing exploration methods, we propose a more effective modification of the Snapshot Ensembles (SSE) for transfer learning setup, StarSSE, which results in stronger ensembles and uniform model soups.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 34 of its 41 references that are in the map. First posted on arXiv 6 Mar 2023.

Builds on

34 of 41 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  9. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  10. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  11. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  12. 2018Moment Matching for Multi-Source Domain Adaptationcited by 1,327
22 more
  1. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  2. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  3. 2017ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseasescited by 969
  4. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  5. 2018Do Better ImageNet Models Transfer Better?cited by 560
  6. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  7. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  8. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
  9. 2020What is being transferred in transfer learning?cited by 365
  10. 2017Snapshot Ensembles: Train 1, get M for freecited by 349
  11. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  12. 2020How Well Do Self-Supervised Models Transfer?cited by 213
  13. 2020Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learningcited by 208
  14. 2019Cyclical Stochastic Gradient MCMC for Bayesian Deep Learningcited by 203
  15. 2019Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Visioncited by 182
  16. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  17. 2021Solo-learn: A Library of Self-supervised Methods for Visual Representation Learningcited by 116
  18. 2021Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensemblingcited by 75
  19. 2021No One Representation to Rule Them All: Overlapping Features of Training Methodscited by 57
  20. 2022Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priorscited by 43
  21. 2017Ensemble Methods as a Defense to Adversarial Perturbations Against Deep Neural Networkscited by 31
  22. 2020Deep Ensembles for Low-Data Transfer Learningcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 1.

6 papers.

  1. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
  2. 2024A Second-Order Perspective on Model Compositionality and Incremental Learningcited by 18
  3. 2025The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditionscited by 9
  4. 2025Asymmetric Duos: Sidekicks Improve Uncertaintycited by 2
  5. 2024Ex Uno Pluria: Insights on Ensembling in Low Precision Number Systemscited by 1
  6. 2026Uncertainty-aware damage identification in short-span bridges via physics-informed variational autoencodercited by 0