mlsanityData to 14 Oct 2026

Snapshot Ensembles: Train 1, get M for free

arXiv 1704.00109 ↗First posted 1 Apr 2017cs.LGcited by 350

Not on the map. Why?

Abstract (arXiv)

Ensembles of neural networks are known to be much more robust and accurate than individual networks. However, training multiple deep networks for model averaging is computationally expensive. In this paper, we propose a method to obtain the seemingly contradictory goal of ensembling multiple neural networks at no additional training cost. We achieve this goal by training a single neural network, converging to several local minima along its optimization path and saving the model parameters. To obtain repeated rapid convergence, we leverage recent work on cyclic learning rate schedules. The resulting technique, which we refer to as Snapshot Ensembling, is simple, yet surprisingly effective. We show in a series of experiments that our approach is compatible with diverse network architectures and learning tasks. It consistently yields lower error rates than state-of-the-art single models at no additional training cost, and compares favorably with traditional network ensembles. On CIFAR-10 and CIFAR-100 our DenseNet Snapshot Ensembles obtain error rates of 3.4% and 17.4% respectively.

Key facts. Cited by 350 papers in the map (data through 14 Oct 2026). First posted on arXiv 1 Apr 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 69; 2022: 66; 2023: 68; 2024: 58; 2025: 46; 2026: 43.

350 papers.

Sort:

By area:

  1. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 982
  2. 2021A Survey of Uncertainty in Deep Neural Networkscited by 680
  3. 2022ActionFormer: Localizing Moments of Actions with Transformerscited by 311
  4. 2021Ensemble deep learning: A reviewcited by 284
  5. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 179
  6. 2021Well-tuned Simple Nets Excel on Tabular Datasetscited by 151
  7. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 138
  8. 2021Estimating and Exploiting the Aleatoric Uncertainty in Surface Normal Estimationcited by 116
  9. 2021Repulsive Deep Ensembles are Bayesiancited by 104
  10. 2024ARGS: Alignment as Reward-Guided Searchcited by 103
  11. 2023Deep Model Fusion: A Surveycited by 91
  12. 2021Learning Neural Network Subspacescited by 82
38 more
  1. 2024Process Reward Model with Q-Value Rankingscited by 80
  2. 2021Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learningcited by 78
  3. 2022When Do Flat Minima Optimizers Work?cited by 78
  4. 2021Adaptive Focus for Efficient Video Recognitioncited by 77
  5. 2024VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimationcited by 73
  6. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 72
  7. 2021Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimationcited by 71
  8. 2023TinyGSM: achieving >80% on GSM8k with small language modelscited by 70
  9. 2023LoRA ensembles for large language model fine-tuningcited by 67
  10. 2022Deep Ensembles Work, But Are They Necessary?cited by 63
  11. 2021Regularization Strategy for Point Cloud via Rigidly Mixed Samplecited by 59
  12. 2023A survey on deep learning in medical image registration: new technologies, uncertainty, evaluation metrics, and beyondcited by 55
  13. 2023Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytescited by 55
  14. 2022Packed-Ensembles for Efficient Uncertainty Estimationcited by 49
  15. 2021DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimationcited by 48
  16. 2021Nondeterminism and Instability in Neural Network Optimizationcited by 48
  17. 2021A Survey on Ensemble Learning under the Era of Deep Learningcited by 47
  18. 2021Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsitycited by 44
  19. 2023PaSCo: Urban 3D Panoptic Scene Completion with Uncertainty Awarenesscited by 42
  20. 2022Efficient Knowledge Distillation from Model Checkpointscited by 37
  21. 2025Stochastic Self-Guidance for Training-Free Enhancement of Diffusion Modelscited by 36
  22. 2021Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safetycited by 35
  23. 2021Unsupervised Time Series Outlier Detection with Diversity-Driven Convolutional Ensembles -- Extended Versioncited by 35
  24. 2023A comprehensive survey on deep active learning in medical image analysiscited by 35
  25. 2022UMIX: Improving Importance Weighting for Subpopulation Shift via Uncertainty-Aware Mixupcited by 34
  26. 2022NeuralUQ: A comprehensive library for uncertainty quantification in neural differential equations and operatorscited by 33
  27. 2024Breaking the Ceiling of the LLM Community by Treating Token Generation as a Classification for Ensemblingcited by 33
  28. 2021MixMo: Mixing Multiple Inputs for Multiple Outputs via Deep Subnetworkscited by 32
  29. 2022Discovering and forecasting extreme events via active learning in neural operatorscited by 32
  30. 2025Pre-training under infinite computecited by 29
  31. 2021What can linear interpolation of neural network loss landscapes tell us?cited by 28
  32. 2023TemporalMaxer: Maximize Temporal Context with only Max Pooling for Temporal Action Localizationcited by 28
  33. 2023Quantification of Uncertainty with Adversarial Modelscited by 28
  34. 2026Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weightscited by 28
  35. 2022RegMixup: Mixup as a Regularizer Can Surprisingly Improve Accuracy and Out Distribution Robustnesscited by 27
  36. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 27
  37. 2021Annealed Stein Variational Gradient Descentcited by 26
  38. 2022Stochastic Weight Averaging Revisitedcited by 26

50 most cited of 350 All 350 (JSON)