mlsanityData to 7 Oct 2026

Averaging Weights Leads to Wider Optima and Better Generalization

arXiv 1803.05407 ↗First posted 14 Mar 2018cs.LG cs.AI cs.CVcited by 1,194

Not on the map. Why?

Abstract (arXiv)

Deep neural networks are typically trained by optimizing a loss function with an SGD variant, in conjunction with a decaying learning rate, until convergence. We show that simple averaging of multiple points along the trajectory of SGD, with a cyclical or constant learning rate, leads to better generalization than conventional training. We also show that this Stochastic Weight Averaging (SWA) procedure finds much flatter solutions than SGD, and approximates the recent Fast Geometric Ensembling (FGE) approach with a single model. Using SWA we achieve notable improvement in test accuracy over conventional SGD training on a range of state-of-the-art residual networks, PyramidNets, DenseNets, and Shake-Shake networks on CIFAR-10, CIFAR-100, and ImageNet. In short, SWA is extremely easy to implement, improves generalization, and has almost no computational overhead.

Key facts. Cited by 1,194 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Mar 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 134; 2022: 195; 2023: 251; 2024: 224; 2025: 221; 2026: 169.

1,194 papers.

Sort:

By area:

  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  2. 2022Editing Models with Task Arithmeticcited by 1,076
  3. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 978
  4. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  5. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  6. 2021Robust fine-tuning of zero-shot modelscited by 785
  7. 2021Domain Generalization: A Surveycited by 760
  8. 2021AST: Audio Spectrogram Transformercited by 634
  9. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  10. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  11. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  12. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
38 more
  1. 2022Dataless Knowledge Fusion by Merging Weights of Language Modelscited by 384
  2. 2021ResNet strikes back: An improved training procedure in timmcited by 365
  3. 2021What Are Bayesian Neural Network Posteriors Really Like?cited by 307
  4. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  5. 2021A Survey on Deep Semi-supervised Learningcited by 250
  6. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  7. 2021Improving Robustness using Generated Datacited by 233
  8. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  9. 2025LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristicscited by 227
  10. 2021Efficiently Identifying Task Groupings for Multi-Task Learningcited by 218
  11. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  12. 2023Better Diffusion Models Further Improve Adversarial Trainingcited by 207
  13. 2023Cross-Entropy Loss Functions: Theoretical Analysis and Applicationscited by 206
  14. 2021Fixing Data Augmentation to Improve Adversarial Robustnesscited by 201
  15. 2021Data Augmentation Can Improve Robustnesscited by 193
  16. 2021SelfReg: Self-supervised Contrastive Regularization for Domain Generalizationcited by 189
  17. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  18. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  19. 2022Patching open-vocabulary models by interpolating weightscited by 187
  20. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  21. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  22. 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 175
  23. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  24. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 160
  25. 2024The Road Less Scheduledcited by 158
  26. 2021Rethinking Rotated Object Detection with Gaussian Wasserstein Distance Losscited by 156
  27. 2021Well-tuned Simple Nets Excel on Tabular Datasetscited by 151
  28. 2021Self-supervised Augmentation Consistency for Adapting Semantic Segmentationcited by 149
  29. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  30. 2022Scene Text Recognition with Permuted Autoregressive Sequence Modelscited by 148
  31. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  32. 2021Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalizationcited by 144
  33. 2022More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsitycited by 142
  34. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  35. 2023Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Maskscited by 130
  36. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  37. 2025EmbeddingGemma: Powerful and Lightweight Text Representationscited by 128
  38. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 119

50 most cited of 1,194 All 1,194 (JSON)