mlsanityData to 14 Oct 2026

When, Where and Why to Average Weights?

arXiv 2502.06761 ↗First posted 10 Feb 2025cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

Averaging checkpoints along the training trajectory is a simple yet powerful approach to improve the generalization performance of Machine Learning models and reduce training time. Motivated by these potential gains, and in an effort to fairly and thoroughly benchmark this technique, we present an extensive evaluation of averaging techniques in modern Deep Learning, which we perform using AlgoPerf citepdahlbenchmarking2023, a large-scale benchmark for optimization algorithms. We investigate whether weight averaging can reduce training time, improve generalization, and replace learning rate decay, as suggested by recent literature. Our evaluation across seven architectures and datasets reveals that averaging significantly accelerates training and yields considerable efficiency gains, at the price of a minimal implementation and memory cost, while mildly improving generalization across all considered workloads. Finally, we explore the relationship between averaging and learning rate annealing and show how to optimally combine the two to achieve the best performances.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 34 of its 66 references that are in the map. First posted on arXiv 10 Feb 2025.

Builds on

34 of 66 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  7. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  9. 2020GLU Variants Improve Transformercited by 1,804
  10. 2019Root Mean Square Layer Normalizationcited by 1,480
  11. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  12. 2018Relational inductive biases, deep learning, and graph networkscited by 1,360
22 more
  1. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  2. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  3. 2018fastMRI: An Open Dataset and Benchmarks for Accelerated MRIcited by 470
  4. 2015Deep Speech 2: End-to-End Speech Recognition in English and Mandarincited by 451
  5. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 434
  6. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  7. 2019Deep Learning Recommendation Model for Personalization and Recommendation Systemscited by 327
  8. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  9. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  10. 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 163
  11. 2021Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalizationcited by 144
  12. 2013Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)cited by 111
  13. 2018There Are Many Consistent Explanations of Unlabeled Data: Why You Should Averagecited by 82
  14. 2023A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scalecited by 77
  15. 2018The Unusual Effectiveness of Averaging in GAN Trainingcited by 73
  16. 2020Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Wellcited by 57
  17. 2019SWALP : Stochastic Weight Averaging in Low-Precision Trainingcited by 42
  18. 2023Early Weight Averaging meets High Learning Rates for LLM Pre-trainingcited by 37
  19. 2018Iterate averaging as regularization for stochastic gradient descentcited by 27
  20. 2023Training trajectories, mini-batch losses and the curious role of the learning ratecited by 21
  21. 2024Switch EMA: A Free Lunch for Better Flatness and Sharpnesscited by 9
  22. 2022Fast Benchmarking of Accuracy vs. Training Time with Cyclic Learning Ratescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2025Training Dynamics Impact Post-Training Quantization Robustnesscited by 15
  2. 2026Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averagingcited by 10
  3. 2025Modality-Balanced Collaborative Distillation for Multi-Modal Domain Generalizationcited by 6
  4. 2026A Unified Generalization Framework for Model Merging: Trade-offs, Non-Linearity, and Scaling Lawscited by 0
  5. 2026Terminal Shrinkage Averaging Reveals a Schedule-Estimator Interaction in LLM Pretrainingcited by 0
  6. 2026QAM: Quadratic-Accurate Checkpoint Merging via Sequential Consistencycited by 0
  7. 2026Conflicting Supervision Moves Commitment, Not Capability: A 12.29σ arrangement effect that is exactly zero under a convention-agnostic scorecited by 0
  8. 2026Understanding the Weight Averaging Mechanism in LLM Training for Post-Training Quantizationcited by 0