mlsanityData to 14 Oct 2026

Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averaging

arXiv 2209.14981 ↗First posted 29 Sep 2022cs.LG cs.AI stat.MLcited by 54

Not on the map. Why?

Abstract (arXiv)

Training vision or language models on large datasets can take days, if not weeks. We show that averaging the weights of the k latest checkpoints, each collected at the end of an epoch, can speed up the training progression in terms of loss and accuracy by dozens of epochs, corresponding to time savings up to ~68 and ~30 GPU hours when training a ResNet50 on ImageNet and RoBERTa-Base model on WikiText-103, respectively. We also provide the code and model checkpoint trajectory to reproduce the results and facilitate research on reusing historical weights for faster convergence.

Key facts. Cited by 54 papers in the map (data through 14 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 29 Sep 2022.

Builds on

32 of 42 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  7. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  8. 2014Going Deeper with Convolutionscited by 4,241
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2016Wide Residual Networkscited by 3,070
  11. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  12. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
20 more
  1. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  2. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  3. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  4. 2016HyperNetworkscited by 1,027
  5. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 339
  6. 2019Lookahead Optimizer: k steps forward, 1 step backcited by 336
  7. 2020Are we done with ImageNet?cited by 333
  8. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 328
  9. 2017Empirical Analysis of the Hessian of Over-Parametrized Neural Networkscited by 290
  10. 2020Long-Tailed Classification by Keeping the Good and Removing the Bad Momentum Causal Effectcited by 256
  11. 2022Prioritized Training on Points that are Learnable, Worth Learning, and Not Yet Learntcited by 236
  12. 2018Gradient Descent Happens in a Tiny Subspacecited by 217
  13. 2017Regularizing and Optimizing LSTM Language Modelscited by 205
  14. 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
  15. 2020The Early Phase of Neural Network Trainingcited by 115
  16. 2022Causal Machine Learning: A Survey and Open Problemscited by 110
  17. 2018Troubling Trends in Machine Learning Scholarshipcited by 108
  18. 2016Importance Sampling for Minibatchescited by 36
  19. 2018Iterate averaging as regularization for stochastic gradient descentcited by 27
  20. 2022Stochastic Weight Averaging Revisitedcited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 15; 2024: 17; 2025: 11; 2026: 8.

54 papers.

Sort:

By area:

  1. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 415
  2. 2023Challenges and Applications of Large Language Modelscited by 254
  3. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  4. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  5. 2024The Road Less Scheduledcited by 163
  6. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 121
  7. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  8. 2023Deep Model Fusion: A Surveycited by 91
  9. 2024Merging Multi-Task Models via Weight-Ensembling Mixture of Expertscited by 85
  10. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 75
  11. 2024Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefitscited by 71
  12. 2023Parameter Efficient Multi-task Model Fusion with Partial Linearizationcited by 65
38 more
  1. 2023Learning from models beyond fine-tuningcited by 48
  2. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 45
  3. 2023Spawrious: A Benchmark for Fine Control of Spurious Correlation Biasescited by 41
  4. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  5. 2023A Survey on Efficient Training of Transformerscited by 39
  6. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  7. 2023Early Weight Averaging meets High Learning Rates for LLM Pre-trainingcited by 37
  8. 2023Concrete Subspace Learning based Interference Elimination for Multi-task Model Fusioncited by 37
  9. 2025Model Merging in Pre-training of Large Language Modelscited by 37
  10. 2025Accelerating Neural Network Training: An Analysis of the AlgoPerf Competitioncited by 32
  11. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  12. 2024Asynchronous Local-SGD Training for Language Modelingcited by 26
  13. 2025FedSWA: Improving Generalization in Federated Learning with Highly Heterogeneous Data via Momentum-Based Stochastic Controlled Weight Averagingcited by 25
  14. 2024Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Mergingcited by 23
  15. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  16. 2025WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-trainingcited by 23
  17. 2023Training trajectories, mini-batch losses and the curious role of the learning ratecited by 21
  18. 2025Training Dynamics Impact Post-Training Quantization Robustnesscited by 15
  19. 2023Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregressioncited by 14
  20. 2024Mitigating the Backdoor Effect for Multi-Task Model Merging via Safety-Aware Subspacecited by 14
  21. 2025How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretrainingcited by 14
  22. 2022Weight Averaging: A Simple Yet Effective Method to Overcome Catastrophic Forgetting in Automatic Speech Recognitioncited by 10
  23. 2026Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averagingcited by 10
  24. 2024Switch EMA: A Free Lunch for Better Flatness and Sharpnesscited by 9
  25. 2024SMILE: Zero-Shot Sparse Mixture of Low-Rank Experts Construction From Pre-Trained Foundation Modelscited by 9
  26. 2023Window-based Model Averaging Improves Generalization in Heterogeneous Federated Learningcited by 7
  27. 2024MOS: Model Synergy for Test-Time Adaptation on LiDAR-Based 3D Object Detectioncited by 7
  28. 2024100K or 100 Days: Trade-offs when Pre-Training with Academic Resourcescited by 6
  29. 2024Learning Scalable Model Soup on a Single GPU: An Efficient Subspace Training Strategycited by 3
  30. 2024A Unified Analysis for Finite Weight Averagingcited by 3
  31. 2025EMA Without the Lag: Bias-Corrected Iterate Averaging Schemescited by 3
  32. 2026Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Trainingcited by 2
  33. 2024Weights Augmentation: it has never ever ever ever let her model downcited by 1
  34. 2025Task Vector Bases: A Unified and Scalable Framework for Compressed Task Arithmeticcited by 1
  35. 2025Stepsize anything: A unified learning rate schedule for budgeted-iteration trainingcited by 1
  36. 2025MaP: A Unified Framework for Reliable Evaluation of Pre-training Dynamicscited by 1
  37. 2026Merging Beyond: Streaming LLM Updates via Activation-Guided Rotationscited by 1
  38. 2026Training for the Model You Return: Improving Optimization for Iterate-Averaged Language Modelscited by 1

50 most cited of 54 All 54 (JSON)