mlsanityData to 14 Oct 2026

Early Weight Averaging meets High Learning Rates for LLM Pre-training

arXiv 2306.03241 ↗First posted 5 Jun 2023cs.LG cs.AI cs.CLcited by 37

Not on the map. Why?

Abstract (arXiv)

Training Large Language Models (LLMs) incurs significant cost; hence, any strategy that accelerates model convergence is helpful. In this paper, we investigate the ability of a simple idea checkpoint averaging along the trajectory of a training run to improve both convergence and generalization quite early on during training. Here we show that models trained with high learning rates observe higher gains due to checkpoint averaging. Furthermore, these gains are amplified when checkpoints are sampled with considerable spacing in training steps. Our training recipe outperforms conventional training and popular checkpoint averaging baselines such as exponential moving average (EMA) and stochastic moving average (SWA). We evaluate our training recipe by pre-training LLMs, where high learning rates are inherently preferred due to extremely large batch sizes. Specifically, we pre-trained nanoGPT-2 models of varying sizes, small (125M), medium (335M), and large (770M)on the OpenWebText dataset, comprised of 9B tokens. Additionally, we present results for publicly available Pythia LLMs, ranging from 1B to 12B, which were trained on the PILE-deduped dataset containing 207B tokens.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Builds on 38 of its 44 references that are in the map. First posted on arXiv 5 Jun 2023.

Builds on

38 of 44 references.

References · 38

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  10. 2016Pointer Sentinel Mixture Modelscited by 3,096
  11. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  12. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
26 more
  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  2. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  3. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  4. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  6. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  7. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  8. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 434
  9. 2023Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-trainingcited by 265
  10. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  11. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  12. 2023Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Clustercited by 109
  13. 2022On the SDEs and Scaling Rules for Adaptive Gradient Algorithmscited by 104
  14. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  15. 2018There Are Many Consistent Explanations of Unlabeled Data: Why You Should Averagecited by 82
  16. 2022Training Trajectories of Language Models Across Scalescited by 81
  17. 2023Stable and low-precision training for large-scale vision-language modelscited by 75
  18. 2020A Mathematical Exploration of Why Language Models Help Solve Downstream Taskscited by 72
  19. 2020Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformerscited by 67
  20. 2022On the Maximum Hessian Eigenvalue and Generalizationcited by 57
  21. 2023PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computingcited by 56
  22. 2022Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averagingcited by 54
  23. 2019SWALP : Stochastic Weight Averaging in Low-Precision Trainingcited by 42
  24. 2018Iterate averaging as regularization for stochastic gradient descentcited by 27
  25. 2023PopulAtion Parameter Averaging (PAPA)cited by 23
  26. 2023Training trajectories, mini-batch losses and the curious role of the learning ratecited by 21

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 10; 2025: 11; 2026: 12.

37 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 254
  2. 2023Mitigating the Alignment Tax of RLHFcited by 183
  3. 2024Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durationscited by 170
  4. 2024The Road Less Scheduledcited by 163
  5. 2024Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefitscited by 71
  6. 2023Learning from models beyond fine-tuningcited by 48
  7. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  8. 2025Model Merging in Pre-training of Large Language Modelscited by 37
  9. 2025Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMscited by 35
  10. 2024Checkpoint Merging via Bayesian Optimization in LLM Pretrainingcited by 27
  11. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  12. 2025WSM: Decay-Free Learning Rate Schedule via Checkpoint Merging for LLM Pre-trainingcited by 23
25 more
  1. 2023Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning and Autoregressioncited by 14
  2. 2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspectivecited by 14
  3. 2025How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretrainingcited by 14
  4. 2026Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuningcited by 13
  5. 2024When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Modelscited by 12
  6. 2024LiLiuM: eBay's Large Language Models for e-commercecited by 10
  7. 2026Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averagingcited by 10
  8. 2025Subspace-Boosted Model Mergingcited by 9
  9. 2024Post-Hoc Reversal: Are We Selecting Models Prematurely?cited by 8
  10. 2025When, Where and Why to Average Weights?cited by 8
  11. 2025Hala Technical Report: Building Arabic-Centric Instruction & Translation Models at Scalecited by 5
  12. 2026Low-rank Optimization Trajectories Modeling for LLM RLVR Accelerationcited by 4
  13. 2024Learning Scalable Model Soup on a Single GPU: An Efficient Subspace Training Strategycited by 3
  14. 2025GTR-Turbo: Merged Checkpoint is Secretly a Free Teacher for Agentic VLM Trainingcited by 3
  15. 2026Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Trainingcited by 2
  16. 2025Task Vector Bases: A Unified and Scalable Framework for Compressed Task Arithmeticcited by 1
  17. 2025MaP: A Unified Framework for Reliable Evaluation of Pre-training Dynamicscited by 1
  18. 2026ORBIT: Preserving Foundational Language Capabilities in GenRetrieval via Origin-Regulated Mergingcited by 1
  19. 2026Optimizing Visual Generative Models via Distribution-wise Rewardscited by 1
  20. 2026Speedrunning Tabular Foundation Model Pretrainingcited by 0
  21. 2026WSqD: A Horizon-Free Learning Rate Schedule for Large Model Trainingcited by 0
  22. 2026Terminal Shrinkage Averaging Reveals a Schedule-Estimator Interaction in LLM Pretrainingcited by 0
  23. 2026QAM: Quadratic-Accurate Checkpoint Merging via Sequential Consistencycited by 0
  24. 2026Trajectory Soup: Pushing the Compute-Scaling Frontier of LLM Mid-training via Diverse Trajectoriescited by 0
  25. 2026Understanding the Weight Averaging Mechanism in LLM Training for Post-Training Quantizationcited by 0