mlsanityData to 13 Oct 2026

Exponential Moving Average of Weights in Deep Learning: Dynamics and Benefits

arXiv 2411.18704 ↗First posted 27 Nov 2024cs.LGcited by 69

Not on the map. Why?

Abstract (arXiv)

Weight averaging of Stochastic Gradient Descent (SGD) iterates is a popular method for training deep learning models. While it is often used as part of complex training pipelines to improve generalization or serve as a `teacher' model, weight averaging lacks proper evaluation on its own. In this work, we present a systematic study of the Exponential Moving Average (EMA) of weights. We first explore the training dynamics of EMA, give guidelines for hyperparameter tuning, and highlight its good early performance, partly explaining its success as a teacher. We also observe that EMA requires less learning rate decay compared to SGD since averaging naturally reduces noise, introducing a form of implicit regularization. Through extensive experiments, we show that EMA solutions differ from last-iterate solutions. EMA models not only generalize better but also exhibit improved i) robustness to noisy labels, ii) prediction consistency, iii) calibration and iv) transfer learning. Therefore, we suggest that an EMA of weights is a simple yet effective plug-in to improve the performance of deep learning models.

Key facts. Cited by 69 papers in the map (data through 13 Oct 2026). Builds on 45 of its 59 references that are in the map. First posted on arXiv 27 Nov 2024.

Builds on

45 of 59 references.

References · 45

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,950
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,321
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,009
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,492
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,915
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,021
  7. 2017On Calibration of Modern Neural Networkscited by 4,902
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,730
  9. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,902
  10. 2016Wide Residual Networkscited by 3,063
  11. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,082
  12. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,391
33 more
  1. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,344
  2. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,202
  3. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,055
  4. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 747
  5. 2020DivideMix: Learning with Noisy Labels as Semi-supervised Learningcited by 692
  6. 2022Continual Test-Time Domain Adaptationcited by 523
  7. 2019Measuring Calibration in Deep Learningcited by 443
  8. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  9. 2020Early-Learning Regularization Prevents Memorization of Noisy Labelscited by 415
  10. 2021DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentationcited by 299
  11. 2020Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examplescited by 277
  12. 2021Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotationscited by 255
  13. 2017Parallel WaveNet: Fast High-Fidelity Speech Synthesiscited by 215
  14. 2021Data Augmentation Can Improve Robustnesscited by 193
  15. 2019SELF: Learning to Filter Noisy Labels with Self-Ensemblingcited by 164
  16. 2021Assessing Generalization of SGD via Disagreementcited by 116
  17. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 109
  18. 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
  19. 2021Soft Calibration Objectives for Neural Networkscited by 90
  20. 2018There Are Many Consistent Explanations of Unlabeled Data: Why You Should Averagecited by 82
  21. 2021Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learningcited by 78
  22. 2018The Unusual Effectiveness of Averaging in GAN Trainingcited by 73
  23. 2016Harder, Better, Faster, Stronger Convergence Rates for Least-Squares Regressioncited by 64
  24. 2020Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Wellcited by 57
  25. 2022Stop Wasting My Time! Saving Days of ImageNet and BERT Training with Latest Weight Averagingcited by 54
  26. 2019SWALP : Stochastic Weight Averaging in Low-Precision Trainingcited by 41
  27. 2023Early Weight Averaging meets High Learning Rates for LLM Pre-trainingcited by 37
  28. 2019On Model Stability as a Function of Random Seedcited by 31
  29. 2021On the Reproducibility of Neural Network Predictionscited by 28
  30. 2018Iterate averaging as regularization for stochastic gradient descentcited by 27
  31. 2019Beating SGD Saturation with Tail-Averaging and Minibatchingcited by 21
  32. 2023Training trajectories, mini-batch losses and the curious role of the learning ratecited by 21
  33. 2021Churn Reduction via Distillationcited by 17

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 1; 2025: 22; 2026: 46.

69 papers.

Sort:

By area:

  1. 2025DiWA: Diffusion Policy Adaptation with World Modelscited by 29
  2. 2026Self-Supervised Flow Matching for Scalable Multi-Modal Synthesiscited by 28
  3. 2025A Stable Whitening Optimizer for Efficient Neural Network Trainingcited by 15
  4. 2025Rethinking Safety in LLM Fine-tuning: An Optimization Perspectivecited by 14
  5. 2026Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averagingcited by 9
  6. 2025FOCUS: First Order Concentrated Updating Schemecited by 8
  7. 2025SNOO: Step-K Nesterov Outer Optimizer - The Surprising Effectiveness of Nesterov Momentum Applied to Pseudo-Gradientscited by 7
  8. 2025Emergent Abilities of Large Language Models under Continued Pretraining for Language Adaptationcited by 5
  9. 2025Smoothing DiLoCo with Primal Averaging for Faster Training of LLMscited by 5
  10. 2026Multi-Domain Riemannian Graph Gluing for Building Graph Foundation Modelscited by 4
  11. 2026LPLCv2: An Expanded Dataset for Fine-Grained License Plate Legibility Classificationcited by 4
  12. 2025When, What, and How: Rethinking Retrieval-Enhanced Speculative Decodingcited by 3
38 more
  1. 2026Benchmarking Optimizers for MLPs in Tabular Deep Learningcited by 3
  2. 2025PANDORA: Diffusion Policy Learning for Dexterous Robotic Piano Playingcited by 2
  3. 2025Variational Online Mirror Descent for Robust Learning in Schr"odinger Bridgecited by 2
  4. 2025MAESTRO: Masked AutoEncoders for Multimodal, Multitemporal, and Multispectral Earth Observation Datacited by 2
  5. 2026Pay for Hints, Not Answers: LLM Shepherding for Cost-Efficient Inferencecited by 2
  6. 2026Do Diffusion Models Dream of Electric Planes? Discrete and Continuous Simulation-Based Inference for Aircraft Designcited by 2
  7. 2026Dreaming the Unseen: World Model-regularized Diffusion Policy for Out-of-Distribution Robustnesscited by 2
  8. 2025Accelerating Learned Image Compression Through Modeling Neural Training Dynamicscited by 1
  9. 2025PADAM: Parallel averaged Adam reduces the error for stochastic optimization in scientific machine learningcited by 1
  10. 2025Two-stage Vision Transformers and Hard Masking offer Robust Object Representationscited by 1
  11. 2025FairFedMed: Benchmarking Group Fairness in Federated Medical Imaging with FairLoRAcited by 1
  12. 2025Combining Textual and Structural Information for Premise Selection in Leancited by 1
  13. 2025Adaptive Agents in Spatial Double-Auction Markets: Modeling the Emergence of Industrial Symbiosiscited by 1
  14. 2026Leveraging Second-Order Curvature for Efficient Learned Image Compression: Theory and Empirical Evidencecited by 1
  15. 2026Merging Beyond: Streaming LLM Updates via Activation-Guided Rotationscited by 1
  16. 2026Uni-LVC: A Unified Method for Intra- and Inter-Mode Learned Video Compressioncited by 1
  17. 2026Model-Agnostic Meta Learning for Class Imbalance Adaptationcited by 1
  18. 2026mathsfVISTA: Decentralized Machine Learning in Adversary Dominated Environmentscited by 1
  19. 2026Refined Analysis of Entropy-Regularized Actor-Criticcited by 1
  20. 2026EMAgnet: Parameter-Space EMA Regularization for Policy Gradient Self-Play in Large Gamescited by 1
  21. 2026Optimizing Visual Generative Models via Distribution-wise Rewardscited by 1
  22. 2024Learning Locally, Revising Globally: Global Reviser for Federated Learning with Noisy Labelscited by 0
  23. 2025DMAF-Net: An Effective Modality Rebalancing Framework for Incomplete Multi-Modal Medical Image Segmentationcited by 0
  24. 2025Classification based deep learning models for lung cancer and disease using medical imagescited by 0
  25. 2025DALI-PD: Diffusion-based Synthetic Layout Heatmap Generation for ML in Physical Designcited by 0
  26. 2025ReNF: Rethinking the Design of Neural Long-Term Time Series Forecasterscited by 0
  27. 2025LayerPipe2: Multistage Pipelining and Weight Recompute via Improved Exponential Moving Average for Training Neural Networkscited by 0
  28. 2026SUG-Occ: Explicit Semantics and Uncertainty Guided Sparse Learning for Efficient 3D Occupancy Predictioncited by 0
  29. 2026A Unified Generalization Framework for Model Merging: Trade-offs, Non-Linearity, and Scaling Lawscited by 0
  30. 2026Taipan: A Query-free Transfer-based Multiple Sensitive Attribute Inference Attack Solely from Auxiliary Graphscited by 0
  31. 2026An Adaptive Differentially Private Federated Learning Frameworkcited by 0
  32. 2026From Misclassifications to Outliers: Joint Reliability Assessment in Classificationcited by 0
  33. 2026Learning through Creation: A Hash-Free Framework for On-the-Fly Category Discoverycited by 0
  34. 2026Exclusivity-Guided Mask Learning for Semi-Supervised Crowd Instance Segmentation and Countingcited by 0
  35. 2026VISTA: Validation-Informed Trajectory Adaptation via Self-Distillationcited by 0
  36. 2026Improving Generative Adversarial Networks with Self-Distillationcited by 0
  37. 2026A Unified Graph Language Model for Multi-Domain Multi-Task Graph Alignment Instruction Tuningcited by 0
  38. 2026PEIRA: Learning Predictive Encoders through Inter-View Regressor Alignmentcited by 0

50 most cited of 69 All 69 (JSON)