mlsanityData to 7 Oct 2026

Switch EMA: A Free Lunch for Better Flatness and Sharpness

arXiv 2402.09240 ↗First posted 14 Feb 2024cs.LG cs.CVcited by 9

Not on the map. Why?

Abstract (arXiv)

Exponential Moving Average (EMA) is a widely used weight averaging (WA) regularization to learn flat optima for better generalizations without extra cost in deep neural network (DNN) optimization. Despite achieving better flatness, existing WA methods might fall into worse final performances or require extra test-time computations. This work unveils the full potential of EMA with a single line of modification, i.e., switching the EMA parameters to the original model after each epoch, dubbed as Switch EMA (SEMA). From both theoretical and empirical aspects, we demonstrate that SEMA can help DNNs to reach generalization optima that better trade-off between flatness and sharpness. To verify the effectiveness of SEMA, we conduct comparison experiments with discriminative, generative, and regression tasks on vision and language datasets, including image classification, self-supervised learning, object detection and segmentation, image generation, video prediction, attribute regression, and language modeling. Comprehensive results with popular optimizers and networks show that SEMA is a free lunch for DNN training by improving performances and boosting convergence speeds.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 82 of its 119 references that are in the map. First posted on arXiv 14 Feb 2024.

Builds on

82 of 119 references.

References · 82

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2017Mask R-CNNcited by 7,259
38 more
  1. 2016Densely Connected Convolutional Networkscited by 6,691
  2. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2014Going Deeper with Convolutionscited by 4,231
  7. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  8. 2022A ConvNet for the 2020scited by 3,545
  9. 2016Wide Residual Networkscited by 3,057
  10. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  11. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  12. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  13. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  14. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  15. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  16. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  17. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  18. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  19. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  20. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  21. 2015Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcastingcited by 1,428
  22. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  23. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,379
  24. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  25. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,265
  26. 2017Random Erasing Data Augmentationcited by 1,235
  27. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  28. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,147
  29. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  30. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  31. 2021YOLOX: Exceeding YOLO Series in 2021cited by 1,064
  32. 2016Deep Networks with Stochastic Depthcited by 973
  33. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 763
  34. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  35. 2021MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformercited by 636
  36. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  37. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  38. 2021MetaFormer Is Actually What You Need for Visioncited by 523

50 most cited of 82 All 82 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 6; 2026: 2.

9 papers.

  1. 2024Retrieval Meets Reasoning: Even High-school Textbook Knowledge Benefits Multimodal Reasoningcited by 13
  2. 2026Anytime Pretraining: Horizon-Free Learning-Rate Schedules with Weight Averagingcited by 8
  3. 2025When, Where and Why to Average Weights?cited by 6
  4. 2025Modality-Balanced Collaborative Distillation for Multi-Modal Domain Generalizationcited by 6
  5. 2025DUNIA: Pixel-Sized Embeddings via Cross-Modal Alignment for Earth Observation Applicationscited by 4
  6. 2025Understanding Flatness in Generative Models: Its Role and Benefitscited by 3
  7. 2025PANDORA: Diffusion Policy Learning for Dexterous Robotic Piano Playingcited by 2
  8. 2025EMA Without the Lag: Bias-Corrected Iterate Averaging Schemescited by 2
  9. 2026C-voting: Confidence-Based Test-Time Voting without Explicit Energy Functionscited by 1