mlsanityData to 7 Oct 2026

When Do Flat Minima Optimizers Work?

arXiv 2202.00661 ↗First posted 1 Feb 2022cs.LG stat.MLcited by 78

Not on the map. Why?

Abstract (arXiv)

Recently, flat-minima optimizers, which seek to find parameters in low-loss neighborhoods, have been shown to improve a neural network's generalization performance over stochastic and adaptive gradient-based optimizers. Two methods have received significant attention due to their scalability: 1. Stochastic Weight Averaging (SWA), and 2. Sharpness-Aware Minimization (SAM). However, there has been limited investigation into their properties and no systematic benchmarking of them across different domains. We fill this gap here by comparing the loss surfaces of the models trained with each method and through broad benchmarking across computer vision, natural language processing, and graph representation learning tasks. We discover several surprising findings from these results, which we hope will help researchers further improve deep learning optimizers, and practitioners identify the right optimizer for their problem.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Builds on 81 of its 118 references that are in the map. First posted on arXiv 1 Feb 2022.

Builds on

81 of 118 references.

References · 81

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2017Inductive Representation Learning on Large Graphscited by 6,284
  11. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  12. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
38 more
  1. 2018How Powerful are Graph Neural Networks?cited by 4,355
  2. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  3. 2016Wide Residual Networkscited by 3,057
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  5. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  6. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  7. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  8. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  9. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  10. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  11. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  12. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  13. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  14. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  15. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  16. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  17. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  18. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  19. 2016Complex Embeddings for Simple Link Predictioncited by 897
  20. 2021Going deeper with Image Transformerscited by 759
  21. 2021Characterizing possible failure modes in physics-informed neural networkscited by 632
  22. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  23. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  24. 2019Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networkscited by 540
  25. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  26. 2019Meta-Learning with Implicit Gradientscited by 523
  27. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  28. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  29. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  30. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 403
  31. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
  32. 2020What is being transferred in transfer learning?cited by 365
  33. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  34. 2017Snapshot Ensembles: Train 1, get M for freecited by 349
  35. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  36. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  37. 2019PyHessian: Neural Networks Through the Lens of the Hessiancited by 291
  38. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289

50 most cited of 81 All 81 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 27; 2024: 18; 2025: 19; 2026: 11.

78 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  3. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 121
  4. 2023The MiniPile Challenge for Data-Efficient Language Modelscited by 74
  5. 2022Mechanistic Mode Connectivitycited by 50
  6. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  7. 2023Spawrious: A Benchmark for Fine Control of Spurious Correlation Biasescited by 41
  8. 2023No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Modelscited by 41
  9. 2023Normalization Layers Are All That Sharpness-Aware Minimization Needscited by 27
  10. 2025FedSWA: Improving Generalization in Federated Learning with Highly Heterogeneous Data via Momentum-Based Stochastic Controlled Weight Averagingcited by 25
  11. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  12. 2023An SDE for Modeling SAM: Theory and Insightscited by 22
38 more
  1. 2025Explicit Eigenvalue Regularization Improves Sharpness-Aware Minimizationcited by 21
  2. 2023Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optimacited by 20
  3. 2023Adaptive Sharpness-Aware Pruning for Robust Sparse Networkscited by 17
  4. 2023Sharpness-Aware Minimization Revisited: Weighted Sharpness as a Regularization Termcited by 16
  5. 2025Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuningcited by 16
  6. 2023Stability Analysis of Sharpness-Aware Minimizationcited by 14
  7. 2023Differentially Private Sharpness-Aware Trainingcited by 14
  8. 2024A Universal Class of Sharpness-Aware Minimization Algorithmscited by 14
  9. 2023How to escape sharp minima with random perturbationscited by 13
  10. 2025TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillationcited by 12
  11. 2026Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuningcited by 12
  12. 2026HTMuon: Improving Muon via Heavy-Tailed Spectral Correctioncited by 11
  13. 2023Flat Seeking Bayesian Neural Networkscited by 10
  14. 2024Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapescited by 10
  15. 2022Improving Generalization of Pre-trained Language Models via Stochastic Weight Averagingcited by 9
  16. 2024Switch EMA: A Free Lunch for Better Flatness and Sharpnesscited by 9
  17. 2024Sharpness-Aware Minimization Enhances Feature Quality via Balanced Learningcited by 9
  18. 2024Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategycited by 8
  19. 2025Adapter Naturally Serves as Decoupler for Cross-Domain Few-Shot Semantic Segmentationcited by 7
  20. 2023TRAM: Bridging Trust Regions and Sharpness Aware Minimizationcited by 6
  21. 2024Mirror Gradient: Towards Robust Multimodal Recommender Systems via Exploring Flat Local Minimacited by 6
  22. 2025Zeroth-Order Optimization Finds Flat Minimacited by 6
  23. 2023On Statistical Properties of Sharpness-Aware Minimization: Provable Guaranteescited by 5
  24. 2023Optimal Transport Model Distributional Robustnesscited by 5
  25. 2024Model Fusion through Bayesian Optimization in Language Model Fine-Tuningcited by 5
  26. 2023Uncertainty-Aware Natural Language Inference with Stochastic Weight Averagingcited by 4
  27. 2023Exploring Flat Minima for Domain Generalization with Large Learning Ratescited by 4
  28. 2026Rethinking LoRA for Privacy-Preserving Federated Learning in Large Modelscited by 4
  29. 2023Gradient Ascent Post-training Enhances Language Model Generalizationcited by 3
  30. 2023Critical Influence of Overparameterization on Sharpness-aware Minimizationcited by 3
  31. 2024Sharpness-diversity tradeoff: improving flat ensembles with SharpBalancecited by 3
  32. 2025Flat Minima and Generalization: Insights from Stochastic Convex Optimizationcited by 3
  33. 2023Going Further: Flatness at the Rescue of Early Stopping for Adversarial Example Transferabilitycited by 2
  34. 2023The Split Matters: Flat Minima Methods for Improving the Performance of GNNscited by 2
  35. 2023FAM: Relative Flatness Aware Minimizationcited by 2
  36. 2025Towards Understanding The Calibration Benefits of Sharpness-Aware Minimizationcited by 2
  37. 2025Sparsity-Driven Plasticity in Multi-Task Reinforcement Learningcited by 2
  38. 2025Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noisecited by 2

50 most cited of 78 All 78 (JSON)