mlsanityData to 7 Oct 2026

Why is SAM Robust to Label Noise?

arXiv 2405.03676 ↗First posted 6 May 2024cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

Sharpness-Aware Minimization (SAM) is most known for achieving state-of the-art performances on natural image and language tasks. However, its most pronounced improvements (of tens of percent) is rather in the presence of label noise. Understanding SAM's label noise robustness requires a departure from characterizing the robustness of minimas lying in "flatter" regions of the loss landscape. In particular, the peak performance under label noise occurs with early stopping, far before the loss converges. We decompose SAM's robustness into two effects: one induced by changes to the logit term and the other induced by changes to the network Jacobian. The first can be observed in linear logistic regression where SAM provably up-weights the gradient contribution from clean examples. Although this explicit up-weighting is also observable in neural networks, when we intervene and modify SAM to remove this effect, surprisingly, we see no visible degradation in performance. We infer that SAM's effect in deeper networks is instead explained entirely by the effect SAM has on the network Jacobian. We theoretically derive the implicit regularization induced by this Jacobian effect in two layer linear networks. Motivated by our analysis, we see that cheaper alternatives to SAM that explicitly induce these regularization effects largely recover the benefits in deep networks trained on real-world datasets.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 38 of its 45 references that are in the map. First posted on arXiv 6 May 2024.

Builds on

38 of 45 references.

References · 38

  1. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  2. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  4. 2018Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labelscited by 994
  5. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  6. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  7. 2017Exploring Generalization in Deep Learningcited by 595
  8. 2023Textbooks Are All You Needcited by 561
  9. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  10. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  11. 2020Early-Learning Regularization Prevents Memorization of Noisy Labelscited by 414
  12. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
26 more
  1. 2019Unsupervised Label Noise Modeling and Loss Correctioncited by 327
  2. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  3. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  4. 2019Understanding and Utilizing Deep Neural Networks Trained with Noisy Labelscited by 209
  5. 2017Deep Learning is Robust to Massive Label Noisecited by 164
  6. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  7. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  8. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  9. 2020Characterizing Structural Regularities of Labeled Data in Overparameterized Modelscited by 93
  10. 2022Robust Training under Label Noise by Over-parameterizationcited by 93
  11. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 91
  12. 2020Regularizing Neural Networks via Adversarial Model Perturbationcited by 78
  13. 2020The Implicit and Explicit Regularization Effects of Dropoutcited by 71
  14. 2019Robust Inference via Generative Classifiers for Handling Noisy Labelscited by 67
  15. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 64
  16. 2020Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalizationcited by 61
  17. 2019Distribution Density, Tails, and Outliers in Machine Learning: Metrics and Applicationscited by 43
  18. 2022Anticorrelated Noise Injection for Improved Generalizationcited by 41
  19. 2022Mitigating Memorization of Noisy Labels by Clipping the Model Predictioncited by 40
  20. 2023Sharpness-Aware Minimization Leads to Low-Rank Featurescited by 38
  21. 2023Why Does Sharpness-Aware Minimization Generalize Better Than SGD?cited by 27
  22. 2019Identifying Mislabeled Instances in Classification Datasetscited by 26
  23. 2018Step Size Matters in Deep Learningcited by 19
  24. 2023Per-Example Gradient Regularization Improves Learning Signals from Noisy Datacited by 7
  25. 2023On Statistical Properties of Sharpness-Aware Minimization: Provable Guaranteescited by 5
  26. 2023On Emergence of Clean-Priority Learning in Early Stopped Neural Networkscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 12; 2026: 4.

21 papers.

Sort:

By area:

  1. 2024Tilted Sharpness-Aware Minimizationcited by 6
  2. 2025An Analysis of Concept Bottleneck Models: Measuring, Understanding, and Mitigating the Impact of Noisy Annotationscited by 5
  3. 2025SAFE: Finding Sparse and Flat Minima to Improve Pruningcited by 5
  4. 2025Enhancing Sample Selection Against Label Noise by Cutting Mislabeled Easy Examplescited by 4
  5. 2025SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximationcited by 2
  6. 2025Enhancing LLM Knowledge Learning through Generalizationcited by 2
  7. 2025Model Merging via Multi-Teacher Knowledge Distillationcited by 2
  8. 2026SHAPO: Sharpness-Aware Policy Optimization for Safe Explorationcited by 2
  9. 2024Noisy Ostracods: A Fine-Grained, Imbalanced Real-World Dataset for Benchmarking Robust Machine Learning and Label Correction Methodscited by 1
  10. 2024Towards Understanding the Role of Sharpness-Aware Minimization Algorithms for Out-of-Distribution Generalizationcited by 1
  11. 2025Flatness-aware Curriculum Learning via Adversarial Difficultycited by 1
  12. 2025Unveiling the Role of Data Uncertainty in Tabular Deep Learningcited by 1
9 more
  1. 2026Sharpness-Aware Minimization in Logit Space Efficiently Enhances Direct Preference Optimizationcited by 1
  2. 2024Forget Sharpness: Perturbed Forgetting of Model Biases Within SAM Dynamicscited by 0
  3. 2024Understanding SAM's Robustness to Noisy Labels through Gradient Down-weightingcited by 0
  4. 2025Unpacking the Implicit Norm Dynamics of Sharpness-Aware Minimization in Tensorized Modelscited by 0
  5. 2025Flatness-Aware Stochastic Gradient Langevin Dynamicscited by 0
  6. 2025Zeroth-Order Sharpness-Aware Learning with Exponential Tiltingcited by 0
  7. 2025SAMOSA: Sharpness Aware Minimization for Open Set Active learningcited by 0
  8. 2026Incorruptible Neural Networks: Training Models that can Generalize to Large Internal Perturbationscited by 0
  9. 2026Certification of Machine Learning Models via Directional Sharpnesscited by 0