# Why is SAM Robust to Label Noise?

arXiv: https://arxiv.org/abs/2405.03676 · HTML: https://mlsanity.com/paper/2405.03676/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 45 references that are in the map.
- First posted on arXiv 6 May 2024.

## Builds on (most cited first, 30 of 38)

- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [Deep Double Descent: Where Bigger Models and More Data Hurt](https://mlsanity.com/paper/1912.02292/) (2019)
- [Exploring Generalization in Deep Learning](https://mlsanity.com/paper/1706.08947/) (2017)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)
- [Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data](https://mlsanity.com/paper/1703.11008/) (2017)
- [Fantastic Generalization Measures and Where to Find Them](https://mlsanity.com/paper/1912.02178/) (2019)
- [Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability](https://mlsanity.com/paper/2103.00065/) (2021)
- [Early-Learning Regularization Prevents Memorization of Noisy Labels](https://mlsanity.com/paper/2007.00151/) (2020)
- [Unsupervised Label Noise Modeling and Loss Correction](https://mlsanity.com/paper/1904.11238/) (2019)
- [Symbolic Knowledge Distillation: from General Language Models to Commonsense Models](https://mlsanity.com/paper/2110.07178/) (2021)
- [High-dimensional dynamics of generalization error in neural networks](https://mlsanity.com/paper/1710.03667/) (2017)
- [Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels](https://mlsanity.com/paper/1905.05040/) (2019)
- [Deep Learning is Robust to Massive Label Noise](https://mlsanity.com/paper/1705.10694/) (2017)
- [Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability](https://mlsanity.com/paper/2209.15594/) (2022)
- [Towards Understanding Sharpness-Aware Minimization](https://mlsanity.com/paper/2206.06232/) (2022)
- [Label Noise SGD Provably Prefers Flat Global Minimizers](https://mlsanity.com/paper/2106.06530/) (2021)
- [Characterizing Structural Regularities of Labeled Data in Overparameterized Models](https://mlsanity.com/paper/2002.03206/) (2020)
- [Robust Training under Label Noise by Over-parameterization](https://mlsanity.com/paper/2202.14026/) (2022)
- [A Modern Look at the Relationship between Sharpness and Generalization](https://mlsanity.com/paper/2302.07011/) (2023)
- [Regularizing Neural Networks via Adversarial Model Perturbation](https://mlsanity.com/paper/2010.04925/) (2020)
- [The Implicit and Explicit Regularization Effects of Dropout](https://mlsanity.com/paper/2002.12915/) (2020)
- [Robust Inference via Generative Classifiers for Handling Noisy Labels](https://mlsanity.com/paper/1901.11300/) (2019)
- [How Does Sharpness-Aware Minimization Minimize Sharpness?](https://mlsanity.com/paper/2211.05729/) (2022)
- [Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization](https://mlsanity.com/paper/2012.14193/) (2020)
- [Distribution Density, Tails, and Outliers in Machine Learning: Metrics and Applications](https://mlsanity.com/paper/1910.13427/) (2019)
- [Anticorrelated Noise Injection for Improved Generalization](https://mlsanity.com/paper/2202.02831/) (2022)

## Built on by (most cited first, 21 of 21)

- [Tilted Sharpness-Aware Minimization](https://mlsanity.com/paper/2410.22656/) (2024)
- [An Analysis of Concept Bottleneck Models: Measuring, Understanding, and Mitigating the Impact of Noisy Annotations](https://mlsanity.com/paper/2505.16705/) (2025)
- [SAFE: Finding Sparse and Flat Minima to Improve Pruning](https://mlsanity.com/paper/2506.06866/) (2025)
- [Enhancing Sample Selection Against Label Noise by Cutting Mislabeled Easy Examples](https://mlsanity.com/paper/2502.08227/) (2025)
- [Towards Understanding the Role of Sharpness-Aware Minimization Algorithms for Out-of-Distribution Generalization](https://mlsanity.com/paper/2412.05169/) (2024)
- [SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation](https://mlsanity.com/paper/2502.18153/) (2025)
- [Enhancing LLM Knowledge Learning through Generalization](https://mlsanity.com/paper/2503.03705/) (2025)
- [Unveiling the Role of Data Uncertainty in Tabular Deep Learning](https://mlsanity.com/paper/2509.04430/) (2025)
- [Model Merging via Multi-Teacher Knowledge Distillation](https://mlsanity.com/paper/2512.21288/) (2025)
- [SHAPO: Sharpness-Aware Policy Optimization for Safe Exploration](https://mlsanity.com/paper/2606.10228/) (2026)
- [Noisy Ostracods: A Fine-Grained, Imbalanced Real-World Dataset for Benchmarking Robust Machine Learning and Label Correction Methods](https://mlsanity.com/paper/2412.02313/) (2024)
- [Flatness-aware Curriculum Learning via Adversarial Difficulty](https://mlsanity.com/paper/2508.18726/) (2025)
- [Sharpness-Aware Minimization in Logit Space Efficiently Enhances Direct Preference Optimization](https://mlsanity.com/paper/2603.18258/) (2026)
- [Forget Sharpness: Perturbed Forgetting of Model Biases Within SAM Dynamics](https://mlsanity.com/paper/2406.06700/) (2024)
- [Understanding SAM's Robustness to Noisy Labels through Gradient Down-weighting](https://mlsanity.com/paper/2411.17132/) (2024)
- [Unpacking the Implicit Norm Dynamics of Sharpness-Aware Minimization in Tensorized Models](https://mlsanity.com/paper/2508.10435/) (2025)
- [Flatness-Aware Stochastic Gradient Langevin Dynamics](https://mlsanity.com/paper/2510.02174/) (2025)
- [Zeroth-Order Sharpness-Aware Learning with Exponential Tilting](https://mlsanity.com/paper/2510.16157/) (2025)
- [SAMOSA: Sharpness Aware Minimization for Open Set Active learning](https://mlsanity.com/paper/2510.16757/) (2025)
- [Incorruptible Neural Networks: Training Models that can Generalize to Large Internal Perturbations](https://mlsanity.com/paper/2602.07320/) (2026)
- [Certification of Machine Learning Models via Directional Sharpness](https://mlsanity.com/paper/2606.25004/) (2026)

## Abstract (arXiv)

Sharpness-Aware Minimization (SAM) is most known for achieving state-of the-art performances on natural image and language tasks. However, its most pronounced improvements (of tens of percent) is rather in the presence of label noise. Understanding SAM's label noise robustness requires a departure from characterizing the robustness of minimas lying in "flatter" regions of the loss landscape. In particular, the peak performance under label noise occurs with early stopping, far before the loss converges. We decompose SAM's robustness into two effects: one induced by changes to the logit term and the other induced by changes to the network Jacobian. The first can be observed in linear logistic regression where SAM provably up-weights the gradient contribution from clean examples. Although this explicit up-weighting is also observable in neural networks, when we intervene and modify SAM to remove this effect, surprisingly, we see no visible degradation in performance. We infer that SAM's effect in deeper networks is instead explained entirely by the effect SAM has on the network Jacobian. We theoretically derive the implicit regularization induced by this Jacobian effect in two layer linear networks. Motivated by our analysis, we see that cheaper alternatives to SAM that explicitly induce these regularization effects largely recover the benefits in deep networks trained on real-world datasets.