mlsanityData to 7 Oct 2026

Normalized Flat Minima: Exploring Scale Invariant Definition of Flat Minima for Neural Networks using PAC-Bayesian Analysis

arXiv 1901.04653 ↗First posted 15 Jan 2019stat.ML cs.LGcited by 58

Not on the map. Why?

Abstract (arXiv)

The notion of flat minima has played a key role in the generalization studies of deep learning models. However, existing definitions of the flatness are known to be sensitive to the rescaling of parameters. The issue suggests that the previous definitions of the flatness might not be a good measure of generalization, because generalization is invariant to such rescalings. In this paper, from the PAC-Bayesian perspective, we scrutinize the discussion concerning the flat minima and introduce the notion of normalized flat minima, which is free from the known scale dependence issues. Additionally, we highlight the scale dependence of existing matrix-norm based generalization error bounds similar to the existing flat minima definitions. Our modified notion of the flatness does not suffer from the insufficiency, either, suggesting it might provide better hierarchy in the hypothesis class.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). First posted on arXiv 15 Jan 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 21; 2023: 8; 2024: 8; 2025: 8; 2026: 8.

58 papers.

Sort:

By area:

  1. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  2. 2021User-friendly introduction to PAC-Bayes boundscited by 234
  3. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  4. 2022Sharpness-Aware Training for Freecited by 95
  5. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 91
  6. 2023Rethinking Model Ensemble in Transfer-based Adversarial Attackscited by 88
  7. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  8. 2022When Do Flat Minima Optimizers Work?cited by 78
  9. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  10. 2024Friendly Sharpness-Aware Minimizationcited by 52
  11. 2023The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descentcited by 44
  12. 2022Anticorrelated Noise Injection for Improved Generalizationcited by 41
38 more
  1. 2022Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guaranteescited by 34
  2. 2022PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Taskscited by 30
  3. 2022Explicit Regularization in Overparametrized Models via Noise Injectioncited by 25
  4. 2022FedCross: Towards Accurate Federated Learning via Multi-Model Cross-Aggregationcited by 21
  5. 2022On PAC-Bayesian reconstruction guarantees for VAEscited by 20
  6. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscapecited by 19
  7. 2023The Geometry of Neural Nets' Parameter Spaces Under Reparametrizationcited by 16
  8. 2023Decentralized SGD and Average-direction SAM are Asymptotically Equivalentcited by 15
  9. 2022On the Power-Law Hessian Spectrums in Deep Learningcited by 14
  10. 2024A Universal Class of Sharpness-Aware Minimization Algorithmscited by 14
  11. 2021Why flatness does and does not correlate with generalization for deep neural networkscited by 13
  12. 2023How to escape sharp minima with random perturbationscited by 13
  13. 2022Special Properties of Gradient Descent with Large Learning Ratescited by 10
  14. 2022Efficient Generalization Improvement Guided by Random Weight Perturbationcited by 8
  15. 2022Scale-invariant Bayesian Neural Networks with Connectivity Tangent Kernelcited by 7
  16. 2022PAC-Bayes Bounds for Bandit Problems: A Survey and Experimental Comparisoncited by 7
  17. 2021Exponential escape efficiency of SGD from sharp minima in non-stationary regimecited by 5
  18. 2022How You Start Matters for Generalizationcited by 5
  19. 2025From Global to Local: A Scalable Benchmark for Local Posterior Samplingcited by 5
  20. 2021Minimum sharpness: Scale-invariant parameter-robustness of neural networkscited by 4
  21. 2024A Rescaling-Invariant Lipschitz Bound Based on Path-Metrics for Modern ReLU Network Parameterizationscited by 4
  22. 2024The Uncanny Valley: Exploring Adversarial Robustness from a Flatness Perspectivecited by 4
  23. 2025Towards Robust Influence Functions with Flat Validation Minimacited by 4
  24. 2026The Effect of Mini-Batch Noise on the Implicit Bias of Adamcited by 4
  25. 2022Engineering flexible machine learning systems by traversing functionally-invariant pathscited by 3
  26. 2022Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rulescited by 3
  27. 2025When Flatness Does (Not) Guarantee Adversarial Robustnesscited by 3
  28. 2023Using Stochastic Gradient Descent to Smooth Nonconvex Functions: Analysis of Implicit Graduated Optimizationcited by 2
  29. 2024Unlocking the Potential of Text-to-Image Diffusion with PAC-Bayesian Theorycited by 2
  30. 2026On the Superlinear Relationship between SGD Noise Covariance and Loss Landscape Curvaturecited by 2
  31. 2026Too Sharp, Too Sure: When Calibration Follows Curvaturecited by 2
  32. 2022Deep learning, stochastic gradient descent and diffusion mapscited by 1
  33. 2022Attack-Agnostic Adversarial Detectioncited by 1
  34. 2024Dimension-free uniform concentration bound for logistic regressioncited by 1
  35. 2025Any-stepsize Gradient Descent for Separable Data under Fenchel-Young Lossescited by 1
  36. 2025R'enyi Sharpness: A Novel Sharpness that Strongly Correlates with Generalizationcited by 1
  37. 2025The Curvature Rate λ: A Scalar Measure of Input-Space Sharpness in Neural Networkscited by 1
  38. 2026Are Flat Minima an Illusion?cited by 1

50 most cited of 58 All 58 (JSON)