mlsanityData to 14 Oct 2026

Relative Flatness and Generalization

arXiv 2001.00939 ↗First posted 3 Jan 2020cs.LG stat.MLcited by 88

Not on the map. Why?

Abstract (arXiv)

Flatness of the loss curve is conjectured to be connected to the generalization ability of machine learning models, in particular neural networks. While it has been empirically observed that flatness measures consistently correlate strongly with generalization, it is still an open theoretical problem why and under which circumstances flatness is connected to generalization, in particular in light of reparameterizations that change certain flatness measures but leave generalization unchanged. We investigate the connection between flatness and generalization by relating it to the interpolation from representative data, deriving notions of representativeness, and feature robustness. The notions allow us to rigorously connect flatness and generalization and to identify conditions under which the connection holds. Moreover, they give rise to a novel, but natural relative flatness measure that correlates strongly with generalization, simplifies to ridge regression for ordinary least squares, and solves the reparameterization issue.

Key facts. Cited by 88 papers in the map (data through 14 Oct 2026). First posted on arXiv 3 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 13; 2023: 23; 2024: 18; 2025: 17; 2026: 16.

88 papers.

Sort:

By area:

  1. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  2. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 92
  3. 2022When Do Flat Minima Optimizers Work?cited by 78
  4. 2023Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalizationcited by 57
  5. 2023The Implicit Regularization of Dynamical Stability in Stochastic Gradient Descentcited by 44
  6. 2023Out-of-distribution Detection with Implicit Outlier Transformationcited by 43
  7. 2023An Adaptive Policy to Employ Sharpness-Aware Minimizationcited by 43
  8. 2022Anticorrelated Noise Injection for Improved Generalizationcited by 42
  9. 2022Symmetries, flat minima, and the conserved quantities of gradient flowcited by 35
  10. 2023Normalization Layers Are All That Sharpness-Aware Minimization Needscited by 27
  11. 2023Flatness-Aware Minimization for Domain Generalizationcited by 27
  12. 2022Why Robust Generalization in Deep Learning is Difficult: Perspective of Expressive Powercited by 23
38 more
  1. 2023Improving Convergence and Generalization Using Parameter Symmetriescited by 23
  2. 2023Layer-wise Linear Mode Connectivitycited by 22
  3. 2023DART: Diversify-Aggregate-Repeat Training Improves Generalization of Neural Networkscited by 20
  4. 2022Bounding the Rademacher Complexity of Fourier neural operatorscited by 16
  5. 2023The Geometry of Neural Nets' Parameter Spaces Under Reparametrizationcited by 16
  6. 2022Beyond Losses Reweighting: Empowering Multi-Task Learning via the Generalization Perspectivecited by 15
  7. 2022Towards Lightweight Black-Box Attacks against Deep Neural Networkscited by 13
  8. 2024A PAC-Bayesian Link Between Generalisation and Flat Minimacited by 13
  9. 2024Towards Robust Out-of-Distribution Generalization Bounds via Sharpnesscited by 11
  10. 2023Towards fully covariant machine learningcited by 10
  11. 2023Flat Seeking Bayesian Neural Networkscited by 10
  12. 2021On the robustness of randomized classifiers to adversarial examplescited by 9
  13. 2023How Sparse Can We Prune A Deep Network: A Fundamental Limit Perspectivecited by 9
  14. 2024Frequency-Masked Embedding Inference: A Non-Contrastive Approach for Time Series Representation Learningcited by 9
  15. 2025Flatness is Necessary, Neural Collapse is Not: Rethinking Generalization via Grokkingcited by 8
  16. 2022Scale-invariant Bayesian Neural Networks with Connectivity Tangent Kernelcited by 7
  17. 2023On progressive sharpening, flat minima and generalisationcited by 7
  18. 2024Singular-limit analysis of gradient descent with noise injectioncited by 7
  19. 2026Position: Weight Space Should Be a First-Class Generative AI Modalitycited by 6
  20. 2023Optimal Transport Model Distributional Robustnesscited by 5
  21. 2023Unveiling the Hessian's Connection to the Decision Boundarycited by 5
  22. 2023Little is Enough: Boosting Privacy by Sharing Only Hard Labels in Federated Semi-Supervised Learningcited by 5
  23. 2024Improving Generalization with Flat Hilbert Bayesian Inferencecited by 5
  24. 2025Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds Itcited by 5
  25. 2024The Uncanny Valley: Exploring Adversarial Robustness from a Flatness Perspectivecited by 4
  26. 2024Slicing Mutual Information Generalization Bounds for Neural Networkscited by 4
  27. 2024Layerwise Change of Knowledge in Neural Networkscited by 4
  28. 2025Curvature Learning for Generalization of Hyperbolic Neural Networkscited by 4
  29. 2025Sharpness-Guided Group Relative Policy Optimization via Probability Shapingcited by 4
  30. 2026A Basin-Selection Perspective on Grokking via Singular Learning Theorycited by 4
  31. 2022Beyond accuracy: generalization properties of bio-plausible temporal credit assignment rulescited by 3
  32. 2024Face Reconstruction Transfer Attack as Out-of-Distribution Generalizationcited by 3
  33. 2025Layer-wise Adaptive Gradient Norm Penalizing Method for Efficient and Accurate Deep Learningcited by 3
  34. 2025Connecting Independently Trained Modes via Layer-Wise Connectivitycited by 3
  35. 2025Promoting Ensemble Diversity with Interactive Bayesian Distributional Robustness for Fine-tuning Foundation Modelscited by 3
  36. 2025When Flatness Does (Not) Guarantee Adversarial Robustnesscited by 3
  37. 2025Generalization Below the Edge of Stability: The Role of Data Geometrycited by 3
  38. 2022Manifold Characteristics That Predict Downstream Task Performancecited by 2

50 most cited of 88 All 88 (JSON)