mlsanityData to 7 Oct 2026

A simple connection from loss flatness to compressed neural representations

arXiv 2310.01770 ↗First posted 3 Oct 2023cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Despite extensive study, the significance of sharpness -- the trace of the loss Hessian at local minima -- remains unclear. We investigate an alternative perspective: how sharpness relates to the geometric structure of neural representations, specifically representation compression, defined as how strongly neural activations concentrate under local input perturbations. We introduce three measures -- Local Volumetric Ratio (LVR), Maximum Local Sensitivity (MLS), and Local Dimensionality -- and derive upper bounds showing these are mathematically constrained by sharpness: flatter minima necessarily limit compression. We extend these bounds to reparametrization-invariant sharpness and introduce network-wide variants (NMLS, NVR) that provide tighter, more stable bounds than prior single-layer analyses. Empirically, we validate consistent positive correlations across feedforward, convolutional, and transformer architectures. Our results suggest that sharpness fundamentally quantifies representation compression, offering a principled resolution to contradictory findings on the sharpness-generalization relationship.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 36 of its 55 references that are in the map. First posted on arXiv 3 Oct 2023.

Builds on

36 of 55 references.

References · 36

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  4. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  5. 2021Robust fine-tuning of zero-shot modelscited by 785
  6. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  7. 2017Exploring Generalization in Deep Learningcited by 595
  8. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  9. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  10. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  11. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  12. 2019Intrinsic dimension of data representations in deep neural networkscited by 276
24 more
  1. 2017Three Factors Influencing Minima in SGDcited by 270
  2. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  3. 2023FastViT: A Fast Hybrid Vision Transformer using Structural Reparameterizationcited by 161
  4. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  5. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  6. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  7. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  8. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  9. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 91
  10. 2020Relative Flatness and Generalizationcited by 88
  11. 2019Convergence rates for the stochastic gradient descent method for non-convex objective functionscited by 66
  12. 2022The alignment property of SGD noise and how it helps select flat minima: A stability analysiscited by 64
  13. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 64
  14. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 61
  15. 2019Normalized Flat Minima: Exploring Scale Invariant Definition of Flat Minima for Neural Networks using PAC-Bayesian Analysiscited by 58
  16. 2022The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minimacited by 51
  17. 2018Analysis of the Generalization Error: Empirical Risk Minimization over Deep Artificial Neural Networks Overcomes the Curse of Dimensionality in the Numerical Approximation of Black-Scholes Partial Differential Equationscited by 44
  18. 2023Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalizationcited by 39
  19. 2018The loss landscape of overparameterized neural networkscited by 38
  20. 2019Dimensionality compression and expansion in Deep Neural Networkscited by 38
  21. 2021Taxonomizing local versus global structure in neural network loss landscapescited by 34
  22. 2022Stochastic gradient descent introduces an effective landscape-dependent regularization favoring flat solutionscited by 22
  23. 2020Flatness is a False Friendcited by 19
  24. 2015Stimuli reduce the dimensionality of cortical activitycited by 4

Built on by

Not cited yet.