mlsanityData to 7 Oct 2026

The Inductive Bias of Flatness Regularization for Deep Matrix Factorization

arXiv 2306.13239 ↗First posted 22 Jun 2023cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Recent works on over-parameterized neural networks have shown that the stochasticity in optimizers has the implicit regularization effect of minimizing the sharpness of the loss function (in particular, the trace of its Hessian) over the family zero-loss solutions. More explicit forms of flatness regularization also empirically improve the generalization performance. However, it remains unclear why and when flatness regularization leads to better generalization. This work takes the first step toward understanding the inductive bias of the minimum trace of the Hessian solutions in an important setting: learning deep linear networks from linear measurements, also known as deep matrix factorization. We show that for all depth greater than one, with the standard Restricted Isometry Property (RIP) on the measurements, minimizing the trace of Hessian is approximately equivalent to minimizing the Schatten 1-norm of the corresponding end-to-end matrix parameters (i.e., the product of all layer matrices), which in turn leads to better generalization. We empirically verify our theoretical findings on synthetic datasets.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 41 of its 55 references that are in the map. First posted on arXiv 22 Jun 2023.

Builds on

41 of 55 references.

References · 41

  1. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  2. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  3. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  4. 2017Exploring Generalization in Deep Learningcited by 595
  5. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  6. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  7. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  8. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  9. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 413
  10. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  11. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  12. 2017Implicit Regularization in Matrix Factorizationcited by 318
29 more
  1. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  2. 2017Three Factors Influencing Minima in SGDcited by 270
  3. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  4. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  5. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  6. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  7. 2022Towards Efficient and Scalable Sharpness-Aware Minimizationcited by 129
  8. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  9. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  10. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  11. 2021Label Noise SGD Provably Prefers Flat Global Minimizerscited by 121
  12. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
  13. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 106
  14. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 91
  15. 2022Adaptive Gradient Methods at the Edge of Stabilitycited by 88
  16. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  17. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  18. 2020Regularizing Neural Networks via Adversarial Model Perturbationcited by 78
  19. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 77
  20. 2019Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentationcited by 71
  21. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  22. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 64
  23. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 61
  24. 2022Analyzing Sharpness along GD Trajectory: Progressive Sharpening and Edge of Stabilitycited by 61
  25. 2019Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margincited by 56
  26. 2021Implicit Regularization in Tensor Factorizationcited by 43
  27. 2022Flat minima generalize for low-rank matrix recoverycited by 28
  28. 2019Diametrical Risk Minimization: Theory and Computationscited by 15
  29. 2020On implicit regularization: Morse functions and applications to matrix factorizationcited by 11

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 4.

6 papers.

  1. 2023Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalizationcited by 39
  2. 2024Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Trainingcited by 24
  3. 2024Improving Generalization and Convergence by Enhancing Implicit Regularizationcited by 14
  4. 2023Achieving Margin Maximization Exponentially Fast via Progressive Norm Rescalingcited by 4
  5. 2024On the Trade-off between Flatness and Optimization in Distributed Learningcited by 3
  6. 2024Improving Generalization of Deep Neural Networks by Optimum Shiftingcited by 1