mlsanityData to 7 Oct 2026

Flat minima generalize for low-rank matrix recovery

arXiv 2203.03756 ↗First posted 7 Mar 2022cs.LG math.OC stat.MLcited by 28

Not on the map. Why?

Abstract (arXiv)

Empirical evidence suggests that for a variety of overparameterized nonlinear models, most notably in neural network training, the growth of the loss around a minimizer strongly impacts its performance. Flat minima -- those around which the loss grows slowly -- appear to generalize well. This work takes a step towards understanding this phenomenon by focusing on the simplest class of overparameterized nonlinear models: those arising in low-rank matrix recovery. We analyze overparameterized matrix and bilinear sensing, robust PCA, covariance matrix estimation, and single hidden layer neural networks with quadratic activation functions. In all cases, we show that flat minima, measured by the trace of the Hessian, exactly recover the ground truth under standard statistical assumptions. For matrix completion, we establish weak recovery, although empirical evidence suggests exact recovery holds here as well. We conclude with synthetic experiments that illustrate our findings and discuss the effect of depth on flat solutions.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 35 of its 65 references that are in the map. First posted on arXiv 7 Mar 2022.

Builds on

35 of 65 references.

References · 35

  1. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  4. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  5. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  6. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  7. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  8. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  9. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  10. 2017Exploring Generalization in Deep Learningcited by 595
  11. 2017Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Datacited by 527
  12. 2017Sharp Minima Can Generalize For Deep Netscited by 462
23 more
  1. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  2. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  3. 2017Implicit Regularization in Matrix Factorizationcited by 318
  4. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  5. 2017Three Factors Influencing Minima in SGDcited by 270
  6. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  7. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  8. 2017A Bayesian Perspective on Generalization and Stochastic Gradient Descentcited by 134
  9. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  10. 2017No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysiscited by 127
  11. 2019A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Casecited by 113
  12. 2019How do infinite width bounded norm networks look in function space?cited by 105
  13. 2018Revisiting Small Batch Training for Deep Neural Networkscited by 100
  14. 2021Global Convergence of Gradient Descent for Asymmetric Low-Rank Matrix Factorizationcited by 44
  15. 2020Compressive sensing with un-trained neural networks: Gradient descent finds the smoothest approximationcited by 39
  16. 2013Incoherence-Optimal Matrix Completioncited by 27
  17. 2013Exact and Stable Covariance Estimation from Quadratic Sampling via Convex Programmingcited by 20
  18. 2013ROP: Matrix recovery via rank-one projectionscited by 18
  19. 2019Diametrical Risk Minimization: Theory and Computationscited by 15
  20. 2021Noisy Gradient Descent Converges to Flat Minima for Nonconvex Matrix Factorizationcited by 15
  21. 2022The Role of Linear Layers in Nonlinear Interpolating Networkscited by 15
  22. 2018Are ResNets Provably Better than Linear Predictors?cited by 12
  23. 2021Beyond Procrustes: Balancing-Free Gradient Descent for Asymmetric Low-Rank Matrix Sensingcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 9; 2024: 5; 2025: 9; 2026: 4.

28 papers.

Sort:

By area:

  1. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  2. 2023Dichotomy of Early and Late Phase Implicit Biases Can Provably Induce Grokkingcited by 76
  3. 2023Sharpness Minimization Algorithms Do Not Only Minimize Sharpness To Achieve Better Generalizationcited by 39
  4. 2023Catapults in SGD: spikes in the training loss and their impact on generalization through feature learningcited by 29
  5. 2023Improving Convergence and Generalization Using Parameter Symmetriescited by 23
  6. 2025Sparse identification of nonlinear dynamics and Koopman operators with Shallow Recurrent Decoder Networkscited by 19
  7. 2024Linear Recursive Feature Machines provably recover low-rank matricescited by 16
  8. 2023Loss Spike in Training Neural Networkscited by 15
  9. 2023How to escape sharp minima with random perturbationscited by 13
  10. 2023The Inductive Bias of Flatness Regularization for Deep Matrix Factorizationcited by 6
  11. 2024Tilted Sharpness-Aware Minimizationcited by 6
  12. 2025Zeroth-Order Optimization Finds Flat Minimacited by 6
16 more
  1. 2023On Statistical Properties of Sharpness-Aware Minimization: Provable Guaranteescited by 5
  2. 2024Simplicity Bias via Global Convergence of Sharpness Minimizationcited by 4
  3. 2025Regularization Implies balancedness in the deep linear networkcited by 4
  4. 2024Bilinear Sequence Regression: A Model for Learning from Long Sequences of High-dimensional Tokenscited by 3
  5. 2025Flat Minima and Generalization: Insights from Stochastic Convex Optimizationcited by 3
  6. 2025Sharpness of Minima in Deep Matrix Factorizationcited by 2
  7. 2025Convergence Dynamics of Over-Parameterized Score Matching for a Single Gaussiancited by 2
  8. 2026Low-Rank Adaptation Redux for Large Modelscited by 2
  9. 2024Probability Distribution Learning and Its Application in Deep Learningcited by 1
  10. 2025Flatness After All?cited by 1
  11. 2025R'enyi Sharpness: A Novel Sharpness that Strongly Correlates with Generalizationcited by 1
  12. 2026Does Sparse Connectivity Improve Generalization? Convolutional Networks Below the Edge of Stabilitycited by 1
  13. 2023Implicit Balancing and Regularization: Generalization and Convergence Guarantees for Overparameterized Asymmetric Matrix Sensingcited by 0
  14. 2025Flatness-Aware Stochastic Gradient Langevin Dynamicscited by 0
  15. 2026Generalization at the Edge of Stabilitycited by 0
  16. 2026Direction-Magnitude Decomposition for Low-Rank Matrix Optimization: Faster Convergence and Saddle-to-saddle Dynamicscited by 0