mlsanityData to 14 Oct 2026

Flat Minima in Linear Estimation and an Extended Gauss Markov Theorem

arXiv 2311.11093 ↗First posted 18 Nov 2023cs.LG stat.MLcited by 1

Not on the map. Why?

Abstract (arXiv)

We consider the problem of linear estimation, and establish an extension of the Gauss-Markov theorem, in which the bias operator is allowed to be non-zero but bounded with respect to a matrix norm of Schatten type. We derive simple and explicit formulas for the optimal estimator in the cases of Nuclear and Spectral norms (with the Frobenius case recovering ridge regression). Additionally, we analytically derive the generalization error in multiple random matrix ensembles, and compare with Ridge regression. Finally, we conduct an extensive simulation study, in which we show that the cross-validated Nuclear and Spectral regressors can outperform Ridge in several circumstances.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 20 of its 50 references that are in the map. First posted on arXiv 18 Nov 2023.

Builds on

20 of 50 references.

References · 20

  1. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  2. 2019Surprises in High-Dimensional Ridgeless Least Squares Interpolationcited by 513
  3. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  4. 2019Two models of double descent for weak featurescited by 195
  5. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 182
  6. 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
  7. 2015High-Dimensional Asymptotics of Prediction: Ridge Regression and Classificationcited by 148
  8. 2020Optimal Regularization Can Mitigate Double Descentcited by 109
  9. 2013Dropout Training as Adaptive Regularizationcited by 93
  10. 2018A Continuous-Time View of Early Stopping for Least Squarescited by 69
  11. 2020Kernel Alignment Risk Estimator: Risk Prediction from Training Datacited by 54
  12. 2020Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsificationcited by 53
8 more
  1. 2022Exploring Low Rank Training of Deep Neural Networkscited by 30
  2. 2021Unveiling the structure of wide flat minima in neural networkscited by 28
  3. 2022The good, the bad and the ugly sides of data augmentation: An implicit spectral regularization perspectivecited by 24
  4. 2022Sharp Asymptotics of Kernel Ridge Regression Beyond the Linear Regimecited by 19
  5. 2013The Squared-Error of Generalized LASSO: A Precise Analysiscited by 15
  6. 2018Low Rank Regularization: A Reviewcited by 15
  7. 2019On Dropout and Nuclear Norm Regularizationcited by 12
  8. 2022Exact Spectral Norm Regularization for Neural Networkscited by 5

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025A Probabilistic Basis for Low-Rank Matrix Learningcited by 1