mlsanityData to 7 Oct 2026

The Implicit Regularization of Stochastic Gradient Flow for Least Squares

arXiv 2003.07802 ↗First posted 17 Mar 2020stat.ML cs.LG math.OCcited by 58

Not on the map. Why?

Abstract (arXiv)

We study the implicit regularization of mini-batch stochastic gradient descent, when applied to the fundamental problem of least squares regression. We leverage a continuous-time stochastic differential equation having the same moments as stochastic gradient descent, which we call stochastic gradient flow. We give a bound on the excess risk of stochastic gradient flow at time t, over ridge regression with tuning parameter λ = 1/t. The bound may be computed from explicit constants (e.g., the mini-batch size, step size, number of iterations), revealing precisely how these quantities drive the excess risk. Numerical examples show the bound can be small, indicating a tight relationship between the two estimators. We give a similar result relating the coefficients of stochastic gradient flow and ridge. These results hold under no conditions on the data matrix X, and across the entire optimization path (not just at convergence).

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). First posted on arXiv 17 Mar 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 15; 2022: 15; 2023: 9; 2024: 11; 2025: 4; 2026: 4.

58 papers.

Sort:

By area:

  1. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 107
  2. 2021Implicit Regularization in Tensor Factorizationcited by 43
  3. 2021Noisy Recurrent Neural Networkscited by 42
  4. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 38
  5. 2021The Benefits of Implicit Regularization from SGD in Least Squares Problemscited by 32
  6. 2024The Implicit Bias of Adam on Separable Datacited by 32
  7. 2022On Implicit Bias in Overparameterized Bilevel Optimizationcited by 30
  8. 2022Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisationcited by 29
  9. 2021Multi-scale Feature Learning Dynamics: Insights for Double Descentcited by 24
  10. 2021Does Momentum Change the Implicit Regularization on Separable Data?cited by 21
  11. 2021RATT: Leveraging Unlabeled Data to Guarantee Generalizationcited by 20
  12. 2021The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusioncited by 19
38 more
  1. 2022NoisyMix: Boosting Model Robustness to Common Corruptionscited by 17
  2. 2021Interpolation can hurt robust generalization even when there is no noisecited by 15
  3. 2022Algorithmic Stability of Heavy-Tailed Stochastic Gradient Descent on Least Squarescited by 12
  4. 2022Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysiscited by 12
  5. 2024Neuro-symbolic Learning Yielding Logical Constraintscited by 12
  6. 2022Sobolev Acceleration and Statistical Optimality for Learning Elliptic Equations via Gradient Descentcited by 11
  7. 2022On the (Non-)Robustness of Two-Layer Neural Networks in Different Learning Regimescited by 10
  8. 2023Generalized equivalences between subsampling and ridge regularizationcited by 10
  9. 2024On Regularization via Early Stopping for Least Squares Regressioncited by 10
  10. 2023Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learningcited by 9
  11. 2021A Continuous-time Stochastic Gradient Descent Method for Continuous Datacited by 7
  12. 2021Group-Sparse Matrix Factorization for Transfer Learning of Word Embeddingscited by 6
  13. 2021Comparing Classes of Estimators: When does Gradient Descent Beat Ridge Regression in Linear Models?cited by 5
  14. 2021Robust Generalization of Quadratic Neural Networks via Function Identificationcited by 5
  15. 2022How You Start Matters for Generalizationcited by 5
  16. 2022On Optimal Early Stopping: Over-informative versus Under-informative Parametrizationcited by 4
  17. 2023Robust Linear Regression: Gradient-descent, Early-stopping, and Beyondcited by 4
  18. 2024Failures and Successes of Cross-Validation for Early-Stopped Gradient Descentcited by 4
  19. 2023On the Effect of Initialization: The Scaling Path of 2-Layer Neural Networkscited by 3
  20. 2024Stochastic Differential Equations models for Least-Squares Stochastic Gradient Descentcited by 3
  21. 2022High-dimensional Asymptotics of Langevin Dynamics in Spiked Matrix Modelscited by 2
  22. 2024Implicit Bias in Noisy-SGD: With Applications to Differentially Private Trainingcited by 2
  23. 2024Improving Implicit Regularization of SGD with Preconditioning for Least Square Problemscited by 2
  24. 2025Comparing regularisation paths of (conjugate) gradient estimators in ridge regressioncited by 2
  25. 2025Basic Inequalities for First-Order Optimization with Applications to Statistical Risk Analysiscited by 2
  26. 2022Implicit Regularization Properties of Variance Reduced Stochastic Mirror Descentcited by 1
  27. 2022The Directional Bias Helps Stochastic Gradient Descent to Generalize in Kernel Regression Modelscited by 1
  28. 2023Statistical Inference for Linear Functionals of Online SGD in High-dimensional Linear Regressioncited by 1
  29. 2023Transgressing the boundaries: towards a rigorous understanding of deep learning and its (non-)robustnesscited by 1
  30. 2023Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relationcited by 1
  31. 2024Stochastic Gradient Flow Dynamics of Test Risk and its Exact Solution for Weak Featurescited by 1
  32. 2024Learning from True-False Labels via Multi-modal Prompt Retrievingcited by 1
  33. 2024Implicit Regularization Paths of Weighted Neural Representationscited by 1
  34. 2025Statistical Inference for Linear Functionals of Online Least-squares SGD when t gtrsim d1+δcited by 1
  35. 2021Generalization by design: Shortcuts to Generalization in Deep Learningcited by 0
  36. 2021AgFlow: Fast Model Selection of Penalized PCA via Implicit Regularization Effects of Gradient Flowcited by 0
  37. 2022The Implicit Regularization of Momentum Gradient Descent with Early Stoppingcited by 0
  38. 2022Accelerated Gradient Flow: Risk, Stability, and Implicit Regularizationcited by 0

50 most cited of 58 All 58 (JSON)