# Homogenization of SGD in high-dimensions: Exact dynamics and generalization properties

arXiv: https://arxiv.org/abs/2205.07069 · HTML: https://mlsanity.com/paper/2205.07069/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 81 references that are in the map.
- First posted on arXiv 14 May 2022.

## Builds on (most cited first, 20 of 20)

- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [Surprises in High-Dimensional Ridgeless Least Squares Interpolation](https://mlsanity.com/paper/1903.08560/) (2019)
- [Measuring the Intrinsic Dimension of Objective Landscapes](https://mlsanity.com/paper/1804.08838/) (2018)
- [In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning](https://mlsanity.com/paper/1412.6614/) (2014)
- [Three Factors Influencing Minima in SGD](https://mlsanity.com/paper/1711.04623/) (2017)
- [The large learning rate phase of deep learning: the catapult mechanism](https://mlsanity.com/paper/2003.02218/) (2020)
- [Benign overfitting in ridge regression](https://mlsanity.com/paper/2009.14286/) (2020)
- [The Heavy-Tail Phenomenon in SGD](https://mlsanity.com/paper/2006.04740/) (2020)
- [A Random Matrix Approach to Neural Networks](https://mlsanity.com/paper/1702.05419/) (2017)
- [A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descent](https://mlsanity.com/paper/2006.05013/) (2020)
- [Sharp convergence rates for Langevin dynamics in the nonconvex setting](https://mlsanity.com/paper/1805.01648/) (2018)
- [A Variational Analysis of Stochastic Gradient Algorithms](https://mlsanity.com/paper/1602.02666/) (2016)
- [SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality](https://mlsanity.com/paper/2102.04396/) (2021)
- [Power-law escape rate of SGD](https://mlsanity.com/paper/2105.09557/) (2021)
- [Eigenvalue distribution of nonlinear models of random matrices](https://mlsanity.com/paper/1904.03090/) (2019)
- [Covariate Shift in High-Dimensional Random Feature Regression](https://mlsanity.com/paper/2111.08234/) (2021)
- [The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects](https://mlsanity.com/paper/1803.00195/) (2018)
- [Risk Bounds of Multi-Pass SGD for Least Squares in the Interpolation Regime](https://mlsanity.com/paper/2203.03159/) (2022)
- [Lower Bounds on the Generalization Error of Nonlinear Learning Models](https://mlsanity.com/paper/2103.14723/) (2021)
- [On the number of variables to use in principal component regression](https://mlsanity.com/paper/1906.01139/) (2019)

## Built on by (most cited first, 27 of 27)

- [Deterministic equivalent and error universality of deep random features learning](https://mlsanity.com/paper/2302.00401/) (2023)
- [Hitting the High-Dimensional Notes: An ODE for SGD learning dynamics on GLMs and multi-index models](https://mlsanity.com/paper/2308.08977/) (2023)
- [SAM operates far from home: eigenvalue regularization as a dynamical phenomenon](https://mlsanity.com/paper/2302.08692/) (2023)
- [Implicit Regularization or Implicit Conditioning? Exact Risk Trajectories of SGD in High Dimensions](https://mlsanity.com/paper/2206.07252/) (2022)
- [Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks](https://mlsanity.com/paper/2507.02119/) (2025)
- [High dimensional analysis reveals conservative sharpening and a stochastic edge of stability](https://mlsanity.com/paper/2404.19261/) (2024)
- [On Regularization via Early Stopping for Least Squares Regression](https://mlsanity.com/paper/2406.04425/) (2024)
- [Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning](https://mlsanity.com/paper/2306.08590/) (2023)
- [To Clip or not to Clip: the Dynamics of SGD with Gradient Clipping in High-Dimensions](https://mlsanity.com/paper/2406.11733/) (2024)
- [Exact Risk Curves of signSGD in High-Dimensions: Quantifying Preconditioning and Noise-Compression Effects](https://mlsanity.com/paper/2411.12135/) (2024)
- [High-dimensional scaling limits and fluctuations of online least-squares SGD with smooth covariance](https://mlsanity.com/paper/2304.00707/) (2023)
- [Emergence of heavy tails in homogenized stochastic gradient descent](https://mlsanity.com/paper/2402.01382/) (2024)
- [The High Line: Exact Risk and Learning Rate Curves of Stochastic Adaptive Learning Rate Algorithms](https://mlsanity.com/paper/2405.19585/) (2024)
- [Online Learning and Information Exponents: On The Importance of Batch size, and Time/Complexity Tradeoffs](https://mlsanity.com/paper/2406.02157/) (2024)
- [Stochastic Differential Equations models for Least-Squares Stochastic Gradient Descent](https://mlsanity.com/paper/2407.02322/) (2024)
- [SGD with memory: fundamental properties and stochastic acceleration](https://mlsanity.com/paper/2410.04228/) (2024)
- [Statistical Inference for Linear Functionals of Online SGD in High-dimensional Linear Regression](https://mlsanity.com/paper/2302.09727/) (2023)
- [Stochastic Gradient Flow Dynamics of Test Risk and its Exact Solution for Weak Features](https://mlsanity.com/paper/2402.07626/) (2024)
- [Statistical Inference for Linear Functionals of Online Least-squares SGD when t gtrsim d1+δ](https://mlsanity.com/paper/2510.19734/) (2025)
- [High-Dimensional Privacy-Utility Dynamics of Noisy Stochastic Gradient Descent on Least Squares](https://mlsanity.com/paper/2510.16687/) (2025)
- [Mini-Batch Covariance, Diffusion Limits, and Oracle Complexity in Stochastic Gradient Descent: A Sampling-Design Perspective](https://mlsanity.com/paper/2603.02417/) (2026)
- [Stochastic Gradient Descent in the Saddle-to-Saddle Regime of Deep Linear Networks](https://mlsanity.com/paper/2604.06366/) (2026)
- [The Role of Symmetry in Optimizing Overparameterized Networks](https://mlsanity.com/paper/2604.25150/) (2026)
- [Factor Augmented High-Dimensional SGD](https://mlsanity.com/paper/2605.19291/) (2026)
- [When Stronger Triggers Backfire: A High-Dimensional Theory of Backdoor Attacks](https://mlsanity.com/paper/2605.22481/) (2026)
- [Homogenization of ℓ2-Adversarial Training in High-Dimensions: Exact Dynamics under Stochastic Gradient Descent](https://mlsanity.com/paper/2607.00207/) (2026)
- [Beyond Negative-Ridge Endpoints: Mixed-Sign Spectral Regularization via Negative-Shifted Gradient Descent](https://mlsanity.com/paper/2607.22474/) (2026)

## Abstract (arXiv)

We develop a stochastic differential equation, called homogenized SGD, for analyzing the dynamics of stochastic gradient descent (SGD) on a high-dimensional random least squares problem with ℓ2-regularization. We show that homogenized SGD is the high-dimensional equivalence of SGD -- for any quadratic statistic (e.g., population risk with quadratic loss), the statistic under the iterates of SGD converges to the statistic under homogenized SGD when the number of samples n and number of features d are polynomially related (dc < n < d1/c for some c > 0). By analyzing homogenized SGD, we provide exact non-asymptotic high-dimensional expressions for the generalization performance of SGD in terms of a solution of a Volterra integral equation. Further we provide the exact value of the limiting excess risk in the case of quadratic losses when trained by SGD. The analysis is formulated for data matrices and target vectors that satisfy a family of resolvent conditions, which can roughly be viewed as a weak (non-quantitative) form of delocalization of sample-side singular vectors of the data. Several motivating applications are provided including sample covariance matrices with independent samples and random features with non-generative model targets.