Scaling Limits of Constant-Stepsize SGD at Flat Minima

For stochastic gradient descent (SGD) with a constant stepsize α, the invariant law of the iterates, centered at a minimizer, describes the behavior of the algorithm over long time horizons. In the strongly convex case, this invariant law has the familiar √α scaling and a Gaussian limit as αdownarrow 0. We show that this behavior changes fundamentally for convex objectives H with flat minima and (sub)quadratic tails. More specifically, we study SGD with Markovian noise generated by a contractive driving chain. For every sufficiently small constant stepsize α, we prove existence, uniqueness, and geometric convergence to an augmented invariant law in a Wasserstein distance induced by an α-dependent metric. When the minimizer x_star has local flatness exponent m≥2, meaning that nabla2 H(x)asymp lVert x-x_starrVertm-2 Id as x→ x_star, we obtain a contraction bound with factor 1-cαm-1, where c>0 is a constant. This recovers the factor 1-cα in the quadratic case m=2. We then analyze the small-stepsize scaling limit. We show that the invariant law concentrates on the scale α1/m and that the rescaled iterates converge weakly to the stationary distribution of the stochastic differential equation dYt=-h0(Yt),dt+Sigma1/2,dBt , where h0 is the limiting drift at the minimizer and Sigma denotes the asymptotic covariance. This recovers the Gaussian limit when m=2 and gives generally non-Gaussian stationary limits in the flat case m>2. Finally, we give corresponding results for coordinate-separable objectives with unequal flatness exponents.