mlsanityData to 14 Oct 2026

Ordinary Nonconvex SGD under Distance-Dependent Moments: Finite-Horizon Stationarity and Nagaev Bounds

arXiv 2609.30499 ↗First posted 24 Sep 2026stat.ML cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Uniform noise-moment bounds exclude stochastic gradients whose variability increases with the iterate. We study ordinary, single-sample stochastic gradient descent for smooth, lower-bounded, possibly nonconvex objectives under distance-dependent conditional moments. Under second moments alone, a direct descent--displacement argument yields T-1/3 expected average squared-gradient stationarity with a horizon-dependent stepsize. An explicit oracle-complexity corollary matches the known smooth Blum--Gladyshev (BG-0) lower bound, including the Lb2Delta3varepsilon-6 and LDeltaσ2varepsilon-4 stochastic terms, where Delta is the initial objective gap and σ2+b2|x-x1|2 bounds the variance. Thus unchanged SGD attains the minimax stochastic complexity in this second-moment class. For p>2, predictable localization and a Hilbert-space Fuk--Nagaev inequality yield a high-probability bound separating logarithmic variance and polynomial rare-shock contributions. The localization radius is derived from the recursion: no bounded-iterate assumption, clipping, normalization, momentum, or increasing batch size is needed. We also give increasing-confidence rates, an objective-gap-growth refinement recovering root-T stationarity, and stochastic Lp-Lipschitz examples. The broad BG-0 optimality statement is distinguished from the smaller mean-square-smooth class, in which additional oracle structure permits faster algorithms.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 29 references that are in the map. First posted on arXiv 24 Sep 2026.

Builds on

15 of 29 references.

References · 15

  1. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  2. 2013Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programmingcited by 724
  3. 2019Lower Bounds for Non-Convex Stochastic Optimizationcited by 307
  4. 2020Better Theory for SGD in the Nonconvex Worldcited by 158
  5. 2020Stochastic Optimization with Heavy-Tailed Noise via Accelerated Gradient Clippingcited by 104
  6. 2017Bridging the Gap between Constant Step Size Stochastic Gradient Descent and Markov Chainscited by 88
  7. 2021High-probability Bounds for Non-Convex Stochastic Optimization with Heavy Tailscited by 80
  8. 2022The Power of Adaptivity in SGD: Self-Tuning Step Sizes with Unbounded Gradients and Affine Variancecited by 61
  9. 2023High-Probability Bounds for Stochastic Optimization and Variational Inequalities: the Case of Unbounded Variancecited by 61
  10. 2023High Probability Convergence of Stochastic Gradient Methodscited by 49
  11. 2024Nonconvex Stochastic Optimization under Heavy-Tailed Noises: Optimal Convergence without Gradient Clippingcited by 42
  12. 2023Provable convergence guarantees for black-box variational inferencecited by 29
3 more
  1. 2025Towards Weaker Variance Assumptions for Stochastic Optimizationcited by 17
  2. 2026Lower Bounds and Proximally Anchored SGD for Non-Convex Minimization Under Unbounded Variancecited by 5
  3. 2026Beyond Bounded Variance: Variance-Reduced Normalized Methods for Nonconvex Optimization under Blum-Gladyshev Noisecited by 2

Built on by

Not cited yet.