mlsanityData to 14 Oct 2026

Stochastic Saddle Avoidance Beyond Unit Excitation and Smoothness: A Pathwise Lyapunov-Perron Framework

arXiv 2608.03001 ↗First posted 4 Aug 2026math.OC cs.LG math.DScited by 0

Not on the map. Why?

Abstract (arXiv)

Unit excitation (UE) is a common assumption in stochastic saddle avoidance: the stochastic error must have a uniformly positive component along every direction, in expectation. This condition gives a direct way to rule out convergence to strict saddles, but it also oversimplifies the actual noise structure, and does not match many stochastic optimization regimes. In overparameterized or interpolation models, the noise may vanish near stationarity. In finite-sum problems, the stochastic gradient noise may lie in a low-dimensional, data-dependent subspace. In these (common) scenarios, UE is naturally not satisfied. In this paper, we prove an abstract almost sure avoidance theorem for stochastic recursions without UE. The theorem replaces UE-type requirements by verifiable pathwise conditions. In applications, these conditions follow, e.g., from local smoothness and finite-moment assumptions under standard i.i.d. sampling, or from the finite-sum structure under without-replacement sampling. Since the stochastically sampled maps generally do not share a fixed point, the celebrated center-stable manifold argument used in deterministic analyses is not directly applicable. Instead, we use a path-dependent change of variables together with a pathwise Lyapunov--Perron-based proof strategy. As applications, we obtain strict saddle avoidance for stochastic mirror descent (including SGD) and for random reshuffling. For nonsmooth composite objectives, we prove avoidance results for a proximal-type stochastic gradient method. Combining these insights with suitable iterate convergence guarantees, this allows establishing convergence to local minimizers of the original objective function.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 43 of its 81 references that are in the map. First posted on arXiv 4 Aug 2026.

Builds on

43 of 81 references.

References · 43

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  4. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 664
  5. 2014SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectivescited by 402
  6. 2017How to Escape Saddle Points Efficientlycited by 285
  7. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 243
  8. 2018Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptroncited by 143
  9. 2020Random Reshuffling: Simple Analysis with Vast Improvementscited by 115
  10. 2016Relatively-Smooth Convex Optimization by First-Order Methods, and Applicationscited by 108
  11. 2016A Geometric Analysis of Phase Retrievalcited by 92
  12. 2016Global Optimality of Local Search for Low Rank Matrix Recoverycited by 82
31 more
  1. 2020On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problemscited by 80
  2. 2015Why Random Reshuffling Beats Stochastic Gradient Descentcited by 79
  3. 2014A Proximal Stochastic Gradient Method with Progressive Variance Reductioncited by 78
  4. 2020SGD for Structured Nonconvex Functions: Learning Rates, Minibatching and Interpolationcited by 67
  5. 2019How Good is SGD with Random Shuffling?cited by 64
  6. 2019SGD without Replacement: Sharper Rates for General Smooth Convex Functionscited by 61
  7. 2020A Unified Convergence Analysis for Shuffling-Type Gradient Methodscited by 58
  8. 2020Closing the convergence gap of SGD without replacementcited by 57
  9. 2013Fast Convergence of Stochastic Gradient Descent under a Strong Growth Conditioncited by 54
  10. 2015Complete Dictionary Recovery over the Sphere I: Overview and the Geometric Picturecited by 49
  11. 2017First Order Methods beyond Convexity and Lipschitz Gradient Continuity with Applications to Quadratic Inverse Problemscited by 37
  12. 2019First-order methods almost always avoid saddle points: the case of vanishing step-sizescited by 35
  13. 2015When Are Nonconvex Problems Not Scary?cited by 28
  14. 2021Convergence of Random Reshuffling Under The Kurdyka-Lojasiewicz Inequalitycited by 16
  15. 2013Stochastic Block Mirror Descent Methods for Nonsmooth and Stochastic Optimizationcited by 12
  16. 2022A Unified Convergence Theorem for Stochastic Optimization Methodscited by 12
  17. 2021Active manifolds, stratifications, and convergence to local minima in nonsmooth optimizationcited by 11
  18. 2023On the Trajectories of SGD Without Replacementcited by 11
  19. 2021Stochastic Subgradient Descent Escapes Active Strict Saddles on Weakly Convex Functionscited by 9
  20. 2019On Nonconvex Optimization for Machine Learning: Gradients, Stochasticity, and Saddle Pointscited by 8
  21. 2023A Normal Map-Based Proximal Stochastic Gradient Method: Convergence and Identification Propertiescited by 5
  22. 2023High Probability Guarantees for Random Reshufflingcited by 5
  23. 2021A trust region-type normal map-based semismooth Newton method for nonsmooth nonconvex composite optimizationcited by 4
  24. 2023A New Random Reshuffling Method for Nonsmooth Nonconvex Finite-sum Optimizationcited by 4
  25. 2024Proximal random reshuffling under local Lipschitz continuitycited by 4
  26. 2024Random Reshuffling with Momentum: Complexity Bounds and Last-iterate Convergencecited by 2
  27. 2026A non-autonomous center-stable set theorem for saddle avoidance in optimizationcited by 2
  28. 2022Identifiability, the KL property in metric spaces, and subgradient curvescited by 1
  29. 2022On the local convergence of the semismooth Newton method for composite optimizationcited by 1
  30. 2023Almost Sure Saddle Avoidance of Stochastic Gradient Methods without the Bounded Gradient Assumptioncited by 1
  31. 2023Variational Properties of Decomposable Functions. Part I: Strict Epi-Calculus and Applicationscited by 1

Built on by

Not cited yet.