mlsanityData to 7 Oct 2026

Stochastic subgradient method converges on tame functions

arXiv 1804.07795 ↗First posted 20 Apr 2018math.OC cs.LGcited by 120

Not on the map. Why?

Abstract (arXiv)

This work considers the question: what convergence guarantees does the stochastic subgradient method have in the absence of smoothness and convexity? We prove that the stochastic subgradient method, on any semialgebraic locally Lipschitz function, produces limit points that are all first-order stationary. More generally, our result applies to any function with a Whitney stratifiable graph. In particular, this work endows the stochastic subgradient method, and its proximal extension, with rigorous convergence guarantees for a wide class of problems arising in data science---including all popular deep learning architectures.

Key facts. Cited by 120 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 19; 2023: 21; 2024: 20; 2025: 21; 2026: 21.

120 papers.

Sort:

By area:

  1. 2021Unbalanced minibatch Optimal Transport; applications to Domain Adaptationcited by 123
  2. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  3. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  4. 2021Minibatch optimal transport distances; analysis and applicationscited by 62
  5. 2021Nonsmooth Implicit Differentiation for Machine Learning and Optimizationcited by 43
  6. 2022Gradient-Free Methods for Deterministic and Stochastic Nonsmooth Nonconvex Optimizationcited by 40
  7. 2021Oracle Complexity in Nonsmooth Nonconvex Optimizationcited by 38
  8. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  9. 2022Training invariances and the low-rank phenomenon: beyond linear networkscited by 33
  10. 2022The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networkscited by 25
  11. 2023Deterministic Nonsmooth Nonconvex Optimizationcited by 25
  12. 2021Convergence of gradient descent for learning linear neural networkscited by 22
38 more
  1. 2023Adam-family Methods for Nonsmooth Optimization with Convergence Guaranteescited by 22
  2. 2024Adam with model exponential moving average is effective for nonconvex optimizationcited by 22
  3. 2021A Stochastic Sequential Quadratic Optimization Algorithm for Nonlinear Equality Constrained Optimization with Rank-Deficient Jacobianscited by 21
  4. 2021Towards Understanding Learning in Neural Networks with Linear Teacherscited by 19
  5. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
  6. 2023Asymptotic normality and optimality in nonsmooth stochastic approximationcited by 14
  7. 2024Flavors of Margin: Implicit Bias of Steepest Descent in Homogeneous Neural Networkscited by 13
  8. 2022A Unified Convergence Theorem for Stochastic Optimization Methodscited by 12
  9. 2024A CMDP-within-online framework for Meta-Safe Reinforcement Learningcited by 12
  10. 2021Active manifolds, stratifications, and convergence to local minima in nonsmooth optimizationcited by 11
  11. 2021Improving Metric Dimensionality Reduction with Distributed Topologycited by 10
  12. 2022Adversarial Reprogramming Revisitedcited by 10
  13. 2023Stochastic Approximation Beyond Gradient for Signal Processing and Machine Learningcited by 10
  14. 2024Directional Convergence Near Small Initializations and Saddles in Two-Homogeneous Neural Networkscited by 10
  15. 2021Stochastic Subgradient Descent Escapes Active Strict Saddles on Weakly Convex Functionscited by 9
  16. 2022Convergence of First-Order Methods for Constrained Nonconvex Optimization with Dependent Datacited by 7
  17. 2022Importance Tempering: Group Robustness for Overparameterized Modelscited by 6
  18. 2022On Solution Functions of Optimization: Universal Approximation and Covering Number Boundscited by 6
  19. 2023On the existence of minimizers in shallow residual ReLU neural network optimization landscapescited by 6
  20. 2024Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embeddingcited by 6
  21. 2026The Implicit Bias of Adam and Muon on Smooth Homogeneous Neural Networkscited by 6
  22. 2021A Gradient Sampling Algorithm for Stratified Maps with Applications to Topological Data Analysiscited by 5
  23. 2022An Inexact Augmented Lagrangian Algorithm for Training Leaky ReLU Neural Network with Group Sparsitycited by 5
  24. 2022Sparse Deep Neural Network for Nonlinear Partial Differential Equationscited by 5
  25. 2022Gradient descent provably escapes saddle points in the training of shallow ReLU networkscited by 5
  26. 2023On the existence of optimal shallow feedforward networks with ReLU activationcited by 5
  27. 2023A Normal Map-Based Proximal Stochastic Gradient Method: Convergence and Identification Propertiescited by 5
  28. 2023Adam-family Methods with Decoupled Weight Decay in Deep Learningcited by 5
  29. 2024Differentiable Mapper For Topological Optimization Of Data Representationcited by 5
  30. 2024D-GRIL: End-to-End Topological Learning with 2-parameter Persistencecited by 5
  31. 2025Differentiable Nonlinear Model Predictive Controlcited by 5
  32. 2025SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structurescited by 5
  33. 2021The structure of conservative gradient fieldscited by 4
  34. 2021First-Order Methods for Convex Optimizationcited by 4
  35. 2023Convergence of SGD for Training Neural Networks with Sliced Wasserstein Lossescited by 4
  36. 2023Taming Binarized Neural Networks and Mixed-Integer Programscited by 4
  37. 2023A New Random Reshuffling Method for Nonsmooth Nonconvex Finite-sum Optimizationcited by 4
  38. 2023Generator Born from Classifiercited by 4

50 most cited of 120 All 120 (JSON)