mlsanityData to 7 Oct 2026

Continuous-time Models for Stochastic Optimization Algorithms

arXiv 1810.02565 ↗First posted 5 Oct 2018math.OC cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

We propose new continuous-time formulations for first-order stochastic optimization algorithms such as mini-batch gradient descent and variance-reduced methods. We exploit these continuous-time models, together with simple Lyapunov analysis as well as tools from stochastic calculus, in order to derive convergence bounds for various types of non-convex functions. Guided by such analysis, we show that the same Lyapunov arguments hold in discrete-time, leading to matching rates. In addition, we use these models and Ito calculus to infer novel insights on the dynamics of SGD, proving that a decreasing learning rate acts as time warping or, equivalently, as landscape stretching.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 2; 2024: 2; 2025: 2; 2026: 6.

16 papers.

Sort:

By area:

  1. 2021Generalization Bounds using Lower Tail Exponents in Stochastic Optimizerscited by 27
  2. 2023An SDE for Modeling SAM: Theory and Insightscited by 22
  3. 2022From Gradient Flow on Population Loss to Learning with Stochastic Gradient Descentcited by 5
  4. 2025Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEscited by 4
  5. 2025On the Interaction of Batch Noise, Adaptivity, and Compression, under (L0,L1)-Smoothness: An SDE Approachcited by 4
  6. 2026Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decaycited by 4
  7. 2026Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Lawscited by 4
  8. 2022Understanding A Class of Decentralized and Federated Optimization Algorithms: A Multi-Rate Feedback Control Perspectivecited by 2
  9. 2026Learning Gradient Flow: Using Equation Discovery to Accelerate Engineering Optimizationcited by 1
  10. 2026On Interpolation Formulas Describing Neural Network Generalizationcited by 1
  11. 2022Generalized Gradient Flows with Provable Fixed-Time Convergence and Fast Evasion of Non-Degenerate Saddle Pointscited by 0
  12. 2023Ito Diffusion Approximation of Universal Ito Chains for Sampling, Optimization and Boostingcited by 0
4 more
  1. 2024Continuous-time Riemannian SGD and SVRG Flows on Wasserstein Probabilistic Spacecited by 0
  2. 2024Provable Accuracy Bounds for Hybrid Dynamical Optimization and Samplingcited by 0
  3. 2026How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Lawscited by 0
  4. 2026A Diffusion Approximation for Temporal-Difference Learning with Linear Features under Markovian Noisecited by 0