# Continuous-time Models for Stochastic Optimization Algorithms

arXiv: https://arxiv.org/abs/1810.02565 · HTML: https://mlsanity.com/paper/1810.02565/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Oct 2018.

## Built on by (most cited first, 17 of 17)

- [Generalization Bounds using Lower Tail Exponents in Stochastic Optimizers](https://mlsanity.com/paper/2108.00781/) (2021)
- [An SDE for Modeling SAM: Theory and Insights](https://mlsanity.com/paper/2301.08203/) (2023)
- [Optimal Learning Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay](https://mlsanity.com/paper/2602.06797/) (2026)
- [Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Laws](https://mlsanity.com/paper/2602.14208/) (2026)
- [From Gradient Flow on Population Loss to Learning with Stochastic Gradient Descent](https://mlsanity.com/paper/2210.06705/) (2022)
- [Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEs](https://mlsanity.com/paper/2502.17009/) (2025)
- [On the Interaction of Batch Noise, Adaptivity, and Compression, under (L0,L1)-Smoothness: An SDE Approach](https://mlsanity.com/paper/2506.00181/) (2025)
- [Understanding A Class of Decentralized and Federated Optimization Algorithms: A Multi-Rate Feedback Control Perspective](https://mlsanity.com/paper/2204.12663/) (2022)
- [Learning Gradient Flow: Using Equation Discovery to Accelerate Engineering Optimization](https://mlsanity.com/paper/2602.13513/) (2026)
- [On Interpolation Formulas Describing Neural Network Generalization](https://mlsanity.com/paper/2603.13872/) (2026)
- [Generalized Gradient Flows with Provable Fixed-Time Convergence and Fast Evasion of Non-Degenerate Saddle Points](https://mlsanity.com/paper/2212.03765/) (2022)
- [Ito Diffusion Approximation of Universal Ito Chains for Sampling, Optimization and Boosting](https://mlsanity.com/paper/2310.06081/) (2023)
- [Continuous-time Riemannian SGD and SVRG Flows on Wasserstein Probabilistic Space](https://mlsanity.com/paper/2401.13530/) (2024)
- [Provable Accuracy Bounds for Hybrid Dynamical Optimization and Sampling](https://mlsanity.com/paper/2410.06397/) (2024)
- [How Should LLMs Consume High-Quality Data? Optimal Data Scheduling via Quality-Aware Functional Scaling Laws](https://mlsanity.com/paper/2605.25698/) (2026)
- [A Diffusion Approximation for Temporal-Difference Learning with Linear Features under Markovian Noise](https://mlsanity.com/paper/2606.18183/) (2026)
- [A Rate Transfer Method: Convergence Rates in Stochastic Gradient Descent via Stochastic Calculus](https://mlsanity.com/paper/2610.18739/) (2026)

## Abstract (arXiv)

We propose new continuous-time formulations for first-order stochastic optimization algorithms such as mini-batch gradient descent and variance-reduced methods. We exploit these continuous-time models, together with simple Lyapunov analysis as well as tools from stochastic calculus, in order to derive convergence bounds for various types of non-convex functions. Guided by such analysis, we show that the same Lyapunov arguments hold in discrete-time, leading to matching rates. In addition, we use these models and Ito calculus to infer novel insights on the dynamics of SGD, proving that a decreasing learning rate acts as time warping or, equivalently, as landscape stretching.