# Shadowing Properties of Optimization Algorithms

arXiv: https://arxiv.org/abs/1911.05206 · HTML: https://mlsanity.com/paper/1911.05206/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2019.

## Built on by (most cited first, 6 of 6)

- [Continuous vs. Discrete Optimization of Deep Neural Networks](https://mlsanity.com/paper/2107.06608/) (2021)
- [Continuous-Time Meta-Learning with Forward Mode Differentiation](https://mlsanity.com/paper/2203.01443/) (2022)
- [Revisiting the Role of Euler Numerical Integration on Acceleration and Stability in Convex Optimization](https://mlsanity.com/paper/2102.11537/) (2021)
- [On the Interaction of Batch Noise, Adaptivity, and Compression, under (L0,L1)-Smoothness: An SDE Approach](https://mlsanity.com/paper/2506.00181/) (2025)
- [On the connections between optimization algorithms, Lyapunov functions, and differential equations: theory and insights](https://mlsanity.com/paper/2305.08658/) (2023)
- [Is Learning in Biological Neural Networks based on Stochastic Gradient Descent? An analysis using stochastic processes](https://mlsanity.com/paper/2309.05102/) (2023)

## Abstract (arXiv)

Ordinary differential equation (ODE) models of gradient-based optimization methods can provide insights into the dynamics of learning and inspire the design of new algorithms. Unfortunately, this thought-provoking perspective is weakened by the fact that, in the worst case, the error between the algorithm steps and its ODE approximation grows exponentially with the number of iterations. In an attempt to encourage the use of continuous-time methods in optimization, we show that, if some additional regularity on the objective is assumed, the ODE representations of Gradient Descent and Heavy-ball do not suffer from the aforementioned problem, once we allow for a small perturbation on the algorithm initial condition. In the dynamical systems literature, this phenomenon is called shadowing. Our analysis relies on the concept of hyperbolicity, as well as on tools from numerical analysis.