# ODE-based Learning to Optimize

arXiv: https://arxiv.org/abs/2406.02006 · HTML: https://mlsanity.com/paper/2406.02006/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 68 references that are in the map.
- First posted on arXiv 4 Jun 2024.

## Builds on (most cited first, 12 of 12)

- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Symbolic Discovery of Optimization Algorithms](https://mlsanity.com/paper/2302.06675/) (2023)
- [Why gradient clipping accelerates training: A theoretical justification for adaptivity](https://mlsanity.com/paper/1905.11881/) (2019)
- [Learning to Optimize: A Primer and A Benchmark](https://mlsanity.com/paper/2103.12828/) (2021)
- [Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds](https://mlsanity.com/paper/1808.10038/) (2018)
- [Understanding the Acceleration Phenomenon via High-Resolution Differential Equations](https://mlsanity.com/paper/1810.08907/) (2018)
- [Direct Runge-Kutta Discretization Achieves Acceleration](https://mlsanity.com/paper/1805.00521/) (2018)
- [Towards Constituting Mathematical Structures for Learning to Optimize](https://mlsanity.com/paper/2305.18577/) (2023)
- [Accelerated Forward-Backward Optimization using Deep Learning](https://mlsanity.com/paper/2105.05210/) (2021)
- [Learning to Learn without Gradient Descent by Gradient Descent](https://mlsanity.com/paper/1611.03824/) (2016)
- [Accelerating Rescaled Gradient Descent: Fast Optimization of Smooth Functions](https://mlsanity.com/paper/1902.08825/) (2019)
- [Revisiting the Role of Euler Numerical Integration on Acceleration and Stability in Convex Optimization](https://mlsanity.com/paper/2102.11537/) (2021)

## Built on by (most cited first, 4 of 4)

- [Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement](https://mlsanity.com/paper/2602.22681/) (2026)
- [A Generalization Result for Convergence in Learning-to-Optimize](https://mlsanity.com/paper/2410.07704/) (2024)
- [Make Optimization Once and for All with Fine-grained Guidance](https://mlsanity.com/paper/2503.11462/) (2025)
- [Accelerating Optimization via Differentiable Stopping Time](https://mlsanity.com/paper/2505.22509/) (2025)

## Abstract (arXiv)

Recent years have seen a growing interest in understanding acceleration methods through the lens of ordinary differential equations (ODEs). Despite the theoretical advancements, translating the rapid convergence observed in continuous-time models to discrete-time iterative methods poses significant challenges. In this paper, we present a comprehensive framework integrating the inertial systems with Hessian-driven damping equation (ISHD) and learning-based approaches for developing optimization methods through a deep synergy of theoretical insights. We first establish the convergence condition for ensuring the convergence of the solution trajectory of ISHD. Then, we show that provided the stability condition, another relaxed requirement on the coefficients of ISHD, the sequence generated through the explicit Euler discretization of ISHD converges, which gives a large family of practical optimization methods. In order to select the best optimization method in this family for certain problems, we introduce the stopping time, the time required for an optimization method derived from ISHD to achieve a predefined level of suboptimality. Then, we formulate a novel learning to optimize (L2O) problem aimed at minimizing the stopping time subject to the convergence and stability condition. To navigate this learning problem, we present an algorithm combining stochastic optimization and the penalty method (StoPM). The convergence of StoPM using the conservative gradient is proved. Empirical validation of our framework is conducted through extensive numerical experiments across a diverse set of optimization problems. These experiments showcase the superior performance of the learned optimization methods.