# Direct Runge-Kutta Discretization Achieves Acceleration

arXiv: https://arxiv.org/abs/1805.00521 · HTML: https://mlsanity.com/paper/1805.00521/

## Key facts

- Cited by 21 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 1 May 2018.

## Built on by (most cited first, 21 of 21)

- [Continuous vs. Discrete Optimization of Deep Neural Networks](https://mlsanity.com/paper/2107.06608/) (2021)
- [A Continuized View on Nesterov Acceleration for Stochastic Gradient Descent and Randomized Gossip](https://mlsanity.com/paper/2106.07644/) (2021)
- [Accelerating optimization over the space of probability measures](https://mlsanity.com/paper/2310.04006/) (2023)
- [A Recursively Recurrent Neural Network (R2N2) Architecture for Learning Iterative Algorithms](https://mlsanity.com/paper/2211.12386/) (2022)
- [Domain Adversarial Training: A Game Perspective](https://mlsanity.com/paper/2202.05352/) (2022)
- [Revisiting the Role of Euler Numerical Integration on Acceleration and Stability in Convex Optimization](https://mlsanity.com/paper/2102.11537/) (2021)
- [Potential Function-based Framework for Making the Gradients Small in Convex and Min-Max Optimization](https://mlsanity.com/paper/2101.12101/) (2021)
- [ODE-based Learning to Optimize](https://mlsanity.com/paper/2406.02006/) (2024)
- [A Contraction Theory Approach to Optimization Algorithms from Acceleration Flows](https://mlsanity.com/paper/2105.08832/) (2021)
- [Online Algorithms and Policies Using Adaptive and Machine Learning Approaches](https://mlsanity.com/paper/2105.06577/) (2021)
- [NAG-GS: Semi-Implicit, Accelerated and Robust Stochastic Optimizer](https://mlsanity.com/paper/2209.14937/) (2022)
- [Fast symplectic integrator for Nesterov-type acceleration method](https://mlsanity.com/paper/2106.07620/) (2021)
- [Resonance in Weight Space: Covariate Shift Can Drive Divergence of SGD with Momentum](https://mlsanity.com/paper/2203.11992/) (2022)
- [Learning by solving differential equations](https://mlsanity.com/paper/2505.13397/) (2025)
- [A More Stable Accelerated Gradient Method Inspired by Continuous-Time Perspective](https://mlsanity.com/paper/2112.04922/) (2021)
- [Optimization using Parallel Gradient Evaluations on Multiple Parameters](https://mlsanity.com/paper/2302.03161/) (2023)
- [Reducing Discretization Error in the Frank-Wolfe Method](https://mlsanity.com/paper/2304.01432/) (2023)
- [Understanding Accelerated Gradient Methods: Lyapunov Analyses and Hamiltonian Assisted Interpretations](https://mlsanity.com/paper/2304.10063/) (2023)
- [Generalized Continuous-Time Models for Nesterov's Accelerated Gradient Methods](https://mlsanity.com/paper/2409.00913/) (2024)
- [Distributed Optimization via Energy Conservation Laws in Dilated Coordinates](https://mlsanity.com/paper/2409.19279/) (2024)
- [Adaptive Runge-Kutta Step Control Buys Training Loss, Not Generalization: An Honest Compute-Matched Study of RK-Adam Optimizers](https://mlsanity.com/paper/2607.14516/) (2026)

## Abstract (arXiv)

We study gradient-based optimization methods obtained by directly discretizing a second-order ordinary differential equation (ODE) related to the continuous limit of Nesterov's accelerated gradient method. When the function is smooth enough, we show that acceleration can be achieved by a stable discretization of this ODE using standard Runge-Kutta integrators. Specifically, we prove that under Lipschitz-gradient, convexity and order-(s+2) differentiability assumptions, the sequence of iterates generated by discretizing the proposed second-order ODE converges to the optimal solution at a rate of O(N^-2fracss+1), where s is the order of the Runge-Kutta numerical integrator. Furthermore, we introduce a new local flatness condition on the objective, under which rates even faster than O(N-2) can be achieved with low-order integrators and only gradient information. Notably, this flatness condition is satisfied by several standard loss functions used in machine learning. We provide numerical experiments that verify the theoretical rates predicted by our results.