Optimization as a Dynamical System: Generative Schedules from Latent ODEs
Not on the map. Why?
Abstract (arXiv)
We present a new meta-learning method to determine the optimal learning rate schedule for gradient descent. It leverages training runs from a hyperparameter search to learn a latent representation of the training process, which is modeled as a dynamical system. Given current training metrics, it predicts the future learning rate schedule with the best long-term validation performance. Our scheduler generalizes beyond previously observed training dynamics and creates specialized schedules that deviate noticeably from even the best-performing parametric functions. It outperforms all baselines we compare to on results for image classification with CNN and ResNet models as well as for next-token prediction with a transformer model. The trained models are located in flatter regions of the loss landscape and thus provide better generalization than those trained with other schedules. Our method is computationally efficient, optimizer-agnostic, and can easily be layered on top of ML experiment-tracking platforms to streamline training of neural networks from scratch.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 46 references that are in the map. First posted on arXiv 27 Sep 2025.
Builds on
34 of 46 references.
References · 34
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
- 2018Neural Ordinary Differential Equationscited by 3,779
- 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
- 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
- 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
- 2017Sharp Minima Can Generalize For Deep Netscited by 466
- 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
- 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
- 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 359
22 more
- 2017Don't Decay the Learning Rate, Increase the Batch Sizecited by 320
- 2014The Loss Surfaces of Multilayer Networkscited by 280
- 2017Three Factors Influencing Minima in SGDcited by 273
- 2022On Neural Differential Equationscited by 268
- 2024The Road Less Scheduledcited by 163
- 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 150
- 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
- 2016Matrix Completion has No Spurious Local Minimumcited by 125
- 2019An Exponential Learning Rate Schedule for Deep Learningcited by 121
- 2017Online Learning Rate Adaptation with Hypergradient Descentcited by 118
- 2018A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillationcited by 110
- 2018On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Lengthcited by 88
- 2020Relative Flatness and Generalizationcited by 88
- 2022On the Maximum Hessian Eigenvalue and Generalizationcited by 57
- 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 43
- 2023Optimal Linear Decay Learning Rate Schedules and Further Refinementscited by 43
- 2019Learning an Adaptive Learning Rate Schedulecited by 29
- 2018WNGrad: Learn the Learning Rate in Gradient Descentcited by 24
- 2019Demystifying Learning Rate Policies for High Accuracy Training of Deep Neural Networkscited by 12
- 2023Modulated Neural ODEscited by 10
- 2024Architecture-Aware Learning Curve Extrapolation via Graph Ordinary Differential Equationcited by 5
- 2024Path-minimizing Latent ODEs for improved extrapolation and inferencecited by 2
Built on by
Not cited yet.