# Learning to Optimize with Dynamic Mode Decomposition

arXiv: https://arxiv.org/abs/2211.16268 · HTML: https://mlsanity.com/paper/2211.16268/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 29 references that are in the map.
- First posted on arXiv 29 Nov 2022.

## Builds on (most cited first, 18 of 18)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Meta-Learning in Neural Networks: A Survey](https://mlsanity.com/paper/2004.05439/) (2020)
- [Learning to learn by gradient descent by gradient descent](https://mlsanity.com/paper/1606.04474/) (2016)
- [Learned Optimizers that Scale and Generalize](https://mlsanity.com/paper/1703.04813/) (2017)
- [Understanding Short-Horizon Bias in Stochastic Meta-Optimization](https://mlsanity.com/paper/1803.02021/) (2018)
- [Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves](https://mlsanity.com/paper/2009.11243/) (2020)
- [Learning Gradient Descent: Better Generalization and Longer Horizons](https://mlsanity.com/paper/1703.03633/) (2017)
- [Meta Learning Backpropagation And Improving It](https://mlsanity.com/paper/2012.14905/) (2020)
- [Optimizing Neural Networks via Koopman Operator Theory](https://mlsanity.com/paper/2006.02361/) (2020)
- [Training Stronger Baselines for Learning to Optimize](https://mlsanity.com/paper/2010.09089/) (2020)
- [Safeguarded Learned Convex Optimization](https://mlsanity.com/paper/2003.01880/) (2020)
- [Online dynamic mode decomposition for time-varying systems](https://mlsanity.com/paper/1707.02876/) (2017)
- [Training Learned Optimizers with Randomly Initialized Learned Optimizers](https://mlsanity.com/paper/2101.07367/) (2021)
- [Applications of Koopman Mode Analysis to Neural Networks](https://mlsanity.com/paper/2006.11765/) (2020)
- [BADGER: Learning to (Learn [Learning Algorithms] through Multi-Agent Communication)](https://mlsanity.com/paper/1912.01513/) (2019)
- [Self-referential basis of undecidable dynamics: from The Liar Paradox and The Halting Problem to The Edge of Chaos](https://mlsanity.com/paper/1711.02456/) (2017)

## Built on by (most cited first, 5 of 5)

- [Identifying Equivalent Training Dynamics](https://mlsanity.com/paper/2302.09160/) (2023)
- [Enhancing Neural Training via a Correlated Dynamics Model](https://mlsanity.com/paper/2312.13247/) (2023)
- [Learning Linear Embeddings for Non-Linear Network Dynamics with Koopman Message Passing](https://mlsanity.com/paper/2305.09060/) (2023)
- [Investigation into the Training Dynamics of Learned Optimizers](https://mlsanity.com/paper/2312.07174/) (2023)
- [Enhancing Fractional Gradient Descent with Learned Optimizers](https://mlsanity.com/paper/2510.18783/) (2025)

## Abstract (arXiv)

Designing faster optimization algorithms is of ever-growing interest. In recent years, learning to learn methods that learn how to optimize demonstrated very encouraging results. Current approaches usually do not effectively include the dynamics of the optimization process during training. They either omit it entirely or only implicitly assume the dynamics of an isolated parameter. In this paper, we show how to utilize the dynamic mode decomposition method for extracting informative features about optimization dynamics. By employing those features, we show that our learned optimizer generalizes much better to unseen optimization problems in short. The improved generalization is illustrated on multiple tasks where training the optimizer on one neural network generalizes to different architectures and distinct datasets.