mlsanityData to 7 Oct 2026

Learning to Optimize with Dynamic Mode Decomposition

arXiv 2211.16268 ↗First posted 29 Nov 2022cs.LG math.OCcited by 5

Not on the map. Why?

Abstract (arXiv)

Designing faster optimization algorithms is of ever-growing interest. In recent years, learning to learn methods that learn how to optimize demonstrated very encouraging results. Current approaches usually do not effectively include the dynamics of the optimization process during training. They either omit it entirely or only implicitly assume the dynamics of an isolated parameter. In this paper, we show how to utilize the dynamic mode decomposition method for extracting informative features about optimization dynamics. By employing those features, we show that our learned optimizer generalizes much better to unseen optimization problems in short. The improved generalization is illustrated on multiple tasks where training the optimizer on one neural network generalizes to different architectures and distinct datasets.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 18 of its 29 references that are in the map. First posted on arXiv 29 Nov 2022.

Builds on

18 of 29 references.

References · 18

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2020Meta-Learning in Neural Networks: A Surveycited by 944
  5. 2016Learning to learn by gradient descent by gradient descentcited by 616
  6. 2017Learned Optimizers that Scale and Generalizecited by 109
  7. 2018Understanding Short-Horizon Bias in Stochastic Meta-Optimizationcited by 68
  8. 2020Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselvescited by 57
  9. 2017Learning Gradient Descent: Better Generalization and Longer Horizonscited by 50
  10. 2020Meta Learning Backpropagation And Improving Itcited by 49
  11. 2020Optimizing Neural Networks via Koopman Operator Theorycited by 32
  12. 2020Training Stronger Baselines for Learning to Optimizecited by 31
6 more
  1. 2020Safeguarded Learned Convex Optimizationcited by 24
  2. 2017Online dynamic mode decomposition for time-varying systemscited by 22
  3. 2021Training Learned Optimizers with Randomly Initialized Learned Optimizerscited by 13
  4. 2020Applications of Koopman Mode Analysis to Neural Networkscited by 12
  5. 2019BADGER: Learning to (Learn [Learning Algorithms] through Multi-Agent Communication)cited by 5
  6. 2017Self-referential basis of undecidable dynamics: from The Liar Paradox and The Halting Problem to The Edge of Chaoscited by 3

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 0; 2025: 1.

5 papers.

  1. 2023Identifying Equivalent Training Dynamicscited by 7
  2. 2023Enhancing Neural Training via a Correlated Dynamics Modelcited by 5
  3. 2023Learning Linear Embeddings for Non-Linear Network Dynamics with Koopman Message Passingcited by 1
  4. 2023Investigation into the Training Dynamics of Learned Optimizerscited by 0
  5. 2025Enhancing Fractional Gradient Descent with Learned Optimizerscited by 0