# Accelerating Training in Artificial Neural Networks with Dynamic Mode Decomposition

arXiv: https://arxiv.org/abs/2006.14371 · HTML: https://mlsanity.com/paper/2006.14371/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jun 2020.

## Built on by (most cited first, 7 of 7)

- [An Operator Theoretic View on Pruning Deep Neural Networks](https://mlsanity.com/paper/2110.14856/) (2021)
- [Identifying Equivalent Training Dynamics](https://mlsanity.com/paper/2302.09160/) (2023)
- [QuACK: Accelerating Gradient-Based Quantum Optimization with Koopman Operator Learning](https://mlsanity.com/paper/2211.01365/) (2022)
- [Neural Capacitance: A New Perspective of Neural Network Selection via Edge Dynamics](https://mlsanity.com/paper/2201.04194/) (2022)
- [Distributional Spectral Diagnostics for Localizing Grokking Transitions](https://mlsanity.com/paper/2605.08237/) (2026)
- [The Underlying Correlated Dynamics in Neural Training](https://mlsanity.com/paper/2212.09040/) (2022)
- [Learning Linear Embeddings for Non-Linear Network Dynamics with Koopman Message Passing](https://mlsanity.com/paper/2305.09060/) (2023)

## Abstract (arXiv)

Training of deep neural networks (DNNs) frequently involves optimizing several millions or even billions of parameters. Even with modern computing architectures, the computational expense of DNN training can inhibit, for instance, network architecture design optimization, hyper-parameter studies, and integration into scientific research cycles. The key factor limiting performance is that both the feed-forward evaluation and the back-propagation rule are needed for each weight during optimization in the update rule. In this work, we propose a method to decouple the evaluation of the update rule at each weight. At first, Proper Orthogonal Decomposition (POD) is used to identify a current estimate of the principal directions of evolution of weights per layer during training based on the evolution observed with a few backpropagation steps. Then, Dynamic Mode Decomposition (DMD) is used to learn the dynamics of the evolution of the weights in each layer according to these principal directions. The DMD model is used to evaluate an approximate converged state when training the ANN. Afterward, some number of backpropagation steps are performed, starting from the DMD estimates, leading to an update to the principal directions and DMD model. This iterative process is repeated until convergence. By fine-tuning the number of backpropagation steps used for each DMD model estimation, a significant reduction in the number of operations required to train the neural networks can be achieved. In this paper, the DMD acceleration method will be explained in detail, along with the theoretical justification for the acceleration provided by DMD. This method is illustrated using a regression problem of key interest for the scientific machine learning community: the prediction of a pollutant concentration field in a diffusion, advection, reaction problem.