# Optimizing Neural Networks via Koopman Operator Theory

arXiv: https://arxiv.org/abs/2006.02361 · HTML: https://mlsanity.com/paper/2006.02361/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Jun 2020.

## Built on by (most cited first, 30 of 33)

- [Generative Modeling of Regular and Irregular Time Series Data via Koopman VAEs](https://mlsanity.com/paper/2310.02619/) (2023)
- [Noisy Recurrent Neural Networks](https://mlsanity.com/paper/2102.04877/) (2021)
- [Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities](https://mlsanity.com/paper/2210.06640/) (2022)
- [An Operator Theoretic View on Pruning Deep Neural Networks](https://mlsanity.com/paper/2110.14856/) (2021)
- [Temporally Consistent Koopman Autoencoders for Forecasting Dynamical Systems](https://mlsanity.com/paper/2403.12335/) (2024)
- [An Operator Theoretic Approach for Analyzing Sequence Neural Networks](https://mlsanity.com/paper/2102.07824/) (2021)
- [On the generalization of learning algorithms that do not converge](https://mlsanity.com/paper/2208.07951/) (2022)
- [Koopman-based generalization bound: New aspect for full-rank weights](https://mlsanity.com/paper/2302.05825/) (2023)
- [Identifying Equivalent Training Dynamics](https://mlsanity.com/paper/2302.09160/) (2023)
- [One-Step Offline Distillation of Diffusion-based Models via Koopman Modeling](https://mlsanity.com/paper/2505.13358/) (2025)
- [QuACK: Accelerating Gradient-Based Quantum Optimization with Koopman Operator Learning](https://mlsanity.com/paper/2211.01365/) (2022)
- [Learning to Optimize with Dynamic Mode Decomposition](https://mlsanity.com/paper/2211.16268/) (2022)
- [Enhancing Neural Training via a Correlated Dynamics Model](https://mlsanity.com/paper/2312.13247/) (2023)
- [Stable Invariant Models via Koopman Spectra](https://mlsanity.com/paper/2207.07475/) (2022)
- [Neural Capacitance: A New Perspective of Neural Network Selection via Edge Dynamics](https://mlsanity.com/paper/2201.04194/) (2022)
- [Maximum Likelihood Learning of Latent Dynamics Without Reconstruction](https://mlsanity.com/paper/2505.23569/) (2025)
- [Representing Neural Network Layers as Linear Operations via Koopman Operator Theory](https://mlsanity.com/paper/2409.01308/) (2024)
- [Distributional Spectral Diagnostics for Localizing Grokking Transitions](https://mlsanity.com/paper/2605.08237/) (2026)
- [KFTD: Koopman-Fourier Time-Differentiable Network for Continuous Ocean Spatiotemporal Forecasting](https://mlsanity.com/paper/2606.17070/) (2026)
- [The Underlying Correlated Dynamics in Neural Training](https://mlsanity.com/paper/2212.09040/) (2022)
- [Enhancing Neural Network Differential Equation Solvers](https://mlsanity.com/paper/2301.13146/) (2022)
- [Learning Linear Embeddings for Non-Linear Network Dynamics with Koopman Message Passing](https://mlsanity.com/paper/2305.09060/) (2023)
- [Iterative Magnitude Pruning as a Renormalisation Group: A Study in The Context of The Lottery Ticket Hypothesis](https://mlsanity.com/paper/2308.03128/) (2023)
- [Koopman-Based Generalization of Deep Reinforcement Learning With Application to Wireless Communications](https://mlsanity.com/paper/2503.02961/) (2025)
- [Accelerating Learned Image Compression Through Modeling Neural Training Dynamics](https://mlsanity.com/paper/2505.18107/) (2025)
- [Transferability of Winning Lottery Tickets in Neural Network Differential Equation Solvers](https://mlsanity.com/paper/2306.09863/) (2023)
- [Glocal Hypergradient Estimation with Koopman Operator](https://mlsanity.com/paper/2402.02741/) (2024)
- [FINDER: Feature Inference on Noisy Datasets using Eigenspace Residuals](https://mlsanity.com/paper/2510.19917/) (2025)
- [Generalization Analysis and Method for Domain Generalization for a Family of Recurrent Neural Networks](https://mlsanity.com/paper/2601.08122/) (2026)
- [Man, Machine, and Mathematics](https://mlsanity.com/paper/2604.27052/) (2026)

## Abstract (arXiv)

Koopman operator theory, a powerful framework for discovering the underlying dynamics of nonlinear dynamical systems, was recently shown to be intimately connected with neural network training. In this work, we take the first steps in making use of this connection. As Koopman operator theory is a linear theory, a successful implementation of it in evolving network weights and biases offers the promise of accelerated training, especially in the context of deep networks, where optimization is inherently a non-convex problem. We show that Koopman operator theoretic methods allow for accurate predictions of weights and biases of feedforward, fully connected deep networks over a non-trivial range of training time. During this window, we find that our approach is >10x faster than various gradient descent based methods (e.g. Adam, Adadelta, Adagrad), in line with our complexity analysis. We end by highlighting open questions in this exciting intersection between dynamical systems and neural network theory. We highlight additional methods by which our results could be expanded to broader classes of networks and larger training intervals, which shall be the focus of future work.