# Representing Neural Network Layers as Linear Operations via Koopman Operator Theory

arXiv: https://arxiv.org/abs/2409.01308 · HTML: https://mlsanity.com/paper/2409.01308/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 12 references that are in the map.
- First posted on arXiv 2 Sep 2024.

## Builds on (most cited first, 4 of 4)

- [Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations](https://mlsanity.com/paper/1710.10121/) (2017)
- [Multi-level Residual Networks from Dynamical Systems View](https://mlsanity.com/paper/1710.10348/) (2017)
- [Optimizing Neural Networks via Koopman Operator Theory](https://mlsanity.com/paper/2006.02361/) (2020)
- [Extraction of nonlinearity in neural networks with Koopman operator](https://mlsanity.com/paper/2402.11740/) (2024)

## Built on by (most cited first, 2 of 2)

- [Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture-of-Experts](https://mlsanity.com/paper/2609.02404/) (2026)
- [Dynamic Mode Decomposition along Depth in Vision Transformers](https://mlsanity.com/paper/2605.07556/) (2026)

## Abstract (arXiv)

The strong performance of simple neural networks is often attributed to their nonlinear activations. However, a linear view of neural networks makes understanding and controlling networks much more approachable. We draw from a dynamical systems view of neural networks, offering a fresh perspective by using Koopman operator theory and its connections with dynamic mode decomposition (DMD). Together, they offer a framework for linearizing dynamical systems by embedding the system into an appropriate observable space. By reframing a neural network as a dynamical system, we demonstrate that we can replace the nonlinear layer in a pretrained multi-layer perceptron (MLP) with a finite-dimensional linear operator. In addition, we analyze the eigenvalues of DMD and the right singular vectors of SVD, to present evidence that time-delayed coordinates provide a straightforward and highly effective observable space for Koopman theory to linearize a network layer. Consequently, we replace layers of an MLP trained on the Yin-Yang dataset with predictions from a DMD model, achieving a mdoel accuracy of up to 97.3%, compared to the original 98.4%. In addition, we replace layers in an MLP trained on the MNIST dataset, achieving up to 95.8%, compared to the original 97.2% on the test set.