# An explicit operator explains end-to-end computation in the modern neural networks used for sequence and language modeling

arXiv: https://arxiv.org/abs/2604.20595 · HTML: https://mlsanity.com/paper/2604.20595/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 43 references that are in the map.
- First posted on arXiv 22 Apr 2026.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention](https://mlsanity.com/paper/2006.16236/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023)
- [Efficient Transformers: A Survey](https://mlsanity.com/paper/2009.06732/) (2020)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Exact solutions to the nonlinear dynamics of learning in deep linear neural networks](https://mlsanity.com/paper/1312.6120/) (2013)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [On the Parameterization and Initialization of Diagonal State Space Models](https://mlsanity.com/paper/2206.11893/) (2022)
- [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023)
- [Diagonal State Spaces are as Effective as Structured State Spaces](https://mlsanity.com/paper/2203.14343/) (2022)
- [Theoretical Foundations of Deep Selective State-Space Models](https://mlsanity.com/paper/2402.19047/) (2024)
- [State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memory](https://mlsanity.com/paper/2309.13414/) (2023)
- [Oscillatory State-Space Models](https://mlsanity.com/paper/2410.03943/) (2024)
- [Traveling Waves Encode the Recent Past and Enhance Sequence Learning](https://mlsanity.com/paper/2309.08045/) (2023)
- [Image segmentation with traveling waves in an exactly solvable recurrent neural network](https://mlsanity.com/paper/2311.16943/) (2023)
- [Transformers and Cortical Waves: Encoders for Pulling In Context Across Time](https://mlsanity.com/paper/2401.14267/) (2024)
- [A Spatiotemporal Perspective on Dynamical Computation in Neural Information Processing Systems](https://mlsanity.com/paper/2409.13669/) (2024)
- [Hidden Traveling Waves bind Working Memory Variables in Recurrent Neural Networks](https://mlsanity.com/paper/2402.10163/) (2024)
- [Carleman Linearization of Nonlinear Systems and Its Finite-Section Approximations](https://mlsanity.com/paper/2207.07755/) (2022)
- [An exact mathematical description of computation with transient spatiotemporal dynamics in a complex-valued neural network](https://mlsanity.com/paper/2311.16431/) (2023)

## Abstract (arXiv)

We establish a mathematical correspondence between state space models, a state-of-the-art architecture for capturing long-range dependencies in data, and an exactly solvable nonlinear oscillator network. As a specific example of this general correspondence, we analyze the diagonal linear time-invariant implementation of the Structured State Space Sequence model (S4). The correspondence embeds S4D, a specific implementation of S4, into a ring network topology, in which recent inputs are encoded, as waves of activity traveling over the one-dimensional spatial layout of the network. We then derive an exact operator expression for the full forward pass of S4D, yielding an analytical characterization of its complete input-output map. This expression reveals that the nonlinear decoder in the system induces interactions between these information-carrying waves that enable classifying real-world sequences. These results generalize across modern SSM architectures, and show that they admit an exact mathematical description with a clear physical interpretation. These insights enable a new level of interpretability for these systems in terms of nonlinear oscillator networks.