# Flowing Through Layers: A Continuous Dynamical Systems Perspective on Transformers

arXiv: https://arxiv.org/abs/2502.05656 · HTML: https://mlsanity.com/paper/2502.05656/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 11 references that are in the map.
- First posted on arXiv 8 Feb 2025.

## Builds on (most cited first, 8 of 8)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Stable Architectures for Deep Neural Networks](https://mlsanity.com/paper/1705.03341/) (2017)
- [Parseval Networks: Improving Robustness to Adversarial Examples](https://mlsanity.com/paper/1704.08847/) (2017)
- [Deep Neural Networks Motivated by Partial Differential Equations](https://mlsanity.com/paper/1804.04272/) (2018)
- [Iterate to Accelerate: A Unified Framework for Iterative Reasoning and Feedback Convergence](https://mlsanity.com/paper/2502.03787/) (2025)

## Built on by (most cited first, 2 of 2)

- [Latent Flow Transformer](https://mlsanity.com/paper/2505.14513/) (2025)
- [Neuronal Group Communication for Efficient Neural representation](https://mlsanity.com/paper/2510.16851/) (2025)

## Abstract (arXiv)

We show that the standard discrete update rule of transformer layers can be naturally interpreted as a forward Euler discretization of a continuous dynamical system. Our Transformer Flow Approximation Theorem demonstrates that, under standard Lipschitz continuity assumptions, token representations converge uniformly to the unique solution of an ODE as the number of layers grows. Moreover, if the underlying mapping satisfies a one-sided Lipschitz condition with a negative constant, the resulting dynamics are contractive, causing perturbations to decay exponentially across layers. Beyond clarifying the empirical stability and expressivity of transformer models, these insights link transformer updates to a broader iterative reasoning framework, suggesting new avenues for accelerated convergence and architectural innovations inspired by dynamical systems theory.