# Dynamics of the Transformer Residual Stream: Coupling Spectral Geometry to Network Topology

arXiv: https://arxiv.org/abs/2605.14258 · HTML: https://mlsanity.com/paper/2605.14258/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 32 references that are in the map.
- First posted on arXiv 14 May 2026.

## Builds on (most cited first, 26 of 26)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [From Louvain to Leiden: guaranteeing well-connected communities](https://mlsanity.com/paper/1810.08473/) (2018)
- [Exponential expressivity in deep neural networks through transient chaos](https://mlsanity.com/paper/1606.05340/) (2016)
- [Deep Information Propagation](https://mlsanity.com/paper/1611.01232/) (2016)
- [Deep Neural Networks Motivated by Partial Differential Equations](https://mlsanity.com/paper/1804.04272/) (2018)
- [A mathematical perspective on Transformers](https://mlsanity.com/paper/2312.10794/) (2023)
- [Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice](https://mlsanity.com/paper/1711.04735/) (2017)
- [Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View](https://mlsanity.com/paper/1906.02762/) (2019)
- [Parcae: Scaling Laws For Stable Looped Language Models](https://mlsanity.com/paper/2604.12946/) (2026)
- [Large language models implicitly learn to straighten neural sentence trajectories to construct a predictive representation of natural language](https://mlsanity.com/paper/2311.04930/) (2023)
- [Block-Recurrent Dynamics in Vision Transformers](https://mlsanity.com/paper/2512.19941/) (2025)
- [Residual Stream Analysis with Multi-Layer SAEs](https://mlsanity.com/paper/2409.04185/) (2024)
- [Finite-time Lyapunov exponents of deep neural networks](https://mlsanity.com/paper/2306.12548/) (2023)
- [Transformer Dynamics: A neuroscientific approach to interpretability of large language models](https://mlsanity.com/paper/2502.12131/) (2025)
- [Transformer Block Coupling and its Correlation with Generalization in LLMs](https://mlsanity.com/paper/2407.07810/) (2024)
- [Understanding Community Structure in Layered Neural Networks](https://mlsanity.com/paper/1804.04778/) (2018)
- [Gradient Flossing: Improving Gradient Descent through Dynamic Control of Jacobians](https://mlsanity.com/paper/2312.17306/) (2023)
- [Homogenized Transformers](https://mlsanity.com/paper/2604.01978/) (2026)
- [Spectral Conditioning of Attention Improves Transformer Performance](https://mlsanity.com/paper/2603.07162/) (2026)
- [CAST: Compositional Analysis via Spectral Tracking for Understanding Transformer Layer Functions](https://mlsanity.com/paper/2510.14262/) (2025)
- [SCORE: Replacing Layer Stacking with Contractive Recurrent Depth](https://mlsanity.com/paper/2603.10544/) (2026)

## Built on by (most cited first, 5 of 5)

- [Off-Axis, On Purpose: Where a Transformer Computes Concepts and Why it Does So](https://mlsanity.com/paper/2608.10251/) (2026)
- [Fingerprint, Not Blueprint: How Positional Schemes Set the Default Spectral Algebra of Attention](https://mlsanity.com/paper/2607.06621/) (2026)
- [Relevant and Irrelevant: A Renormalization Group Analysis of Transformer Attention](https://mlsanity.com/paper/2607.15449/) (2026)
- [The Depth Flow of Token Representations Is Nonlinear and Does Not Descend Its Own Density](https://mlsanity.com/paper/2608.29706/) (2026)
- [Pretraining Shapes Spectral Structure: Architecture- and Strategy-Conditional Prediction of OOD Robustness in Foundation Models](https://mlsanity.com/paper/2610.09709/) (2026)

## Abstract (arXiv)

Large language models are remarkably capable, yet how computation propagates through their layers remains poorly understood. A growing line of work treats depth as discrete time and the residual stream as a dynamical system, where each layer's nonlinear update has a local linear description. However, previous analyses have relied on scalar summaries or approximate linearizations, leaving the full spectral geometry of trained LLMs unknown. We perform full Jacobian eigendecomposition across three production--scale LLMs and show that training installs a monotonic spectral gradient through depth -- from non-normal, rotation-dominated early layers to near--symmetric late layers -- together with a cumulative low-rank bottleneck that funnels perturbations into a small fraction of the residual stream's effective dimensions. Our experiments reveal that this gradient and the dimensional collapse are learned rather than architectural, and is largely dissolved when structured non-normality is removed. We further show that the topological positioning of graph communities predicts whether the Jacobian amplifies or suppresses them, with the sign of the coupling determined by the local operator type, a relationship absent at initialization. These results map a learned spectral geometry in LLMs that links perturbation propagation and compression to the network's functional topology.