# The Role of Feed-Forward Layers in Transformer Dynamics

arXiv: https://arxiv.org/abs/2609.36230 · HTML: https://mlsanity.com/paper/2609.36230/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 22 references that are in the map.
- First posted on arXiv 28 Sep 2026.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [A mathematical perspective on Transformers](https://mlsanity.com/paper/2312.10794/) (2023)
- [Dynamic metastability in the self-attention model](https://mlsanity.com/paper/2410.06833/) (2024)
- [Transformers are Universal In-context Learners](https://mlsanity.com/paper/2408.01367/) (2024)
- [Emergence of meta-stable clustering in mean-field transformer models](https://mlsanity.com/paper/2410.23228/) (2024)
- [A Unified Perspective on the Dynamics of Deep Transformers](https://mlsanity.com/paper/2501.18322/) (2025)
- [A multiscale analysis of mean-field transformers in the moderate interaction regime](https://mlsanity.com/paper/2509.25040/) (2025)
- [Perceptrons and localization of attention's mean-field landscape](https://mlsanity.com/paper/2601.21366/) (2026)
- [Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime](https://mlsanity.com/paper/2605.10931/) (2026)
- [On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers](https://mlsanity.com/paper/2607.18584/) (2026)
- [Universal Approximation Theorem for a Single-Layer Transformer](https://mlsanity.com/paper/2507.10581/) (2025)

## Abstract (arXiv)

We study the dynamical behavior of tokens in transformers from a control-theoretic perspective. Our model includes the feed-forward layer present after the self-attention mechanism, with the self-attention mechanism interpreted as an interacting particle system and the feed-forward layer as an independent control. Our main theoretical result establishes that the feed-forward network can steer the tokens arbitrarily close to consensus regardless of the key, query, and value matrices. Our result are easily extended to convergence to many clusters and to multi-head attention. We conduct numerical experiments to verify our results, and compare thresholding behavior from our theory to real-world LLMs.