mlsanityData to 14 Oct 2026

The Role of Feed-Forward Layers in Transformer Dynamics

arXiv 2609.36230 ↗First posted 28 Sep 2026cs.LG math.DScited by 0

Not on the map. Why?

Abstract (arXiv)

We study the dynamical behavior of tokens in transformers from a control-theoretic perspective. Our model includes the feed-forward layer present after the self-attention mechanism, with the self-attention mechanism interpreted as an interacting particle system and the feed-forward layer as an independent control. Our main theoretical result establishes that the feed-forward network can steer the tokens arbitrarily close to consensus regardless of the key, query, and value matrices. Our result are easily extended to convergence to many clusters and to multi-head attention. We conduct numerical experiments to verify our results, and compare thresholding behavior from our theory to real-world LLMs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 15 of its 22 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

15 of 22 references.

References · 15

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2023A mathematical perspective on Transformerscited by 164
  7. 2024Dynamic metastability in the self-attention modelcited by 46
  8. 2024Transformers are Universal In-context Learnerscited by 43
  9. 2024Emergence of meta-stable clustering in mean-field transformer modelscited by 39
  10. 2025A Unified Perspective on the Dynamics of Deep Transformerscited by 32
  11. 2025A multiscale analysis of mean-field transformers in the moderate interaction regimecited by 25
  12. 2026Perceptrons and localization of attention's mean-field landscapecited by 8
3 more
  1. 2026Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regimecited by 5
  2. 2026On the Diverse Dynamical Behaviors Arising in Deep Linear Transformerscited by 3
  3. 2025Universal Approximation Theorem for a Single-Layer Transformercited by 1

Built on by

Not cited yet.