mlsanityData to 14 Oct 2026

Normalization in Attention Dynamics

arXiv 2510.22026 ↗First posted 24 Oct 2025cs.LG cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

We study the effect of normalization schemes on token representations in deep transformers. Modeling their evolution as interacting particles on the sphere, we show that normalization acts as a form of speed regulation. This perspective enables a unified analysis of several schemes -- including Post-LN, Pre-LN, Mix-LN, Peri-LN, nGPT -- revealing how they influence clustering dynamics and representation collapse. Our framework clarifies how different schemes shape token representations across layers and provides a principled basis for comparing them, identifying Peri-LN as a particularly effective choice.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 26 of its 33 references that are in the map. First posted on arXiv 24 Oct 2025.

Builds on

26 of 33 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  4. 2016Identity Mappings in Deep Residual Networkscited by 1,965
  5. 2019Root Mean Square Layer Normalizationcited by 1,480
  6. 2020On Layer Normalization in the Transformer Architecturecited by 922
  7. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 226
  8. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  9. 2023A mathematical perspective on Transformerscited by 164
  10. 2024Compact Language Models via Pruning and Knowledge Distillationcited by 148
  11. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
  12. 2023The emergence of clusters in self-attention dynamicscited by 140
14 more
  1. 2021Sinkformers: Transformers with Doubly Stochastic Attentioncited by 138
  2. 2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 70
  3. 2025The Curse of Depth in Large Language Modelscited by 62
  4. 2024Dynamic metastability in the self-attention modelcited by 46
  5. 2024Emergence of meta-stable clustering in mean-field transformer modelscited by 39
  6. 2024Mix-LN: Unleashing the Power of Deeper Layers by Combining Pre-LN and Post-LNcited by 37
  7. 2024Clustering in Causal Attention Maskingcited by 32
  8. 2024Measure-to-measure interpolation using Transformerscited by 29
  9. 2025Quantitative Clustering in Mean-Field Transformer Modelscited by 26
  10. 2025A multiscale analysis of mean-field transformers in the moderate interaction regimecited by 25
  11. 2024Geometric Dynamics of Signal Propagation Predict Trainability of Transformerscited by 24
  12. 2024A deeper look at depth pruning of LLMscited by 23
  13. 2024Transformers Get Stable: An End-to-End Signal Propagation Theory for Language Modelscited by 18
  14. 2024Synchronization on circles and spheres with nonlinear interactionscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 8.

9 papers.

  1. 2025The Mean-Field Dynamics of Transformerscited by 30
  2. 2026Clustering in Deep Stochastic Transformerscited by 8
  3. 2026Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regimecited by 5
  4. 2026Explaining Grokking in Transformers through the Lens of Inductive Biascited by 2
  5. 2026Gradient Flow Structure and Quantitative Dynamics of Multi-Head Self-Attentioncited by 2
  6. 2026Polaris: Coupled Orbital Polar Embeddings for Hierarchical Concept Learningcited by 1
  7. 2026Feed-Forward Steering in Transformer Residual Dynamicscited by 1
  8. 2026When Does Removing LayerNorm Help? Activation Bounding as a Regime-Dependent Implicit Regularizercited by 0
  9. 2026Self-Attention Dynamics with Rotary Position Embeddings: Twisted States and Explicit Consensus Rates on the Spherecited by 0