# Hidden Dynamics of Massive Activations in Transformer Training

arXiv: https://arxiv.org/abs/2508.03616 · HTML: https://mlsanity.com/paper/2508.03616/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 45 references that are in the map.
- First posted on arXiv 5 Aug 2025.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Massive Activations in Large Language Models](https://mlsanity.com/paper/2402.17762/) (2024)
- [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022)
- [DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs](https://mlsanity.com/paper/2406.01721/) (2024)
- [Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing](https://mlsanity.com/paper/2306.12929/) (2023)
- [Dive into Deep Learning](https://mlsanity.com/paper/2106.11342/) (2021)
- [Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understanding](https://mlsanity.com/paper/2502.01563/) (2025)
- [Peri-LN: Revisiting Normalization Layer in the Transformer Architecture](https://mlsanity.com/paper/2502.02732/) (2025)
- [Systematic Outliers in Large Language Models](https://mlsanity.com/paper/2502.06415/) (2025)
- [Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization](https://mlsanity.com/paper/2404.03605/) (2024)
- [Unleashing Diffusion Transformers for Visual Correspondence by Modulating Massive Activations](https://mlsanity.com/paper/2505.18584/) (2025)
- [Softpick: No Attention Sink, No Massive Activations with Rectified Softmax](https://mlsanity.com/paper/2504.20966/) (2025)
- [Mitigating Quantization Errors Due to Activation Spikes in GLU-Based LLMs](https://mlsanity.com/paper/2405.14428/) (2024)
- [Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversion](https://mlsanity.com/paper/2310.04361/) (2023)
- [Tracking the Feature Dynamics in LLM Training: A Mechanistic Study](https://mlsanity.com/paper/2412.17626/) (2024)
- [From Attention to Activation: Unravelling the Enigmas of Large Language Models](https://mlsanity.com/paper/2410.17174/) (2024)
- [Activation Control for Efficiently Eliciting Long Chain-of-thought Ability of Language Models](https://mlsanity.com/paper/2505.17697/) (2025)
- [A Refined Analysis of Massive Activations in LLMs](https://mlsanity.com/paper/2503.22329/) (2025)
- [House of Cards: Massive Weights in LLMs](https://mlsanity.com/paper/2410.01866/) (2024)
- [First Activations Matter: Training-Free Methods for Dynamic Activation in Large Language Models](https://mlsanity.com/paper/2408.11393/) (2024)

## Built on by (most cited first, 10 of 10)

- [TWEO: Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies](https://mlsanity.com/paper/2511.23225/) (2025)
- [How Do Transformers Learn to Associate Tokens: Gradient Leading Terms Bring Mechanistic Interpretability](https://mlsanity.com/paper/2601.19208/) (2026)
- [Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformers](https://mlsanity.com/paper/2603.17771/) (2026)
- [A Single Layer to Explain Them All:Understanding Massive Activations in Large Language Models](https://mlsanity.com/paper/2605.08504/) (2026)
- [Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus](https://mlsanity.com/paper/2608.12149/) (2026)
- [A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventions](https://mlsanity.com/paper/2606.08105/) (2026)
- [Transformer MLP Gate Thresholds Are Couplings to a Carried Reference Direction](https://mlsanity.com/paper/2609.31956/) (2026)
- [Which the Eye Fears: Writing with Read-Blindness Explains Massive Activations in Transformers](https://mlsanity.com/paper/2609.35630/) (2026)
- [The Life Cycle of a Massive Activation: Stochastic Birth, Weight-Decay-Driven Growth, and Competitive Consolidation](https://mlsanity.com/paper/2610.00423/) (2026)
- [Fault-tolerant foundation models](https://mlsanity.com/paper/2610.10311/) (2026)

## Abstract (arXiv)

We present the first comprehensive analysis of massive activation development throughout transformer training, using the Pythia model family as our testbed, and release our full dataset publicly to support further research. Through systematic analysis of various model sizes across multiple training checkpoints, we demonstrate that massive activation emergence follows highly predictable mathematical patterns that can be accurately modeled using an exponentially-modulated logarithmic function with five key parameters. Additionally, We develop a machine learning framework to predict these mathematical parameters from architectural specifications alone, achieving high accuracy for steady-state behavior and moderate accuracy for emergence timing and magnitude. These findings enable architects to predict and potentially control key aspects of massive activation emergence through design choices, with significant implications for model stability, training cycle length, interpretability, and optimization. Our findings demonstrate that the emergence of massive activations is governed by model design and can be anticipated, and potentially controlled, before training begins. Code is available at https://github.com/Aimpoint-Digital/massive-activations-fork