Superposition unifies power-law training dynamics
Not on the map. Why?
Abstract (arXiv)
We investigate the role of feature superposition in the emergence of power-law training dynamics using a teacher-student framework. We first derive an analytic theory for training without superposition, establishing that the power-law training exponent depends on both the input data statistics and channel importance. Remarkably, we discover that a superposition bottleneck induces a transition to a universal power-law exponent of ~ 1, independent of data and channel statistics. This one over time training with superposition represents an up to tenfold acceleration compared to the purely sequential learning that takes place in the absence of superposition. Our finding that superposition leads to rapid training with a data-independent power law exponent may have important implications for a wide range of neural networks that employ superposition, including production-scale large language models.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 33 of its 37 references that are in the map. First posted on arXiv 1 Feb 2026.
Builds on
33 of 37 references.
References · 33
- 2023GPT-4 Technical Reportcited by 18,132
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2022Emergent Abilities of Large Language Modelscited by 2,483
- 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
- 2018On the Spectral Bias of Neural Networkscited by 1,321
- 2021Scaling Vision Transformerscited by 993
- 2022Toy Models of Superpositioncited by 928
- 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
- 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
21 more
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
- 2018On Lazy Training in Differentiable Programmingcited by 671
- 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
- 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
- 2024Scaling and evaluating sparse autoencoderscited by 530
- 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 519
- 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
- 2021Scaling Laws for Transfercited by 265
- 2022Unified Scaling Laws for Routed Language Modelscited by 249
- 2017High-dimensional dynamics of generalization error in neural networkscited by 238
- 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
- 2018A mathematical theory of semantic development in deep neural networkscited by 182
- 2020Feature Learning in Infinite-Width Neural Networkscited by 182
- 2020Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networkscited by 166
- 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
- 2022A Solvable Model of Neural Scaling Lawscited by 101
- 2021Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regimecited by 95
- 2024How Feature Learning Can Improve Neural Scaling Lawscited by 55
- 2020A Neural Scaling Law from the Dimension of the Data Manifoldcited by 48
- 2025Superposition Yields Robust Neural Scalingcited by 30
- 2025Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Timecited by 15
Built on by
Citing papers per year: 2026: 1.
1 paper.