# Universal One-third Time Scaling in Learning Peaked Distributions

arXiv: https://arxiv.org/abs/2602.03685 · HTML: https://mlsanity.com/paper/2602.03685/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 46 references that are in the map.
- First posted on arXiv 3 Feb 2026.

## Builds on (most cited first, 30 of 35)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale](https://mlsanity.com/paper/2406.17557/) (2024)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Scaling Laws for Autoregressive Generative Modeling](https://mlsanity.com/paper/2010.14701/) (2020)
- [Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability](https://mlsanity.com/paper/2103.00065/) (2021)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [2 OLMo 2 Furious](https://mlsanity.com/paper/2501.00656/) (2024)
- [Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks](https://mlsanity.com/paper/2002.02561/) (2020)
- [The Quantization Model of Neural Scaling](https://mlsanity.com/paper/2303.13506/) (2023)
- [A Solvable Model of Neural Scaling Laws](https://mlsanity.com/paper/2210.16859/) (2022)
- [A Theory for Emergence of Complex Skills in Language Models](https://mlsanity.com/paper/2307.15936/) (2023)
- [A Dynamical Model of Neural Scaling Laws](https://mlsanity.com/paper/2402.01092/) (2024)
- [Neural Networks as Kernel Learners: The Silent Alignment Effect](https://mlsanity.com/paper/2111.00034/) (2021)
- [Learning Curve Theory](https://mlsanity.com/paper/2102.04074/) (2021)
- [Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective](https://mlsanity.com/paper/2410.05192/) (2024)
- [nGPT: Normalized Transformer with Representation Learning on the Hypersphere](https://mlsanity.com/paper/2410.01131/) (2024)
- [4+3 Phases of Compute-Optimal Neural Scaling Laws](https://mlsanity.com/paper/2405.15074/) (2024)
- [Scaling Laws in Linear Regression: Compute, Parameters, and Data](https://mlsanity.com/paper/2406.08466/) (2024)
- [How Feature Learning Can Improve Neural Scaling Laws](https://mlsanity.com/paper/2409.17858/) (2024)
- [Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning](https://mlsanity.com/paper/2406.06158/) (2024)
- [Superposition Yields Robust Neural Scaling](https://mlsanity.com/paper/2505.10465/) (2025)
- [An exactly solvable model for emergence and scaling laws in the multitask sparse parity problem](https://mlsanity.com/paper/2404.17563/) (2024)
- [Scaling Laws and Spectra of Shallow Neural Networks in the Feature Learning Regime](https://mlsanity.com/paper/2509.24882/) (2025)
- [Theory of Scaling Laws for In-Context Regression: Depth, Width, Context and Time](https://mlsanity.com/paper/2510.01098/) (2025)
- [Neural Scaling Laws Rooted in the Data Distribution](https://mlsanity.com/paper/2412.07942/) (2024)

## Built on by (most cited first, 10 of 10)

- [Inverse Depth Scaling From Most Layers Being Similar](https://mlsanity.com/paper/2602.05970/) (2026)
- [Neural Scaling Universality: If Exponents Are Fixed, Time to Understand Coefficients](https://mlsanity.com/paper/2606.25008/) (2026)
- [Jailbreak Scaling Laws for Large Language Models: Polynomial-Exponential Crossover](https://mlsanity.com/paper/2603.11331/) (2026)
- [Power-Law Spectrum of the Random Feature Model](https://mlsanity.com/paper/2603.14578/) (2026)
- [A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classification](https://mlsanity.com/paper/2605.22341/) (2026)
- [Attractor Geometry of Transformer Memory: From Conflict Arbitration to Confident Hallucination](https://mlsanity.com/paper/2605.05686/) (2026)
- [Emergent One-Third Scaling Law as Attention Tries to Concentrate](https://mlsanity.com/paper/2609.32100/) (2026)
- [A Solvable Theory of Pre-training Data Poisoning: Regime-Dependent Scaling Exponents](https://mlsanity.com/paper/2609.32288/) (2026)
- [Optimizer-dependent training dynamics converge to the same one-third optimal data scaling](https://mlsanity.com/paper/2609.37745/) (2026)
- [From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes](https://mlsanity.com/paper/2609.40148/) (2026)

## Abstract (arXiv)

Training large language models (LLMs) is computationally expensive, partly because the loss exhibits slow power-law convergence whose origin remains debatable. Through systematic analysis of toy models and empirical evaluation of LLMs, we show that this behavior can arise intrinsically from the use of softmax and cross-entropy. When learning peaked probability distributions, e.g., next-token distributions, these components generically yield power-law vanishing losses and gradients, regardless of many microscopic details, creating a fundamental optimization bottleneck. This ultimately leads to power-law time scaling of the loss with a universal exponent of 1/3. Our results provide a mechanistic explanation for observed neural scaling and suggest new directions for improving LLM training efficiency.