# A Boundary-Layer Mechanism for One-Third Scaling in Online Softmax Classification

arXiv: https://arxiv.org/abs/2605.22341 · HTML: https://mlsanity.com/paper/2605.22341/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 36 references that are in the map.
- First posted on arXiv 21 May 2026.

## Builds on (most cited first, 25 of 25)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [Deep Learning Scaling is Predictable, Empirically](https://mlsanity.com/paper/1712.00409/) (2017)
- [Explaining Neural Scaling Laws](https://mlsanity.com/paper/2102.06701/) (2021)
- [High-dimensional dynamics of generalization error in neural networks](https://mlsanity.com/paper/1710.03667/) (2017)
- [Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks](https://mlsanity.com/paper/2002.02561/) (2020)
- [Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks](https://mlsanity.com/paper/2006.13198/) (2020)
- [Learning curves of generic features maps for realistic datasets with a teacher-student model](https://mlsanity.com/paper/2102.08127/) (2021)
- [A Solvable Model of Neural Scaling Laws](https://mlsanity.com/paper/2210.16859/) (2022)
- [Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup](https://mlsanity.com/paper/1906.08632/) (2019)
- [A Dynamical Model of Neural Scaling Laws](https://mlsanity.com/paper/2402.01092/) (2024)
- [Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Rate](https://mlsanity.com/paper/1806.01796/) (2018)
- [Scaling Laws in Linear Regression: Compute, Parameters, and Data](https://mlsanity.com/paper/2406.08466/) (2024)
- [How Feature Learning Can Improve Neural Scaling Laws](https://mlsanity.com/paper/2409.17858/) (2024)
- [Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimization](https://mlsanity.com/paper/2006.06560/) (2020)
- [A Universal Growth Rate for Learning with Smooth Surrogate Losses](https://mlsanity.com/paper/2405.05968/) (2024)
- [Learning curves for the multi-class teacher-student perceptron](https://mlsanity.com/paper/2203.12094/) (2022)
- [On the origin of neural scaling laws: from random graphs to natural language](https://mlsanity.com/paper/2601.10684/) (2026)
- [The Implicit Bias of Gradient Descent on Separable Multiclass Data](https://mlsanity.com/paper/2411.01350/) (2024)
- [Universal One-third Time Scaling in Learning Peaked Distributions](https://mlsanity.com/paper/2602.03685/) (2026)
- [Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra](https://mlsanity.com/paper/2410.09005/) (2024)
- [Unified Binary and Multiclass Margin-Based Classification](https://mlsanity.com/paper/2311.17778/) (2023)
- [Soft Mode in the Dynamics of Over-realizable On-line Learning for Soft Committee Machines](https://mlsanity.com/paper/2104.14546/) (2021)

## Built on by (most cited first, 3 of 3)

- [Emergent One-Third Scaling Law as Attention Tries to Concentrate](https://mlsanity.com/paper/2609.32100/) (2026)
- [Optimizer-dependent training dynamics converge to the same one-third optimal data scaling](https://mlsanity.com/paper/2609.37745/) (2026)
- [Emergent Inverse-Depth Scaling From Nonlinearity In Attention](https://mlsanity.com/paper/2610.11063/) (2026)

## Abstract (arXiv)

Hard-label classification is usually trained with smooth surrogate losses, most prominently softmax cross-entropy. We isolate an asymptotic mechanism by which this mismatch between smooth surrogate and discrete labels produces power-law learning curves in an online teacher-student model. After subtracting the mean logit, the thermodynamic-limit dynamics close in centered variables: a growing centered student-teacher alignment D and the residual student variance Delta. At late times, examples away from teacher decision boundaries are already classified confidently and contribute exponentially little. Only boundary layers of width O(D-1) remain active, while the noise of fixed-learning-rate online gradient descent maintains a nonzero Delta. As a function of the training time α the late-time solution yields a α-1/3 power law not only for the test loss but also for the generalization error εg, i.e., one minus test accuracy. This is much slower than the α-1 Bayes-optimal reference for the same model. We further show that learning-rate schedules can improve the generalization error towards a εg ~ α-1/2 power law. Simulations support the predicted order parameter dynamics and learning curves. Controlled experiments with correlated Gaussian inputs and whitened pretrained features show that data structure can dominate transients. Therefore, our result is an asymptotic, complementary mechanism rather than an alternative to spectral explanations of neural scaling laws.