mlsanityData to 14 Oct 2026

Dimension-Corrected Hitting Times for Heavy-Tailed Spectral Emergence in Neural Optimizer Dynamics

arXiv 2609.12994 ↗First posted 10 Jul 2026cs.LG cs.NE stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

Heavy-tailed empirical spectral densities of neural-network weight matrices are widely used as diagnostics of implicit self-regularization, but the step complexity of heavy-tail emergence remains poorly understood. We formulate spectral heavy-tail formation as a right-censored hitting-time problem: a run that does not reach a heavy-tail diagnostic within the observation horizon is treated as censored rather than discarded. In controlled full-batch teacher--student dynamics, we find that the first-step spike--bulk gap alone does not explain onset time. Instead, finite-onset regression supports a dimension-corrected spectral-gap law, (τHT≈ CDelta1-γdρ), with (R2=0.683), (γ=0.626), and (ρ=0.772) across 330 completed runs. Right-censored lognormal accelerated-failure-time models further favor the dimension-corrected model over a gap-only model, improving AIC from 706.62 to 628.70. Theoretically, we prove that exact early loss dynamics in linear networks do not determine factor spectral tails, that Adam recurrences alone do not imply spectral redistribution, and that projected singular-basis spreading implies contraction of a spectral-tail potential and hence a dimension-corrected hitting-time bound. Empirically, projected-kernel profiles support the sufficient spreading mechanism, Adam and AdamW agree under tested grids, GD and signGD do not reach onset in the same regimes, and real pretrained Qwen2.5-0.5B and Pythia-70M transformer weights show non-Gaussian spectral-tail structure relative to matched Gaussian nulls. The result is a reproducible spectral hitting-time law with rigorous conditional theory, not a claim that Adam necessarily generates heavy tails from first principles.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 28 references that are in the map. First posted on arXiv 10 Jul 2026.

Builds on

14 of 28 references.

References · 14

  1. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 215
  2. 2019A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networkscited by 214
  3. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  4. 2020The Heavy-Tail Phenomenon in SGDcited by 124
  5. 2020Predicting trends in the quality of state-of-the-art neural networks without access to training or testing datacited by 115
  6. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  7. 2020Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networkscited by 62
  8. 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 49
  9. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 38
  10. 2021Generalization Bounds using Lower Tail Exponents in Stochastic Optimizerscited by 27
  11. 2023Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Trainingcited by 20
  12. 2020Multiplicative noise and heavy tails in stochastic optimizationcited by 17
2 more
  1. 2023Algorithmic Stability of Heavy-Tailed SGD with General Loss Functionscited by 15
  2. 2024From Spikes to Heavy Tails: Unveiling the Spectral Evolution of Neural Networkscited by 9

Built on by

Not cited yet.