A Full Adam Theorem for Spectral Heavy-Tail Onset
On the map: Knowledge distillation → General knowledge distillation methods
Abstract (arXiv)
We prove a full Adam theorem for spectral heavy-tail onset in a closed Gaussian Stein-Hermite teacher-student state-evolution model. The theorem begins with the actual full-batch Adam recurrences, derives the population gradient by Stein-Hermite calculus, proves finite-width covariance concentration, converts multi-step Adam momentum into an exact non-centered Gaussian sign kernel, controls the diagonal Adam denominator by a basis-homogenization theorem, derives a regularly varying projected update response from a Hermite edge-transfer theorem, pushes the response through the exact Gram update, and proves approximate-target KL contraction with matching upper and lower hitting bounds. The final law is (τ_varepsilon=Theta(Delta1-γdρlog(Psi0/varepsilon))), where (Delta1) is the first spike-bulk spectral gap. The result is full in the following precise sense: every step from Adam's momentum and denominator to the spectral hitting law is formalized inside the closed state-evolution model. We also prove that a stronger arbitrary-gradient Adam theorem is impossible, and that exact two-step linear-network loss dynamics do not identify factor spectra or heavy-tail hitting times.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General knowledge distillation methods, one of 12 directions in Knowledge distillation. Builds on 10 of its 21 references that are in the map. First posted on arXiv 10 Jul 2026.
Builds on
10 of 21 references.
References · 10
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2019On the Convergence of Adam and Beyondcited by 841
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
- 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
- 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
- 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 215
- 2020Tensor Programs II: Neural Tangent Kernel for Any Architecturecited by 143
- 2023Convergence of Adam Under Relaxed Assumptionscited by 99
- 2024From Spikes to Heavy Tails: Unveiling the Spectral Evolution of Neural Networkscited by 9
- 2026Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networkscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026The Hidden Ratio in Adam: Stable Structure, Compression, and Sign Dynamics
- 2026Beyond Dense Adam States: Adaptive Log-Space Quantization for Memory-Efficient Optimizers
- 2022Efficient-Adam: Communication-Efficient Distributed Adam
- 2022Maximizing Communication Efficiency for Large-scale Training via 0/1 Adam
- 2026Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization
- 20211-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed
- 2026Spectral phase transitions and trainability in neural network learning dynamics
- 2026Spectral Evolution Search: Efficient Inference-Time Scaling for Reward-Aligned Image Generation