When Does Learning Renormalize? Sufficient Conditions for Power Law Spectral Dynamics
Not on the map. Why?
Abstract (arXiv)
Empirical power--law scaling has been widely observed across modern deep learning systems, yet its theoretical origins and scope of validity remain incompletely understood. The Generalized Resolution--Shell Dynamics (GRSD) framework models learning as spectral energy transport across logarithmic resolution shells, providing a coarse--grained dynamical description of training. Within GRSD, power--law scaling corresponds to a particularly simple renormalized shell dynamics; however, such behavior is not automatic and requires additional structural properties of the learning process. In this work, we identify a set of sufficient conditions under which the GRSD shell dynamics admits a renormalizable coarse--grained description. These conditions constrain the learning configuration at multiple levels, including boundedness of gradient propagation in the computation graph, weak functional incoherence at initialization, controlled Jacobian evolution along training, and log--shift invariance of renormalized shell couplings. We further show that power--law scaling does not follow from renormalizability alone, but instead arises as a rigidity consequence: once log--shift invariance is combined with the intrinsic time--rescaling covariance of gradient flow, the renormalized GRSD velocity field is forced into a power--law form.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 29 of its 54 references that are in the map. First posted on arXiv 20 Dec 2025.
Builds on
29 of 54 references.
References · 29
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2016Layer Normalizationcited by 5,669
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
- 2022Training Compute-Optimal Large Language Modelscited by 3,019
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
- 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
- 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
17 more
- 2022Simplified State Space Layers for Sequence Modelingcited by 788
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
- 2017Self-Normalizing Neural Networkscited by 598
- 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
- 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
- 2020Scaling Laws for Autoregressive Generative Modelingcited by 522
- 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 292
- 2024A Dynamical Model of Neural Scaling Lawscited by 111
- 2024Scaling Laws for Precisioncited by 95
- 2024How Feature Learning Can Improve Neural Scaling Lawscited by 56
- 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 49
- 2018Dynamical Isometry is Achieved in Residual Networks in a Universal Way for any Activation Functioncited by 22
- 2025Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedulescited by 17
- 2025Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokkingcited by 15
- 2025AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understandingcited by 9
- 2025A Generalized Spectral Framework to Expain Neural Scaling and Compression Dynamicscited by 3
- 2022A Framework for Overparameterized Learningcited by 2
Built on by
Not cited yet.