# Scaling Laws from Sequential Feature Recovery: A Solvable Hierarchical Model

arXiv: https://arxiv.org/abs/2605.14567 · HTML: https://mlsanity.com/paper/2605.14567/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 68 references that are in the map.
- First posted on arXiv 14 May 2026.

## Builds on (most cited first, 16 of 16)

- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [A Solvable Model of Neural Scaling Laws](https://mlsanity.com/paper/2210.16859/) (2022)
- [Scaling and renormalization in high-dimensional regression](https://mlsanity.com/paper/2405.00592/) (2024)
- [Emergence and scaling laws in SGD learning of shallow neural networks](https://mlsanity.com/paper/2504.19983/) (2025)
- [Asymptotics of Random Feature Regression Beyond the Linear Scaling Regime](https://mlsanity.com/paper/2403.08160/) (2024)
- [Deep Learning and Hierarchal Generative Models](https://mlsanity.com/paper/1612.09057/) (2016)
- [Learning quadratic neural networks in high dimensions: SGD dynamics and scaling laws](https://mlsanity.com/paper/2508.03688/) (2025)
- [Kernel ridge regression under power-law data: spectrum and generalization](https://mlsanity.com/paper/2510.04780/) (2025)
- [When does Gaussian equivalence fail and how to fix it: Non-universal behavior of random features with quadratic scaling](https://mlsanity.com/paper/2512.03325/) (2025)
- [ResNets of All Shapes and Sizes: Convergence of Training Dynamics in the Large-scale Limit](https://mlsanity.com/paper/2603.18168/) (2026)
- [Learning Hierarchical Polynomials with Three-Layer Neural Networks](https://mlsanity.com/paper/2311.13774/) (2023)
- [Optimal scaling laws in learning hierarchical multi-index models](https://mlsanity.com/paper/2602.05846/) (2026)
- [Spectral Estimators for Multi-Index Models: Precise Asymptotics and Optimal Weak Recovery](https://mlsanity.com/paper/2502.01583/) (2025)
- [Deep Learning of Compositional Targets with Hierarchical Spectral Methods](https://mlsanity.com/paper/2602.10867/) (2026)
- [The Computational Advantage of Depth: Learning High-Dimensional Hierarchical Functions with Gradient Descent](https://mlsanity.com/paper/2502.13961/) (2025)

## Abstract (arXiv)

We propose a simple mechanism by which scaling laws emerge from feature learning in multi-layer networks. We study a high-dimensional hierarchical target that is a globally high-degree function, but that can be represented by a combination of latent compositional features whose weights decrease as a power law. We show that a layer-wise spectral algorithm adapted to this compositional structure achieves improved scaling relative to shallow, non-adaptive methods, and recovers the latent directions sequentially: strong features become detectable at small sample sizes, while weaker features require more data. We prove sharp feature-wise recovery thresholds and show that aggregating these transitions yields an explicit power-law decay of the prediction error. Technically, the analysis relies on random matrix methods and a resolvent-based perturbation argument, which gives matching upper and lower bounds for individual eigenvector recovery beyond what standard gap-based perturbation bounds provide. Numerical experiments confirm the predicted sequential recovery, finite-size smoothing of the thresholds, and separation from non-hierarchical kernel baselines. Together, these results show how smooth scaling laws can emerge from a cascade of sharp feature-learning transitions.