# The Multiscale Single-Index Model: A Stylized Model for Hierarchical Feature Learning

arXiv: https://arxiv.org/abs/2607.03347 · HTML: https://mlsanity.com/paper/2607.03347/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 57 references that are in the map.
- First posted on arXiv 3 Jul 2026.

## Builds on (most cited first, 15 of 15)

- [Stochastic Interpolants: A Unifying Framework for Flows and Diffusions](https://mlsanity.com/paper/2303.08797/) (2023)
- [Learning One-hidden-layer Neural Networks with Landscape Design](https://mlsanity.com/paper/1711.00501/) (2017)
- [How Two-Layer Neural Networks Learn, One (Giant) Step at a Time](https://mlsanity.com/paper/2305.18270/) (2023)
- [On Learning Gaussian Multi-index Models with Gradient Flow](https://mlsanity.com/paper/2310.19793/) (2023)
- [Computational-Statistical Gaps in Gaussian Single-Index Models](https://mlsanity.com/paper/2403.05529/) (2024)
- [Asymptotics of feature learning in two-layer networks after one gradient-step](https://mlsanity.com/paper/2402.04980/) (2024)
- [The Hidden Width of Deep ResNets: Tight Error Bounds and Phase Diagram](https://mlsanity.com/paper/2509.10167/) (2025)
- [On the Complexity of Learning Sparse Functions with Statistical and Gradient Queries](https://mlsanity.com/paper/2407.05622/) (2024)
- [ResNets of All Shapes and Sizes: Convergence of Training Dynamics in the Large-scale Limit](https://mlsanity.com/paper/2603.18168/) (2026)
- [The Generative Leap: Sharp Sample Complexity for Efficiently Learning Gaussian Multi-Index Models](https://mlsanity.com/paper/2506.05500/) (2025)
- [Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods](https://mlsanity.com/paper/1506.08473/) (2015)
- [Deep Learning of Compositional Targets with Hierarchical Spectral Methods](https://mlsanity.com/paper/2602.10867/) (2026)
- [Langevin dynamics for high-dimensional optimization: the case of multi-spiked tensor PCA](https://mlsanity.com/paper/2408.06401/) (2024)
- [Universality of high-dimensional scaling limits of stochastic gradient descent](https://mlsanity.com/paper/2512.13634/) (2025)
- [Statistical-Computational Trade-offs in Learning Multi-Index Models via Harmonic Analysis](https://mlsanity.com/paper/2602.09959/) (2026)

## Built on by (most cited first, 2 of 2)

- [Algorithmic Separation between Constant-Depth and Logarithmic-Depth Neural Networks](https://mlsanity.com/paper/2607.25200/) (2026)
- [Barron Optimal Transport I: Generative Modeling](https://mlsanity.com/paper/2610.10875/) (2026)

## Abstract (arXiv)

We consider the Multiscale Single-Index Model (MSIM), first introduced in citeoymak2021learning, as a stylized model for hierarchical learning with scale separation. Each layer extracts a shared single-index feature at one physical scale and passes it to the next, thus defining a tractable setting in which to study how deep architectures learn multiscale representations. Under non-degeneracy and delocalization assumptions on the link function and planted features respectively, for fixed depth K and local scale d, the first Wiener chaos of the target behaves as a perturbed spiked tensor, where the perturbation of order d-1/2 comes from the non-linearity -- revealing the MSIM as a natural non-linear analogue of the Tensor PCA model citemontanari2014statistical. While this perturbative picture is sufficient to enable efficient spectral recovery based on Tensor unfolding (as already observed in citeoymak2021learning), it is not precise enough for the analysis of backpropagation gradient-based methods. In this work, we address this limitation by performing a fine-grained analysis of the Wiener chaos using Edgeworth expansions. In the first chaos, this gives a finite-rank hierarchy at scales d-q/2. In higher chaoses, balanced flattenings exhibit staircase singular-value plateaus of size d-ρ/2 and multiplicity dρ under a natural higher-chaos non-cancellation condition. Using this higher-chaos structure, and under an additional slow Hermite-energy tail condition, we first establish shallow-network approximation lower bounds, quantifying the benefit of depth in this model. Next, and most importantly, we prove that online SGD on the correlation objective, where all layers evolve in the same timescale, achieves 1 - od(1) recovery with n = widetildeO( dK-1) samples, recovering the same sample complexity as in the linear counterpart.