mlsanityData to 7 Oct 2026

On Learning Gaussian Multi-index Models with Gradient Flow

arXiv 2310.19793 ↗First posted 30 Oct 2023stat.ML cs.LG math.OCcited by 53

Not on the map. Why?

Abstract (arXiv)

We study gradient flow on the multi-index regression problem for high-dimensional Gaussian data. Multi-index functions consist of a composition of an unknown low-rank linear projection and an arbitrary unknown, low-dimensional link function. As such, they constitute a natural template for feature learning in neural networks. We consider a two-timescale algorithm, whereby the low-dimensional link function is learnt with a non-parametric model infinitely faster than the subspace parametrizing the low-rank projection. By appropriately exploiting the matrix semigroup structure arising over the subspace correlation matrices, we establish global convergence of the resulting Grassmannian population gradient flow dynamics, and provide a quantitative description of its associated `saddle-to-saddle' dynamics. Notably, the timescales associated with each saddle can be explicitly characterized in terms of an appropriate Hermite decomposition of the target link function. In contrast with these positive results, we also show that the related planted problem, where the link function is known and fixed, in fact has a rough optimization landscape, in which gradient flow dynamics might get trapped with high probability.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 22 of its 79 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

22 of 79 references.

References · 22

  1. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  2. 2019Towards Understanding the Spectral Bias of Deep Learningcited by 173
  3. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  4. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  5. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
  6. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 141
  7. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  8. 2016The Landscape of Empirical Risk for Non-convex Lossescited by 86
  9. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  10. 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 69
  11. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 67
  12. 2023Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Modelscited by 60
10 more
  1. 2023Learning time-scales in two-layers neural networkscited by 57
  2. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 54
  3. 2023From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networkscited by 42
  4. 2023SGD Finds then Tunes Features in Two-Layer Neural Networks with near-Optimal Sample Complexity: A Case Study in the XOR problemcited by 36
  5. 2023Beyond NTK with Vanilla Gradient Descent: A Mean-Field Analysis of Neural Networks with Polynomial Width, Samples, and Timecited by 20
  6. 2023Leveraging the two timescale regime to demonstrate convergence of neural networkscited by 18
  7. 2023On Single Index Models beyond Gaussian Datacited by 17
  8. 2023Learning Narrow One-Hidden-Layer ReLU Networkscited by 14
  9. 2023Nonparametric Linear Feature Learning in Regression Through Regularisationcited by 5
  10. 2023Symmetric Single Index Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 25; 2025: 21; 2026: 6.

53 papers.

Sort:

By area:

  1. 2024The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponentscited by 50
  2. 2024Neural network learns low-dimensional polynomials with SGD near the information-theoretic limitcited by 45
  3. 2024Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscapecited by 43
  4. 2024Learning Mixtures of Gaussians Using Diffusion Modelscited by 41
  5. 2024Computational-Statistical Gaps in Gaussian Single-Index Modelscited by 39
  6. 2024Repetita Iuvant: Data Repetition Allows SGD to Learn High-Dimensional Multi-Index Functionscited by 35
  7. 2024Pretrained transformer efficiently learns low-dimensional target functions in-contextcited by 29
  8. 2024Bayes-optimal learning of an extensive-width neural network from quadratically many samplescited by 23
  9. 2024Fundamental computational limits of weak learnability in high-dimensional multi-index modelscited by 17
  10. 2025Survey on Algorithms for multi-index modelscited by 15
  11. 2024Towards a theory of model distillationcited by 14
  12. 2024Learning Multi-Index Models with Neural Networks via Mean-Field Langevin Dynamicscited by 14
38 more
  1. 2024A Random Matrix Theory Perspective on the Spectrum of Learned Features and Asymptotic Generalization Capabilitiescited by 14
  2. 2024Learning sum of diverse features: computational hardness and efficient gradient-based training for ridge combinationscited by 13
  3. 2024On the Complexity of Learning Sparse Functions with Statistical and Gradient Queriescited by 13
  4. 2024Mean-field Analysis on Two-layer Neural Networks from a Kernel Perspectivecited by 12
  5. 2024Sliding down the stairs: how correlated latent variables accelerate learning with neural networkscited by 12
  6. 2025The Nuclear Route: Sharp Asymptotics of ERM in Overparameterized Quadratic Networkscited by 11
  7. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 11
  8. 2025Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networkscited by 9
  9. 2023Should Under-parameterized Student Networks Copy or Average Teacher Weights?cited by 8
  10. 2024Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networkscited by 8
  11. 2024Learning Gaussian Multi-Index Models with Gradient Flow: Time Complexity and Directional Convergencecited by 8
  12. 2025Low-dimensional Functions are Efficiently Learnable under Randomly Biased Distributionscited by 8
  13. 2024How Transformers Get Rich: Approximation and Dynamics Analysiscited by 7
  14. 2025Spectral Estimators for Multi-Index Models: Precise Asymptotics and Optimal Weak Recoverycited by 6
  15. 2025Optimal Spectral Transitions in High-Dimensional Multi-Index Modelscited by 6
  16. 2025Provable Benefits of Unsupervised Pre-training and Transfer Learning via Single-Index Modelscited by 6
  17. 2025The Generative Leap: Sharp Sample Complexity for Efficiently Learning Gaussian Multi-Index Modelscited by 6
  18. 2025Understanding the Mechanisms of Fast Hyperparameter Transfercited by 6
  19. 2025Online Learning of Neural Networkscited by 5
  20. 2025Ultra-fast feature learning for the training of two-layer neural networks in the two-timescale regimecited by 4
  21. 2025Mixture of Experts Provably Detect and Learn the Latent Cluster Structure in Gradient-Based Learningcited by 4
  22. 2024Three Mechanisms of Feature Learning in a Linear Networkcited by 3
  23. 2024Stochastic Differential Equations models for Least-Squares Stochastic Gradient Descentcited by 3
  24. 2025Bayes optimal learning of attention-indexed modelscited by 3
  25. 2024Online Learning and Information Exponents: On The Importance of Batch size, and Time/Complexity Tradeoffscited by 2
  26. 2024Learning with Hidden Factorial Structurecited by 2
  27. 2025From Information to Generative Exponent: Learning Rate Induces Phase Transitions in SGDcited by 2
  28. 2026Learning Sparse Compositional Functions with Norm-Constrained Neural Networkscited by 2
  29. 2024Mean-Field Analysis for Learning Subspace-Sparse Polynomials with Gaussian Inputcited by 1
  30. 2024Learning Hierarchical Polynomials of Multiple Nonlinear Features with Three-Layer Networkscited by 1
  31. 2025Joint Learning in the Gaussian Single Index Modelcited by 1
  32. 2025Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Expertscited by 1
  33. 2025Provably Extracting the Features from a General Superpositioncited by 1
  34. 2026A Noise Sensitivity Exponent Controls Large Statistical-to-Computational Gaps in Single- and Multi-Index Modelscited by 1
  35. 2026Uniform-in-Time Weak Propagation-of-Chaos in Shallow Neural Networkscited by 1
  36. 2026Incremental Learning in Mirror Flowscited by 1
  37. 2025Learning with Restricted Boltzmann Machines: Asymptotics of AMP and GD in High Dimensionscited by 0
  38. 2025A Derandomization Framework for Structure Discovery: Applications in Neural Networks and Beyondcited by 0

50 most cited of 53 All 53 (JSON)