mlsanityData to 14 Oct 2026

Gradient-Based Feature Learning under Structured Data

arXiv 2309.03843 ↗First posted 7 Sep 2023stat.ML cs.LGcited by 32

Not on the map. Why?

Abstract (arXiv)

Recent works have demonstrated that the sample complexity of gradient-based learning of single index models, i.e. functions that depend on a 1-dimensional projection of the input data, is governed by their information exponent. However, these results are only concerned with isotropic data, while in practice the input often contains additional structure which can implicitly guide the algorithm. In this work, we investigate the effect of a spiked covariance structure and reveal several interesting phenomena. First, we show that in the anisotropic setting, the commonly used spherical gradient dynamics may fail to recover the true direction, even when the spike is perfectly aligned with the target direction. Next, we show that appropriate weight normalization that is reminiscent of batch normalization can alleviate this issue. Further, by exploiting the alignment between the (spiked) input covariance and the target, we obtain improved sample complexity compared to the isotropic case. In particular, under the spiked model with a suitably large spike, the sample complexity of gradient-based training can be made independent of the information exponent while also outperforming lower bounds for rotationally invariant kernel methods.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 38 of its 66 references that are in the map. First posted on arXiv 7 Sep 2023.

Builds on

38 of 66 references.

References · 38

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2018On Lazy Training in Differentiable Programmingcited by 675
  4. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  5. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  6. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  7. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 187
  8. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  9. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  10. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  11. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 143
  12. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 142
26 more
  1. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  2. 2020Online stochastic gradient descent on non-convex losses from high-dimensional inferencecited by 119
  3. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  4. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  5. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  6. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 69
  7. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 63
  8. 2022Mean-Field Langevin Dynamics: Exponential Convergence and Annealingcited by 61
  9. 2023Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Modelscited by 61
  10. 2023Learning time-scales in two-layers neural networkscited by 57
  11. 2022Convex Analysis of the Mean Field Langevin Dynamicscited by 55
  12. 2017Learning ReLUs via Gradient Descentcited by 54
  13. 2020Learning a Single Neuron with Gradient Methodscited by 50
  14. 2020Agnostic Learning of a Single Neuron with Gradient Descentcited by 46
  15. 2023From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networkscited by 43
  16. 2020Learning Polynomials of Few Relevant Dimensionscited by 34
  17. 2019Time/Accuracy Tradeoffs for Learning a ReLU with respect to Gaussian Marginalscited by 32
  18. 2021How rotational invariance of common kernels prevents generalization in high dimensionscited by 32
  19. 2023Hitting the High-Dimensional Notes: An ODE for SGD learning dynamics on GLMs and multi-index modelscited by 26
  20. 2020Statistical Queries and Statistical Algorithms: Foundations and Applicationscited by 22
  21. 2022Learning a Single Neuron with Adversarial Label Noise via Gradient Descentcited by 20
  22. 2023Beyond NTK with Vanilla Gradient Descent: A Mean-Field Analysis of Neural Networks with Polynomial Width, Samples, and Timecited by 20
  23. 2022Agnostic Learning of General ReLU Activation Using Gradient Descentcited by 12
  24. 2022Generalization Bounds for Stochastic Gradient Descent via Localized varepsilon-Coverscited by 11
  25. 2022On the symmetries in the dynamics of wide two-layer neural networkscited by 8
  26. 2016Scalable Approximations for Generalized Linear Problemscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 12; 2025: 13; 2026: 7.

32 papers.

Sort:

By area:

  1. 2024Neural network learns low-dimensional polynomials with SGD near the information-theoretic limitcited by 46
  2. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 38
  3. 2024Repetita Iuvant: Data Repetition Allows SGD to Learn High-Dimensional Multi-Index Functionscited by 37
  4. 2024Pretrained transformer efficiently learns low-dimensional target functions in-contextcited by 31
  5. 2024Nonlinear spiked covariance matrices and signal propagation in deep neural networkscited by 26
  6. 2024Learning Multi-Index Models with Neural Networks via Mean-Field Langevin Dynamicscited by 15
  7. 2025Survey on Algorithms for multi-index modelscited by 15
  8. 2024Sliding down the stairs: how correlated latent variables accelerate learning with neural networkscited by 12
  9. 2025Kernel ridge regression under power-law data: spectrum and generalizationcited by 10
  10. 2024From Spikes to Heavy Tails: Unveiling the Spectral Evolution of Neural Networkscited by 9
  11. 2025Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networkscited by 9
  12. 2025Low-dimensional Functions are Efficiently Learnable under Randomly Biased Distributionscited by 8
20 more
  1. 2025On the Mechanisms of Weak-to-Strong Generalization: A Theoretical Perspectivecited by 8
  2. 2025Provable Benefits of Unsupervised Pre-training and Transfer Learning via Single-Index Modelscited by 6
  3. 2024Robust Feature Learning for Multi-Index Models in High Dimensionscited by 5
  4. 2025Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structurecited by 4
  5. 2025Learning a Single Index Model from Anisotropic Data with vanilla Stochastic Gradient Descentcited by 4
  6. 2025Low Rank Gradients and Where to Find Themcited by 4
  7. 2024Online Learning and Information Exponents: On The Importance of Batch size, and Time/Complexity Tradeoffscited by 3
  8. 2025Universality of high-dimensional scaling limits of stochastic gradient descentcited by 3
  9. 2026How Do Transformers Learn to Associate Tokens: Gradient Leading Terms Bring Mechanistic Interpretabilitycited by 3
  10. 2026Positive Distribution Shift as a Framework for Understanding Tractable Learningcited by 3
  11. 2024Input-Label Correlation Governs a Linear-to-Nonlinear Transition in Random Features under Spiked Covariancecited by 2
  12. 2025Guided by the Experts: Provable Feature Learning Dynamic of Soft-Routed Mixture-of-Expertscited by 2
  13. 2025From Information to Generative Exponent: Learning Rate Induces Phase Transitions in SGDcited by 2
  14. 2024Learning Hierarchical Polynomials of Multiple Nonlinear Features with Three-Layer Networkscited by 1
  15. 2025Mamba Can Learn Low-Dimensional Targets In-Context via Test-Time Feature Learningcited by 1
  16. 2026The Mechanism of Weak-to-Strong Generalization: Feature Elicitation from Latent Knowledgecited by 1
  17. 2026Learning with Shallow Neural Networks on Cluster-Structured Featurescited by 1
  18. 2026Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probingcited by 1
  19. 2026The Power of Power Law: Asymmetry Enables Compositional Reasoningcited by 0
  20. 2026A Fourier perspective on the learning dynamics of neural networks: from sample complexities to mechanistic insightscited by 0