mlsanityData to 14 Oct 2026

Asymptotic Analysis of Two-Layer Neural Networks after One Gradient Step under Gaussian Mixtures Data with Structure

arXiv 2503.00856 ↗First posted 2 Mar 2025stat.ML cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

In this work, we study the training and generalization performance of two-layer neural networks (NNs) after one gradient descent step under structured data modeled by Gaussian mixtures. While previous research has extensively analyzed this model under isotropic data assumption, such simplifications overlook the complexities inherent in real-world datasets. Our work addresses this limitation by analyzing two-layer NNs under Gaussian mixture data assumption in the asymptotically proportional limit, where the input dimension, number of hidden neurons, and sample size grow with finite ratios. We characterize the training and generalization errors by leveraging recent advancements in Gaussian universality. Specifically, we prove that a high-order polynomial model performs equivalent to the nonlinear neural networks under certain conditions. The degree of the equivalent model is intricately linked to both the "data spread" and the learning rate employed during one gradient step. Through extensive simulations, we demonstrate the equivalence between the original model and its polynomial counterpart across various regression and classification tasks. Additionally, we explore how different properties of Gaussian mixtures affect learning outcomes. Finally, we illustrate experimental results on Fashion-MNIST classification, indicating that our findings can translate to realistic data.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 31 of its 49 references that are in the map. First posted on arXiv 2 Mar 2025.

Builds on

31 of 49 references.

References · 31

  1. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  3. 2014Conditional Generative Adversarial Netscited by 2,162
  4. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  5. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  6. 2018Estimating the intrinsic dimension of datasets by a minimal neighborhood informationcited by 250
  7. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  8. 2019Linearized two-layers neural networks in high dimensioncited by 155
  9. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  10. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  11. 2020The Gaussian equivalence of generative models for learning with shallow neural networkscited by 113
  12. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
19 more
  1. 2022Universality of empirical risk minimizationcited by 75
  2. 2021Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeedcited by 69
  3. 2023How Two-Layer Neural Networks Learn, One (Giant) Step at a Timecited by 68
  4. 2020The role of regularization in classification of high-dimensional noisy Gaussian mixturecited by 63
  5. 2021Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensionscited by 54
  6. 2023A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networkscited by 49
  7. 2020Random Matrix Theory Proves that Deep Learning Representations of GAN-data Behave as Gaussian Mixturescited by 47
  8. 2023Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networkscited by 43
  9. 2022The curse of overparametrization in adversarial training: Precise analysis of robust generalization for random features regressioncited by 41
  10. 2024Asymptotics of feature learning in two-layer networks after one gradient-stepcited by 38
  11. 2023Deterministic equivalent and error universality of deep random features learningcited by 35
  12. 2023Gradient-Based Feature Learning under Structured Datacited by 32
  13. 2024Asymptotics of Random Feature Regression Beyond the Linear Scaling Regimecited by 26
  14. 2021Covariate Shift in High-Dimensional Random Feature Regressioncited by 24
  15. 2023Precise Asymptotic Analysis of Deep Random Feature Modelscited by 22
  16. 2022On double-descent in uncertainty quantification in overparametrized modelscited by 16
  17. 2023Learning curves for deep structured Gaussian feature modelscited by 13
  18. 2023Demystifying Disagreement-on-the-Line in High Dimensionscited by 12
  19. 2024Input-Label Correlation Governs a Linear-to-Nonlinear Transition in Random Features under Spiked Covariancecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Asymptotic Study of In-context Learning with Random Transformers through Equivalent Modelscited by 3
  2. 2025How Data Mixing Shapes In-Context Learning: Asymptotic Equivalence for Transformers with MLPscited by 1
  3. 2026Learning Beyond the Gaussian Data: Learning Dynamics of Neural Networks on an Expressive and Cumulant-Controllable Data Modelcited by 0
  4. 2026How Does Attention Help? Insights from Random Matrices on Signal Recovery from Sequence Modelscited by 0