mlsanityData to 14 Oct 2026

Learning and generalization of one-hidden-layer neural networks, going beyond standard Gaussian data

arXiv 2207.03615 ↗First posted 7 Jul 2022cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

This paper analyzes the convergence and generalization of training a one-hidden-layer neural network when the input features follow the Gaussian mixture model consisting of a finite number of Gaussian distributions. Assuming the labels are generated from a teacher model with an unknown ground truth weight, the learning problem is to estimate the underlying teacher model by minimizing a non-convex risk function over a student neural network. With a finite number of training samples, referred to the sample complexity, the iterations are proved to converge linearly to a critical point with guaranteed generalization error. In addition, for the first time, this paper characterizes the impact of the input distributions on the sample complexity and the learning rate.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 37 of its 54 references that are in the map. First posted on arXiv 7 Jul 2022.

Builds on

37 of 54 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  5. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 787
  6. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 669
  7. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  8. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  9. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 466
  10. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 426
  11. 2018How Does Batch Normalization Help Optimization?cited by 357
  12. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
25 more
  1. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  2. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  3. 2021Free Lunch for Few-shot Learning: Distribution Calibrationcited by 165
  4. 2018Understanding Batch Normalizationcited by 159
  5. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  6. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 141
  7. 2017Convergence Analysis of Two-layer Neural Networks with ReLU Activationcited by 137
  8. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  9. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 121
  10. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  11. 2017Recovery Guarantees for One-hidden-layer Neural Networkscited by 94
  12. 2016The Landscape of Empirical Risk for Non-convex Lossescited by 86
  13. 2014On the Computational Efficiency of Training Neural Networkscited by 71
  14. 2020Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classificationcited by 60
  15. 2019Mean Field Limit of the Learning Dynamics of Multilayer Neural Networkscited by 25
  16. 2020Fast Learning of Graph Neural Networks with Guaranteed Generalizability: One-hidden-layer Casecited by 24
  17. 2020Data-Dependence of Plateau Phenomenon in Learning with Neural Network --- Statistical Mechanical Analysiscited by 23
  18. 2018Guaranteed Recovery of One-Hidden-Layer Neural Networks via Cross Entropycited by 22
  19. 2017When is a Convolutional Filter Easy To Learn?cited by 21
  20. 2019Convex Formulation of Overparameterized Deep Neural Networkscited by 16
  21. 2015Tensor Factorization via Matrix Factorizationcited by 15
  22. 2021Why Lottery Ticket Wins? A Theoretical Perspective of Sample Complexity on Pruned Neural Networkscited by 14
  23. 2019Modelling the influence of data structure on learning in neural networks: the hidden manifold modelcited by 13
  24. 2014Score Function Features for Discriminative Learning: Matrix and Tensor Frameworkcited by 7
  25. 2020Separating the Effects of Batch Normalization on CNN Training Speed and Stability Using Classical Adaptive Filter Theorycited by 4

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 4; 2025: 2.

8 papers.

  1. 2023A Theoretical Understanding of Shallow Vision Transformers: Learning, Generalization, and Sample Complexitycited by 77
  2. 2023Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networkscited by 35
  3. 2024How Do Nonlinear Transformers Learn and Generalize in In-Context Learning?cited by 35
  4. 2024What Improves the Generalization of Graph Transformers? A Theoretical Dive into the Self-attention and Positional Encodingcited by 14
  5. 2024Learning on Transformers is Provable Low-Rank and Sparse: A One-layer Analysiscited by 8
  6. 2024How does promoting the minority fraction affect generalization? A theoretical study of the one-hidden-layer neural network on group imbalancecited by 6
  7. 2025How Can Mamba Learn In Context with Outliers and Generalize Provably?cited by 2
  8. 2025Theoretical Learning Performance of Graph Neural Networks: The Impact of Jumping Connections and Layer-wise Sparsificationcited by 0