mlsanityData to 7 Oct 2026

Finite Sample Identification of Wide Shallow Neural Networks with Biases

arXiv 2211.04589 ↗First posted 8 Nov 2022cs.LG stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

Artificial neural networks are functions depending on a finite number of parameters typically encoded as weights and biases. The identification of the parameters of the network from finite samples of input-output pairs is often referred to as the teacher-student model, and this model has represented a popular framework for understanding training and generalization. Even if the problem is NP-complete in the worst case, a rapidly growing literature -- after adding suitable distributional assumptions -- has established finite sample identification of two-layer networks with a number of neurons m=mathcal O(D), D being the input dimension. For the range D<m<D2 the problem becomes harder, and truly little is known for networks parametrized by biases as well. This paper fills the gap by providing constructive methods and theoretical guarantees of finite sample identification for such wider shallow networks with biases. Our approach is based on a two-step pipeline: first, we recover the direction of the weights, by exploiting second order information; next, we identify the signs by suitable algebraic evaluations, and we recover the biases by empirical risk minimization via gradient descent. Numerical results demonstrate the effectiveness of our approach.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 46 of its 74 references that are in the map. First posted on arXiv 8 Nov 2022.

Builds on

46 of 74 references.

References · 46

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  5. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  6. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  7. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  8. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  9. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  10. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  11. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  12. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
34 more
  1. 2017Convergence Analysis of Two-layer Neural Networks with ReLU Activationcited by 137
  2. 2017Theoretical insights into the optimization landscape of over-parameterized shallow neural networkscited by 128
  3. 2019An Improved Analysis of Training Over-parameterized Deep Neural Networkscited by 120
  4. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  5. 2017Spurious Local Minima are Common in Two-Layer ReLU Neural Networkscited by 104
  6. 2017Recovery Guarantees for One-hidden-layer Neural Networkscited by 94
  7. 2020The Interpolation Phase Transition in Neural Networks: Memorization and Generalization under Lazy Trainingcited by 84
  8. 2018Small ReLU networks are powerful memorizers: a tight analysis of memorization capacitycited by 76
  9. 2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 70
  10. 2020Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topologycited by 66
  11. 2017Globally Optimal Gradient Descent for a ConvNet with Gaussian Inputscited by 61
  12. 2019Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizerscited by 61
  13. 2019Quadratic Suffices for Over-parametrization via Matrix Chernoff Boundcited by 58
  14. 2017Learning ReLUs via Gradient Descentcited by 54
  15. 2017An Analytical Formula of Population Gradient for two-layered ReLU network and its Applications in Convergence and Critical Point Analysiscited by 52
  16. 2018Learning One-hidden-layer ReLU Networks via Gradient Descentcited by 50
  17. 2021A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Networkcited by 47
  18. 2021On the Proof of Global Convergence of Gradient Descent for Deep ReLU Networks with Linear Widthscited by 44
  19. 2017Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minimacited by 40
  20. 2022Memorization and Optimization in Deep Neural Networks with Minimum Over-parameterizationcited by 36
  21. 2018Small nonlinearities in activation functions create bad local minima in neural networkscited by 28
  22. 2019Global Convergence of Adaptive Gradient Methods for An Over-parameterized Neural Networkcited by 27
  23. 2021Subquadratic Overparameterization for Shallow Neural Networkscited by 26
  24. 2018Guaranteed Recovery of One-Hidden-Layer Neural Networks via Cross Entropycited by 22
  25. 2014Guaranteed Non-Orthogonal Tensor Decomposition via Alternating Rank-1 Updatescited by 21
  26. 2017When is a Convolutional Filter Easy To Learn?cited by 21
  27. 2020Affine symmetries and neural network identifiabilitycited by 17
  28. 2020Network size and weights size for memorization with two-layers neural networkscited by 16
  29. 2018On the Connection Between Learning Two-Layers Neural Networks and Tensor Decompositioncited by 14
  30. 2019Subspace power method for symmetric tensor decompositioncited by 14
  31. 2019Robust and Resource Efficient Identification of Two Hidden Layer Neural Networkscited by 10
  32. 2021Stable Recovery of Entangled Weights: Towards Robust Identification of Deep Neural Networks from Minimal Samplescited by 8
  33. 2021Landscape analysis of an improved power method for tensor decompositioncited by 7
  34. 2014Provable Methods for Training Neural Networks with Sparse Connectivitycited by 4

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 1; 2025: 2.

5 papers.

  1. 2023Sampling weights of deep neural networkscited by 20
  2. 2023On the accuracy of interpolation based on single-layer artificial neural networks with a focus on defeating the Runge phenomenoncited by 0
  3. 2024Sequencing the Neurome: Towards Scalable Exact Parameter Reconstruction of Black-Box Neural Networkscited by 0
  4. 2025Finite Samples for Shallow Neural Networkscited by 0
  5. 2025Data Augmentation Techniques to Reverse-Engineer Neural Network Weights from Input-Output Queriescited by 0