mlsanityData to 7 Oct 2026

Disentangle Sample Size and Initialization Effect on Perfect Generalization for Single-Neuron Target

arXiv 2405.13787 ↗First posted 22 May 2024cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Overparameterized models like deep neural networks have the intriguing ability to recover target functions with fewer sampled data points than parameters (see arXiv:2307.08921). To gain insights into this phenomenon, we concentrate on a single-neuron target recovery scenario, offering a systematic examination of how initialization and sample size influence the performance of two-layer neural networks. Our experiments reveal that a smaller initialization scale is associated with improved generalization, and we identify a critical quantity called the "initial imbalance ratio" that governs training dynamics and generalization under small initialization, supported by theoretical proofs. Additionally, we empirically delineate two critical thresholds in sample size--termed the "optimistic sample size" and the "separation sample size"--that align with the theoretical frameworks established by (see arXiv:2307.08921 and arXiv:2309.00508). Our results indicate a transition in the model's ability to recover the target function: below the optimistic sample size, recovery is unattainable; at the optimistic sample size, recovery becomes attainable albeit with a set of initialization of zero measure. Upon reaching the separation sample size, the set of initialization that can successfully recover the target function shifts from zero to positive measure. These insights, derived from a simplified context, provide a perspective on the intricate yet decipherable complexities of perfect generalization in overparameterized neural networks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 15 of its 23 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

15 of 23 references.

References · 15

  1. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  2. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  3. 2018Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?cited by 87
  4. 2020Learning a Single Neuron with Gradient Methodscited by 50
  5. 2020Agnostic Learning of a Single Neuron with Gradient Descentcited by 46
  6. 2019First-order methods almost always avoid saddle points: the case of vanishing step-sizescited by 35
  7. 2021Towards Understanding the Condensation of Neural Networks at Initial Trainingcited by 31
  8. 2023Over-Parameterization Exponentially Slows Down Gradient Descent for Learning a Single Neuroncited by 25
  9. 2022On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Biascited by 21
  10. 2021Learning a Single Neuron with Bias Using Gradient Descentcited by 17
  11. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
  12. 2022Learning a Single Neuron for Non-monotonic Activation Functionscited by 13
3 more
  1. 2022Linear Stability Hypothesis and Rank Stratification for Nonlinear Modelscited by 7
  2. 2023Optimistic Estimate Uncovers the Potential of Nonlinear Modelscited by 5
  3. 2023Geometry and Local Recovery of Global Minima of Two-layer Neural Networks at Overparameterizationcited by 4

Built on by

Not cited yet.