mlsanityData to 14 Oct 2026

How many samples are needed to train a deep neural network?

arXiv 2405.16696 ↗First posted 26 May 2024math.ST stat.ML stat.THcited by 5

Not on the map. Why?

Abstract (arXiv)

Neural networks have become standard tools in many areas, yet many important statistical questions remain open. This paper studies the question of how much data are needed to train a ReLU feed-forward neural network. Our theoretical and empirical results suggest that the generalization error of ReLU feed-forward neural networks scales at the rate 1/√n in the sample size n rather than the usual "parametric rate" 1/n. Thus, broadly speaking, our results underpin the common belief that neural networks need "many" training samples.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 31 of its 48 references that are in the map. First posted on arXiv 26 May 2024.

Builds on

31 of 48 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  5. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,487
  6. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  7. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 669
  8. 2017Exploring Generalization in Deep Learningcited by 600
  9. 2016Error bounds for approximations with deep ReLU networkscited by 574
  10. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 433
  11. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
  12. 2018Stronger generalization bounds for deep nets via a compression approachcited by 307
19 more
  1. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  2. 2017Size-Independent Sample Complexity of Neural Networkscited by 290
  3. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 162
  4. 2019LassoNet: A Neural Network with Feature Sparsitycited by 82
  5. 2020The Real-World-Weight Cross-Entropy Loss Function: Modeling the Costs of Mislabelingcited by 64
  6. 2021What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theorycited by 60
  7. 2018Deep Neural Networks Learn Non-Smooth Functions Effectivelycited by 57
  8. 2019Generalization in Deep Networks: The Role of Distance from Initializationcited by 51
  9. 2018Performance Metrics (Error Measures) in Machine Learning Regression, Forecasting and Prognostics: Properties and Typologycited by 35
  10. 2019An Introductory Guide to Fano's Inequality with Applications in Statistical Estimationcited by 31
  11. 2020Statistical Guarantees for Regularized Neural Networkscited by 25
  12. 2019On the minimax optimality and superiority of deep neural network learning over sparse parameter spacescited by 24
  13. 2021Convergence rates of deep ReLU networks for multiclass classificationcited by 21
  14. 2018How Many Samples are Needed to Estimate a Convolutional or Recurrent Neural Network?cited by 19
  15. 2020Estimation error analysis of deep learning on the regression problem on the variable exponent Besov spacecited by 13
  16. 2022Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?cited by 10
  17. 2022Statistical guarantees for sparse deep learningcited by 9
  18. 2020Layer Sparsity in Neural Networkscited by 6
  19. 2017Minimax Lower Bounds for Ridge Combinations Including Neural Netscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2025Regularization can make diffusion models more efficientcited by 2
  2. 2025Learning solutions of parameterized stiff ODEs using Gaussian processescited by 1
  3. 2026Generative Model via Quantile Assignmentcited by 0
  4. 2026How Many Training Samples Are Needed for the Inverse Kinematics Solutions by Artificial Neural Networkscited by 0
  5. 2026Beyond Humans: Multispecies Animal Face Recognition Using Transfer Learningcited by 0