# How many samples are needed to train a deep neural network?

arXiv: https://arxiv.org/abs/2405.16696 · HTML: https://mlsanity.com/paper/2405.16696/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 48 references that are in the map.
- First posted on arXiv 26 May 2024.

## Builds on (most cited first, 30 of 31)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation](https://mlsanity.com/paper/1511.00561/) (2015)
- [Spectrally-normalized margin bounds for neural networks](https://mlsanity.com/paper/1706.08498/) (2017)
- [Speech Recognition with Deep Recurrent Neural Networks](https://mlsanity.com/paper/1303.5778/) (2013)
- [Exploring Generalization in Deep Learning](https://mlsanity.com/paper/1706.08947/) (2017)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks](https://mlsanity.com/paper/1707.09564/) (2017)
- [Stronger generalization bounds for deep nets via a compression approach](https://mlsanity.com/paper/1802.05296/) (2018)
- [Norm-Based Capacity Control in Neural Networks](https://mlsanity.com/paper/1503.00036/) (2015)
- [Size-Independent Sample Complexity of Neural Networks](https://mlsanity.com/paper/1712.06541/) (2017)
- [Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality](https://mlsanity.com/paper/1810.08033/) (2018)
- [LassoNet: A Neural Network with Feature Sparsity](https://mlsanity.com/paper/1907.12207/) (2019)
- [The Real-World-Weight Cross-Entropy Loss Function: Modeling the Costs of Mislabeling](https://mlsanity.com/paper/2001.00570/) (2020)
- [What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theory](https://mlsanity.com/paper/2105.03361/) (2021)
- [Deep Neural Networks Learn Non-Smooth Functions Effectively](https://mlsanity.com/paper/1802.04474/) (2018)
- [Generalization in Deep Networks: The Role of Distance from Initialization](https://mlsanity.com/paper/1901.01672/) (2019)
- [Performance Metrics (Error Measures) in Machine Learning Regression, Forecasting and Prognostics: Properties and Typology](https://mlsanity.com/paper/1809.03006/) (2018)
- [An Introductory Guide to Fano's Inequality with Applications in Statistical Estimation](https://mlsanity.com/paper/1901.00555/) (2019)
- [Statistical Guarantees for Regularized Neural Networks](https://mlsanity.com/paper/2006.00294/) (2020)
- [On the minimax optimality and superiority of deep neural network learning over sparse parameter spaces](https://mlsanity.com/paper/1905.09195/) (2019)
- [Convergence rates of deep ReLU networks for multiclass classification](https://mlsanity.com/paper/2108.00969/) (2021)
- [How Many Samples are Needed to Estimate a Convolutional or Recurrent Neural Network?](https://mlsanity.com/paper/1805.07883/) (2018)
- [Estimation error analysis of deep learning on the regression problem on the variable exponent Besov space](https://mlsanity.com/paper/2009.11285/) (2020)
- [Deep Learning meets Nonparametric Regression: Are Weight-Decayed DNNs Locally Adaptive?](https://mlsanity.com/paper/2204.09664/) (2022)
- [Statistical guarantees for sparse deep learning](https://mlsanity.com/paper/2212.05427/) (2022)
- [Layer Sparsity in Neural Networks](https://mlsanity.com/paper/2006.15604/) (2020)

## Built on by (most cited first, 5 of 5)

- [Regularization can make diffusion models more efficient](https://mlsanity.com/paper/2502.09151/) (2025)
- [Learning solutions of parameterized stiff ODEs using Gaussian processes](https://mlsanity.com/paper/2511.05990/) (2025)
- [Generative Model via Quantile Assignment](https://mlsanity.com/paper/2602.18216/) (2026)
- [How Many Training Samples Are Needed for the Inverse Kinematics Solutions by Artificial Neural Networks](https://mlsanity.com/paper/2605.23583/) (2026)
- [Beyond Humans: Multispecies Animal Face Recognition Using Transfer Learning](https://mlsanity.com/paper/2606.09353/) (2026)

## Abstract (arXiv)

Neural networks have become standard tools in many areas, yet many important statistical questions remain open. This paper studies the question of how much data are needed to train a ReLU feed-forward neural network. Our theoretical and empirical results suggest that the generalization error of ReLU feed-forward neural networks scales at the rate 1/√n in the sample size n rather than the usual "parametric rate" 1/n. Thus, broadly speaking, our results underpin the common belief that neural networks need "many" training samples.