mlsanityData to 14 Oct 2026

Nonparametric regression using deep neural networks with ReLU activation function

arXiv 1708.06633 ↗First posted 22 Aug 2017math.ST cs.LG stat.MLcited by 433

Not on the map. Why?

Abstract (arXiv)

Consider the multivariate nonparametric regression model. It is shown that estimators based on sparsely connected deep neural networks with ReLU activation function and properly chosen network architecture achieve the minimax rates of convergence (up to log n-factors) under a general composition assumption on the regression function. The framework includes many well-studied structural constraints such as (generalized) additive models. While there is a lot of flexibility in the network architecture, the tuning parameter is the sparsity of the network. Specifically, we consider large networks with number of potential network parameters exceeding the sample size. The analysis gives some insights into why multilayer feedforward neural networks perform well in practice. Interestingly, for ReLU activation function the depth (number of layers) of the neural network architectures plays an important role and our theory suggests that for nonparametric regression, scaling the network depth with the sample size is natural. It is also shown that under the composition assumption wavelet estimators can only achieve suboptimal rates.

Key facts. Cited by 433 papers in the map (data through 14 Oct 2026). First posted on arXiv 22 Aug 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 47; 2022: 61; 2023: 71; 2024: 92; 2025: 75; 2026: 87.

433 papers.

Sort:

By area:

  1. 2024KAN: Kolmogorov-Arnold Networkscited by 649
  2. 2023Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Datacited by 174
  3. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  4. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 143
  5. 2024A Tale of Tails: Model Collapse as a Change of Scaling Lawscited by 130
  6. 2024A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Datacited by 93
  7. 2024Operator Learning: Algorithms and Analysiscited by 85
  8. 2024Model Collapse Demystified: The Case of Regressioncited by 82
  9. 2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimizationcited by 73
  10. 2023How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Modelcited by 71
  11. 2022Deep Nonparametric Estimation of Operators between Infinite Dimensional Spacescited by 45
  12. 2023DNeRV: Modeling Inherent Dynamics via Difference Neural Representation for Videoscited by 42
38 more
  1. 2021Near-Minimax Optimal Estimation With Shallow ReLU Neural Networkscited by 41
  2. 2024Transformers are Minimax Optimal Nonparametric In-Context Learnerscited by 37
  3. 2024On the expressiveness and spectral bias of KANscited by 37
  4. 2021Early-stopped neural networks are consistentcited by 34
  5. 2021Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networkscited by 32
  6. 2023Approximation and Estimation Ability of Transformers for Sequence-to-Sequence Functions with Infinite Dimensional Inputcited by 32
  7. 2021Approximation and Learning with Deep Convolutional Models: a Kernel Perspectivecited by 31
  8. 2022Off-Policy Confidence Interval Estimation with Confounded Markov Decision Processcited by 31
  9. 2021On the Existence of Universal Lottery Ticketscited by 30
  10. 2023Generalization Ability of Wide Neural Networks on ℝcited by 30
  11. 2024Applied Causal Inference Powered by ML and AIcited by 30
  12. 2024Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Datacited by 30
  13. 2022Simultaneous approximation of a smooth function and its derivatives by deep neural networks with piecewise-polynomial activationscited by 29
  14. 2021Consistent Sparse Deep Learning: Theory and Computationcited by 27
  15. 2023A Primer on Bayesian Neural Networks: Review and Debatescited by 27
  16. 2024Probing the Latent Hierarchical Structure of Data via Diffusion Modelscited by 26
  17. 2022How do noise tails impact on deep ReLU networks?cited by 25
  18. 2021Wasserstein Generative Learning of Conditional Distributioncited by 24
  19. 2023Nonparametric regression using over-parameterized shallow ReLU neural networkscited by 24
  20. 2021Theory of Deep Convolutional Neural Networks III: Approximating Radial Functionscited by 22
  21. 2022Approximation bounds for norm constrained neural networks with applications to regression and GANscited by 22
  22. 2023Distribution learning via neural differential equations: a nonparametric statistical perspectivecited by 22
  23. 2021Convergence rates of deep ReLU networks for multiclass classificationcited by 21
  24. 2023Deep Learning Meets Sparse Regularization: A Signal Processing Perspectivecited by 21
  25. 2026Demystifying Group Relative Policy Optimization: Its Policy Gradient is a U-Statisticcited by 21
  26. 2023Causal Deep Learningcited by 20
  27. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
  28. 2023Deep Operator Learning Lessens the Curse of Dimensionality for PDEscited by 18
  29. 2023Sampling-based Nystr"om Approximation and Kernel Quadraturecited by 17
  30. 2024U-Nets as Belief Propagation: Efficient Classification, Denoising, and Diffusion in Generative Hierarchical Modelscited by 17
  31. 2021Universal Consistency of Deep Convolutional Neural Networkscited by 16
  32. 2021On the rate of convergence of a classifier based on a Transformer encodercited by 16
  33. 2022What Can Be Learnt With Wide Convolutional Neural Networks?cited by 16
  34. 2022Theoretical analysis of deep neural networks for temporally dependent observationscited by 16
  35. 2023Extreme sparsification of physics-augmented neural networks for interpretable model discovery in mechanicscited by 16
  36. 2021Significance tests of feature relevance for a black-box learnercited by 15
  37. 2023Deep Neural Networks for Nonparametric Interaction Models with Diverging Dimensioncited by 15
  38. 2023Optimal rates of approximation by shallow ReLUk neural networks and applications to nonparametric regressioncited by 15

50 most cited of 433 All 433 (JSON)