mlsanityData to 14 Oct 2026

Deep Neural Networks Learn Non-Smooth Functions Effectively

arXiv 1802.04474 ↗First posted 13 Feb 2018stat.MLcited by 57

Not on the map. Why?

Abstract (arXiv)

We theoretically discuss why deep neural networks (DNNs) performs better than other models in some cases by investigating statistical properties of DNNs for non-smooth functions. While DNNs have empirically shown higher performance than other standard methods, understanding its mechanism is still a challenging problem. From an aspect of the statistical theory, it is known many standard methods attain the optimal rate of generalization errors for smooth functions in large sample asymptotics, and thus it has not been straightforward to find theoretical advantages of DNNs. This paper fills this gap by considering learning of a certain class of non-smooth functions, which was not covered by the previous theory. We derive the generalization error of estimators by DNNs with a ReLU activation, and show that convergence rates of the generalization by DNNs are almost optimal to estimate the non-smooth functions, while some of the popular models do not attain the optimal rate. In addition, our theoretical result provides guidelines for selecting an appropriate number of layers and edges of DNNs. We provide numerical experiments to support the theoretical results.

Key facts. Cited by 57 papers in the map (data through 14 Oct 2026). First posted on arXiv 13 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 12; 2023: 9; 2024: 16; 2025: 5; 2026: 4.

57 papers.

Sort:

By area:

  1. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 150
  2. 2021Near-Minimax Optimal Estimation With Shallow ReLU Neural Networkscited by 41
  3. 2023Approximation and Estimation Ability of Transformers for Sequence-to-Sequence Functions with Infinite Dimensional Inputcited by 32
  4. 2023Generalization Ability of Wide Neural Networks on ℝcited by 30
  5. 2021Theory of Deep Convolutional Neural Networks III: Approximating Radial Functionscited by 22
  6. 2024A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Modelscited by 20
  7. 2022Improving aircraft performance using machine learning: a reviewcited by 19
  8. 2021On the rate of convergence of a classifier based on a Transformer encodercited by 16
  9. 2023Classification with Deep Neural Networks and Logistic Losscited by 16
  10. 2024Neural Scaling Laws of Deep ReLU and Deep Operator Network: A Theoretical Studycited by 14
  11. 2021A likelihood approach to nonparametric estimation of a singular distribution using deep generative modelscited by 12
  12. 2022Adaptive deep learning for nonlinear time series modelscited by 11
38 more
  1. 2021Optimal learning of high-dimensional classification problems using deep neural networkscited by 9
  2. 2023Testing for the Markov Property in Time Series via Deep Conditional Generative Learningcited by 9
  3. 2022A Review of Machine Learning Methods Applied to Structural Dynamics and Vibroacousticcited by 7
  4. 2022On the inability of Gaussian process regression to optimally learn compositional functionscited by 7
  5. 2024Misclassification bounds for PAC-Bayesian sparse deep learningcited by 7
  6. 2023Sup-Norm Convergence of Deep Neural Network Estimator for Nonparametric Regression by Adversarial Trainingcited by 6
  7. 2024State Space Models are Provably Comparable to Transformers in Dynamic Token Selectioncited by 6
  8. 2021Estimation of a regression function on a manifold by fully connected deep neural networkscited by 5
  9. 2022On generalization bounds for deep networks based on loss surface implicit regularizationcited by 5
  10. 2022Quantile Off-Policy Evaluation via Deep Conditional Generative Learningcited by 5
  11. 2024How many samples are needed to train a deep neural network?cited by 5
  12. 2021Deep neural network surrogates for non-smooth quantities of interest in shape uncertainty quantificationcited by 4
  13. 2021Testing Directed Acyclic Graph via Structural, Supervised and Generative Adversarial Learningcited by 4
  14. 2021Generalization Performance of Empirical Risk Minimization on Over-parameterized Deep ReLU Netscited by 4
  15. 2022Analysis of convolutional neural network image classifiers in a rotationally symmetric modelcited by 4
  16. 2022Smooth Sailing: Improving Active Learning for Pre-trained Language Models with Representation Smoothness Analysiscited by 4
  17. 2023Exploring the Approximation Capabilities of Multiplicative Neural Networks for Smooth Functionscited by 4
  18. 2025On the Rate of Convergence of Kolmogorov-Arnold Network Regression Estimatorscited by 4
  19. 2024Deep Neural Networks are Adaptive to Function Regularity and Data Distribution in Approximation and Estimationcited by 3
  20. 2024Deep learning from strongly mixing observations: Sparse-penalized regularization and minimax optimalitycited by 3
  21. 2024Mitigating Suboptimality of Deterministic Policy Gradients in Complex Q-functionscited by 3
  22. 2025Approximation of Permutation Invariant Polynomials by Transformers: Efficient Construction in Column-Sizecited by 3
  23. 2025Deep learning with missing datacited by 3
  24. 2021Analysis of convolutional neural network image classifiers in a hierarchical max-pooling model with additional local poolingcited by 2
  25. 2022Distributional Reinforcement Learning for Scheduling of Chemical Production Processescited by 2
  26. 2024Nonsmooth Nonparametric Regression via Fractional Laplacian Eigenmapscited by 2
  27. 2024Causal Deepsets for Off-policy Evaluation under Spatial or Spatio-temporal Interferencescited by 2
  28. 2024Gradient Descent Finds Over-Parameterized Neural Networks with Sharp Generalization for Nonparametric Regressioncited by 2
  29. 2022Optimal Convergence Rates of Deep Convolutional Neural Networks: Additive Ridge Functionscited by 1
  30. 2023Pathway to a fully data-driven geotechnics: lessons from materials informaticscited by 1
  31. 2026Regret Analysis of Guided Diffusion for Black-Box Optimization over Structured Inputscited by 1
  32. 2021The Randomness of Input Data Spaces is an A Priori Predictor for Generalizationcited by 0
  33. 2022Interplay between depth of neural networks and locality of target functionscited by 0
  34. 2023Mathematical Challenges in Deep Learningcited by 0
  35. 2023Lifting the Veil: Unlocking the Power of Depth in Q-learningcited by 0
  36. 2024A least distance estimator for a multivariate regression model using deep neural networkscited by 0
  37. 2024Dimension-independent learning rates for high-dimensional classification problemscited by 0
  38. 2024ReLU neural network approximation to piecewise constant functionscited by 0

50 most cited of 57 All 57 (JSON)