mlsanityData to 7 Oct 2026

How do noise tails impact on deep ReLU networks?

arXiv 2203.10418 ↗First posted 20 Mar 2022math.ST stat.ML stat.THcited by 25

Not on the map. Why?

Abstract (arXiv)

This paper investigates the stability of deep ReLU neural networks for nonparametric regression under the assumption that the noise has only a finite p-th moment. We unveil how the optimal rate of convergence depends on p, the degree of smoothness and the intrinsic dimension in a class of nonparametric regression functions with hierarchical composition structure when both the adaptive Huber loss and deep ReLU neural networks are used. This optimal rate of convergence cannot be obtained by the ordinary least squares but can be achieved by the Huber loss with a properly chosen parameter that adapts to the sample size, smoothness, and moment parameters. A concentration inequality for the adaptive Huber ReLU neural network estimators with allowable optimization errors is also derived. To establish a matching lower bound within the class of neural network estimators using the Huber loss, we employ a different strategy from the traditional route: constructing a deep ReLU network estimator that has a better empirical loss than the true function and the difference between these two functions furnishes a low bound. This step is related to the Huberization bias, yet more critically to the approximability of deep ReLU networks. As a result, we also contribute some new results on the approximation theory of deep ReLU neural networks.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 15 of its 54 references that are in the map. First posted on arXiv 20 Mar 2022.

Builds on

15 of 54 references.

References · 15

  1. 2016Error bounds for approximations with deep ReLU networkscited by 567
  2. 2017Nonparametric regression using deep neural networks with ReLU activation functioncited by 431
  3. 2017Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networkscited by 304
  4. 2016Benefits of depth in neural networkscited by 248
  5. 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 165
  6. 2020Deep Network Approximation for Smooth Functionscited by 163
  7. 2018Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionalitycited by 160
  8. 2019Deep Network Approximation Characterized by Number of Neuronscited by 120
  9. 2021Optimal Approximation Rate of ReLU Networks in terms of Width and Depthcited by 95
  10. 2018Deep Neural Networks for Estimation and Inferencecited by 62
  11. 2017Adaptive Huber Regressioncited by 59
  12. 2020Quantile regression with deep ReLU Networks: Estimators and minimax ratescited by 24
3 more
  1. 2016A Shrinkage Principle for Heavy-Tailed Data: High-Dimensional Robust Low-Rank Matrix Recoverycited by 23
  2. 2020Risk Bounds for Robust Deep Learningcited by 9
  3. 2019On Least Squares Estimation under Heteroscedastic and Heavy-Tailed Errorscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 5; 2025: 7; 2026: 10.

25 papers.

Sort:

By area:

  1. 2023Deep Neural Networks for Nonparametric Interaction Models with Diverging Dimensioncited by 15
  2. 2023UTOPIA: Universally Trainable Optimal Prediction Intervals Aggregationcited by 8
  3. 2024Causality Pursuit from Heterogeneous Environments via Neural Adversarial Invariance Learningcited by 7
  4. 2024Robust deep learning from weakly dependent datacited by 5
  5. 2024Deep learning from strongly mixing observations: Sparse-penalized regularization and minimax optimalitycited by 3
  6. 2025Wasserstein Distributionally Robust Nonparametric Regressioncited by 3
  7. 2026Optimally taming biases in black-box models for efficient semiparametric estimationcited by 3
  8. 2024Approximation Bounds for Recurrent Neural Networks with Application to Regressioncited by 2
  9. 2025Factor Informed Double Deep Learning For Average Treatment Effect Estimationcited by 2
  10. 2025A general framework for deep learningcited by 2
  11. 2025Adaptive Classification of Interval-Valued Time Seriescited by 1
  12. 2025CINDES: Classification induced neural density estimator and simulatorcited by 1
13 more
  1. 2026Optimal use of a black-box learner in semiparametric estimationcited by 1
  2. 2022Robust High-dimensional Tuning Free Multiple Testingcited by 0
  3. 2024Consistency for Large Neural Networks: Regression and Classificationcited by 0
  4. 2025Boosting Statistic Learning with Synthetic Data from Pretrained Large Modelscited by 0
  5. 2025Understanding Robust Machine Learning for Nonparametric Regression with Heavy-Tailed Noisecited by 0
  6. 2026Deep regression learning from dependent observations with minimum error entropy principlecited by 0
  7. 2026On Uniform Error Bounds for Kernel Regression under Non-Gaussian Noisecited by 0
  8. 2026Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activationscited by 0
  9. 2026Minimax Theory of Likelihood-Based Deep Learning for Speckle Regressioncited by 0
  10. 2026Adaptive deep nonparametric regression from dependent data under covariate shiftcited by 0
  11. 2026Prediction Inference of Time Series with Standard ReLU Deep Neural Networkscited by 0
  12. 2026Robust LassoNet: Enhancing Feature Selection in Neural Networks via Robust Loss Functionscited by 0
  13. 2026Minimax rates for learning spectral Barron functions by deep ReLU neural networkscited by 0