# How do noise tails impact on deep ReLU networks?

arXiv: https://arxiv.org/abs/2203.10418 · HTML: https://mlsanity.com/paper/2203.10418/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 54 references that are in the map.
- First posted on arXiv 20 Mar 2022.

## Builds on (most cited first, 15 of 15)

- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [Nonparametric regression using deep neural networks with ReLU activation function](https://mlsanity.com/paper/1708.06633/) (2017)
- [Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks](https://mlsanity.com/paper/1703.02930/) (2017)
- [Benefits of depth in neural networks](https://mlsanity.com/paper/1602.04485/) (2016)
- [Optimal approximation of continuous functions by very deep ReLU networks](https://mlsanity.com/paper/1802.03620/) (2018)
- [Deep Network Approximation for Smooth Functions](https://mlsanity.com/paper/2001.03040/) (2020)
- [Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality](https://mlsanity.com/paper/1810.08033/) (2018)
- [Deep Network Approximation Characterized by Number of Neurons](https://mlsanity.com/paper/1906.05497/) (2019)
- [Optimal Approximation Rate of ReLU Networks in terms of Width and Depth](https://mlsanity.com/paper/2103.00502/) (2021)
- [Deep Neural Networks for Estimation and Inference](https://mlsanity.com/paper/1809.09953/) (2018)
- [Adaptive Huber Regression](https://mlsanity.com/paper/1706.06991/) (2017)
- [Quantile regression with deep ReLU Networks: Estimators and minimax rates](https://mlsanity.com/paper/2010.08236/) (2020)
- [A Shrinkage Principle for Heavy-Tailed Data: High-Dimensional Robust Low-Rank Matrix Recovery](https://mlsanity.com/paper/1603.08315/) (2016)
- [Risk Bounds for Robust Deep Learning](https://mlsanity.com/paper/2009.06202/) (2020)
- [On Least Squares Estimation under Heteroscedastic and Heavy-Tailed Errors](https://mlsanity.com/paper/1909.02088/) (2019)

## Built on by (most cited first, 25 of 25)

- [Deep Neural Networks for Nonparametric Interaction Models with Diverging Dimension](https://mlsanity.com/paper/2302.05851/) (2023)
- [UTOPIA: Universally Trainable Optimal Prediction Intervals Aggregation](https://mlsanity.com/paper/2306.16549/) (2023)
- [Causality Pursuit from Heterogeneous Environments via Neural Adversarial Invariance Learning](https://mlsanity.com/paper/2405.04715/) (2024)
- [Robust deep learning from weakly dependent data](https://mlsanity.com/paper/2405.05081/) (2024)
- [Approximation Bounds for Recurrent Neural Networks with Application to Regression](https://mlsanity.com/paper/2409.05577/) (2024)
- [Deep learning from strongly mixing observations: Sparse-penalized regularization and minimax optimality](https://mlsanity.com/paper/2406.08321/) (2024)
- [Wasserstein Distributionally Robust Nonparametric Regression](https://mlsanity.com/paper/2505.07967/) (2025)
- [Optimally taming biases in black-box models for efficient semiparametric estimation](https://mlsanity.com/paper/2606.06368/) (2026)
- [Factor Informed Double Deep Learning For Average Treatment Effect Estimation](https://mlsanity.com/paper/2508.17136/) (2025)
- [A general framework for deep learning](https://mlsanity.com/paper/2512.23425/) (2025)
- [Adaptive Classification of Interval-Valued Time Series](https://mlsanity.com/paper/2504.03318/) (2025)
- [CINDES: Classification induced neural density estimator and simulator](https://mlsanity.com/paper/2510.00367/) (2025)
- [Optimal use of a black-box learner in semiparametric estimation](https://mlsanity.com/paper/2607.21541/) (2026)
- [Robust High-dimensional Tuning Free Multiple Testing](https://mlsanity.com/paper/2211.11959/) (2022)
- [Consistency for Large Neural Networks: Regression and Classification](https://mlsanity.com/paper/2409.14123/) (2024)
- [Boosting Statistic Learning with Synthetic Data from Pretrained Large Models](https://mlsanity.com/paper/2505.04992/) (2025)
- [Understanding Robust Machine Learning for Nonparametric Regression with Heavy-Tailed Noise](https://mlsanity.com/paper/2510.09888/) (2025)
- [Deep regression learning from dependent observations with minimum error entropy principle](https://mlsanity.com/paper/2603.11138/) (2026)
- [On Uniform Error Bounds for Kernel Regression under Non-Gaussian Noise](https://mlsanity.com/paper/2605.09757/) (2026)
- [Mitigating the Curse of Dimensionality in Uniform Convergence of Deep Neural Networks via Smooth Activations](https://mlsanity.com/paper/2606.05599/) (2026)
- [Minimax Theory of Likelihood-Based Deep Learning for Speckle Regression](https://mlsanity.com/paper/2607.14064/) (2026)
- [Adaptive deep nonparametric regression from dependent data under covariate shift](https://mlsanity.com/paper/2607.20309/) (2026)
- [Prediction Inference of Time Series with Standard ReLU Deep Neural Networks](https://mlsanity.com/paper/2608.15362/) (2026)
- [Robust LassoNet: Enhancing Feature Selection in Neural Networks via Robust Loss Functions](https://mlsanity.com/paper/2609.38263/) (2026)
- [Minimax rates for learning spectral Barron functions by deep ReLU neural networks](https://mlsanity.com/paper/2609.39020/) (2026)

## Abstract (arXiv)

This paper investigates the stability of deep ReLU neural networks for nonparametric regression under the assumption that the noise has only a finite p-th moment. We unveil how the optimal rate of convergence depends on p, the degree of smoothness and the intrinsic dimension in a class of nonparametric regression functions with hierarchical composition structure when both the adaptive Huber loss and deep ReLU neural networks are used. This optimal rate of convergence cannot be obtained by the ordinary least squares but can be achieved by the Huber loss with a properly chosen parameter that adapts to the sample size, smoothness, and moment parameters. A concentration inequality for the adaptive Huber ReLU neural network estimators with allowable optimization errors is also derived. To establish a matching lower bound within the class of neural network estimators using the Huber loss, we employ a different strategy from the traditional route: constructing a deep ReLU network estimator that has a better empirical loss than the true function and the difference between these two functions furnishes a low bound. This step is related to the Huberization bias, yet more critically to the approximability of deep ReLU networks. As a result, we also contribute some new results on the approximation theory of deep ReLU neural networks.