# Understanding Robust Machine Learning for Nonparametric Regression with Heavy-Tailed Noise

arXiv: https://arxiv.org/abs/2510.09888 · HTML: https://mlsanity.com/paper/2510.09888/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 28 references that are in the map.
- First posted on arXiv 10 Oct 2025.

## Builds on (most cited first, 7 of 7)

- [A General and Adaptive Robust Loss Function](https://mlsanity.com/paper/1701.03077/) (2017)
- [Mean estimation and regression under heavy-tailed distributions--a survey](https://mlsanity.com/paper/1906.04280/) (2019)
- [Empirical risk minimization for heavy-tailed losses](https://mlsanity.com/paper/1406.2462/) (2014)
- [How do noise tails impact on deep ReLU networks?](https://mlsanity.com/paper/2203.10418/) (2022)
- [Consistency Analysis of an Empirical Minimum Error Entropy Algorithm](https://mlsanity.com/paper/1412.5272/) (2014)
- [A Statistical Learning Assessment of Huber Regression](https://mlsanity.com/paper/2009.12755/) (2020)
- [A Framework of Learning Through Empirical Gain Maximization](https://mlsanity.com/paper/2009.14250/) (2020)

## Abstract (arXiv)

We investigate robust nonparametric regression in the presence of heavy-tailed noise, where the hypothesis class may contain unbounded functions and robustness is ensured via a robust loss function ℓσ. Using Huber regression as a close-up example within Tikhonov-regularized risk minimization in reproducing kernel Hilbert spaces (RKHS), we address two central challenges: (i) the breakdown of standard concentration tools under weak moment assumptions, and (ii) the analytical difficulties introduced by unbounded hypothesis spaces. Our first message is conceptual: conventional generalization-error bounds for robust losses do not faithfully capture out-of-sample performance. We argue that learnability should instead be quantified through prediction error, namely the L2-distance to the truth f^star, which is σ-independent and directly reflects the target of robust estimation. To make this workable under unboundedness, we introduce a probabilistic effective hypothesis space that confines the estimator with high probability and enables a meaningful bias--variance decomposition under weak (1+ε)-moment conditions. Technically, we establish new comparison theorems linking the excess robust risk to the L2 prediction error up to a residual of order O(σ-2ε), clarifying the robustness--bias trade-off induced by the scale parameter σ. Building on this, we derive explicit finite-sample error bounds and convergence rates for Huber regression in RKHS that hold without uniform boundedness and under heavy-tailed noise. Our study delivers principled tuning rules, extends beyond Huber to other robust losses, and highlights prediction error, not excess generalization risk, as the fundamental lens for analyzing robust learning.