# Outlier-robust neural network training: variation regularization meets trimmed loss to prevent functional breakdown

arXiv: https://arxiv.org/abs/2308.02293 · HTML: https://mlsanity.com/paper/2308.02293/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 60 references that are in the map.
- First posted on arXiv 4 Aug 2023.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Optimization Methods for Large-Scale Machine Learning](https://mlsanity.com/paper/1606.04838/) (2016)
- [Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programming](https://mlsanity.com/paper/1309.5549/) (2013)
- [Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What's next](https://mlsanity.com/paper/2201.05624/) (2022)
- [Error bounds for approximations with deep ReLU networks](https://mlsanity.com/paper/1610.01145/) (2016)
- [Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://mlsanity.com/paper/1703.09039/) (2017)
- [A unified deep artificial neural network approach to partial differential equations in complex geometries](https://mlsanity.com/paper/1711.06464/) (2017)
- [Data augmentation instead of explicit regularization](https://mlsanity.com/paper/1806.03852/) (2018)
- [What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theory](https://mlsanity.com/paper/2105.03361/) (2021)
- [Towards Understanding Regularization in Batch Normalization](https://mlsanity.com/paper/1809.00846/) (2018)
- [Neural Splines: Fitting 3D Surfaces with Infinitely-Wide Neural Networks](https://mlsanity.com/paper/2006.13782/) (2020)
- [Understanding neural networks with reproducing kernel Banach spaces](https://mlsanity.com/paper/2109.09710/) (2021)
- [Ridges, Neural Networks, and the Radon Transform](https://mlsanity.com/paper/2203.02543/) (2022)
- [Structured Sparsity Inducing Adaptive Optimizers for Deep Learning](https://mlsanity.com/paper/2102.03869/) (2021)
- [Stochastic Optimization for DC Functions and Non-smooth Non-convex Regularizers with Non-asymptotic Convergence](https://mlsanity.com/paper/1811.11829/) (2018)
- [Non-asymptotic Analysis of Stochastic Methods for Non-Smooth Non-Convex Regularized Problems](https://mlsanity.com/paper/1902.07672/) (2019)
- [Higher-order total variation approaches and generalisations](https://mlsanity.com/paper/1912.01587/) (2019)
- [Stochastic Proximal Methods for Non-Smooth Non-Convex Constrained Sparse Optimization](https://mlsanity.com/paper/1905.10188/) (2019)
- [Simple Stochastic Gradient Methods for Non-Smooth Non-Convex Regularized Optimization](https://mlsanity.com/paper/1901.08369/) (2019)

## Abstract (arXiv)

In this study, we tackle the challenge of outlier-robust predictive modeling using highly expressive neural networks. Our approach integrates two key components: (1) a transformed trimmed loss (TTL), a computationally efficient variant of the classical trimmed loss, and (2) higher-order variation regularization (HOVR), which imposes smoothness constraints on the prediction function. While traditional robust statistics typically assume low-complexity models such as linear and kernel models, applying TTL alone to modern neural networks may fail to ensure robustness, as their high expressive power allows them to fit both inliers and outliers, even when a robust loss is used. To address this, we revisit the traditional notion of breakdown point and adapt it to the nonlinear function setting, introducing a regularization scheme via HOVR that controls the model's capacity and suppresses overfitting to outliers. We theoretically establish that our training procedure retains a high functional breakdown point, thereby ensuring robustness to outlier contamination. We develop a stochastic optimization algorithm tailored to this framework and provide a theoretical guarantee of its convergence.