# Alternate loss functions and regression models that achieve robustness to outliers by modulating the learning rate

arXiv: https://arxiv.org/abs/2606.22068 · HTML: https://mlsanity.com/paper/2606.22068/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 35 references that are in the map.
- First posted on arXiv 20 Jun 2026.

## Builds on (most cited first, 7 of 7)

- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks](https://mlsanity.com/paper/2006.07322/) (2020)
- [On Loss Functions for Deep Neural Networks in Classification](https://mlsanity.com/paper/1702.05659/) (2017)
- [Loss Functions and Metrics in Deep Learning](https://mlsanity.com/paper/2307.02694/) (2023)
- [Loss Functions for Neural Networks for Image Processing](https://mlsanity.com/paper/1511.08861/) (2015)
- [On Mean Absolute Error for Deep Neural Network Based Vector-to-Vector Regression](https://mlsanity.com/paper/2008.07281/) (2020)
- [Statistical Properties of the log-cosh Loss Function Used in Machine Learning](https://mlsanity.com/paper/2208.04564/) (2022)

## Abstract (arXiv)

Most real-world datasets used for training supervised learning models are contaminated with noisy data and outliers leading to large prediction errors. This paper proposes a new approach for achieving robustness where the learning rate is modulated by a factor that is sensitive to outliers. In this approach a reduction of the learning rate is shown to be achieved by using alternate loss functions that are infinitely differentiable, strictly convex or quasiconvex and more closely approximate the absolute error than Huber and log-cosh losses. A comparison of the performance of regression models trained with different loss functions on a wide variety of benchmarks and datasets is presented to demonstrate the superior performance of the Square Root Loss (SRL) and Smooth Mean Absolute Error (SMAE) losses proposed in this paper. Two new robust linear regression models are presented. Highly vectorized robust parameter update formulae that take advantage of modern GPUs for both stochastic and batch gradient descent are presented.