# Smoothly Giving up: Robustness for Simple Models

arXiv: https://arxiv.org/abs/2302.09114 · HTML: https://mlsanity.com/paper/2302.09114/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 30 of its 80 references that are in the map.
- First posted on arXiv 17 Feb 2023.

## Builds on (most cited first, 30 of 30)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [AutoML: A Survey of the State-of-the-Art](https://mlsanity.com/paper/1908.00709/) (2019)
- [Normalized Loss Functions for Deep Learning with Noisy Labels](https://mlsanity.com/paper/2006.13554/) (2020)
- [A General and Adaptive Robust Loss Function](https://mlsanity.com/paper/1701.03077/) (2017)
- [Deep Learning is Robust to Massive Label Noise](https://mlsanity.com/paper/1705.10694/) (2017)
- [Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Rates](https://mlsanity.com/paper/1910.03231/) (2019)
- [Learning with Symmetric Label Noise: The Importance of Being Unhinged](https://mlsanity.com/paper/1505.07634/) (2015)
- [Understanding and Improving Early Stopping for Learning with Noisy Labels](https://mlsanity.com/paper/2106.15853/) (2021)
- [The Landscape of Empirical Risk for Non-convex Losses](https://mlsanity.com/paper/1607.06534/) (2016)
- [Combating Label Noise in Deep Learning Using Abstention](https://mlsanity.com/paper/1905.10964/) (2019)
- [PolyLoss: A Polynomial Expansion Perspective of Classification Loss Functions](https://mlsanity.com/paper/2204.12511/) (2022)
- [Instance-dependent Label-noise Learning under a Structural Causal Model](https://mlsanity.com/paper/2109.02986/) (2021)
- [Regularization via Structural Label Smoothing](https://mlsanity.com/paper/2001.01900/) (2020)
- [Accuracy, Interpretability, and Differential Privacy via Explainable Boosting](https://mlsanity.com/paper/2106.09680/) (2021)
- [A Tunable Loss Function for Robust Classification: Calibration, Landscape, and Generalization](https://mlsanity.com/paper/1906.02314/) (2019)
- [A Tunable Loss Function for Binary Classification](https://mlsanity.com/paper/1902.04639/) (2019)
- [Being Properly Improper](https://mlsanity.com/paper/2106.09920/) (2021)
- [Realizing GANs via a Tunable Loss Function](https://mlsanity.com/paper/2106.05232/) (2021)
- [Learning Not to Learn in the Presence of Noisy Labels](https://mlsanity.com/paper/2002.06541/) (2020)
- [Boosting with the Logistic Loss is Consistent](https://mlsanity.com/paper/1305.2648/) (2013)
- [On the alpha-loss Landscape in the Logistic Model](https://mlsanity.com/paper/2006.12406/) (2020)
- [Non-Convex Boosting Overcomes Random Label Noise](https://mlsanity.com/paper/1409.2905/) (2014)
- [The perils of being unhinged: On the accuracy of classifiers minimizing a noise-robust convex loss](https://mlsanity.com/paper/2112.04590/) (2021)
- [Agnostic Learnability of Halfspaces via Logistic Loss](https://mlsanity.com/paper/2201.13419/) (2022)
- [What killed the Convex Booster ?](https://mlsanity.com/paper/2205.09628/) (2022)

## Abstract (arXiv)

There is a growing need for models that are interpretable and have reduced energy and computational cost (e.g., in health care analytics and federated learning). Examples of algorithms to train such models include logistic regression and boosting. However, one challenge facing these algorithms is that they provably suffer from label noise; this has been attributed to the joint interaction between oft-used convex loss functions and simpler hypothesis classes, resulting in too much emphasis being placed on outliers. In this work, we use the margin-based α-loss, which continuously tunes between canonical convex and quasi-convex losses, to robustly train simple models. We show that the α hyperparameter smoothly introduces non-convexity and offers the benefit of "giving up" on noisy training examples. We also provide results on the Long-Servedio dataset for boosting and a COVID-19 survey dataset for logistic regression, highlighting the efficacy of our approach across multiple relevant domains.