# Being Properly Improper

arXiv: https://arxiv.org/abs/2106.09920 · HTML: https://mlsanity.com/paper/2106.09920/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 88 references that are in the map.
- First posted on arXiv 18 Jun 2021.

## Builds on (most cited first, 30 of 33)

- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [Robust Loss Functions under Label Noise for Deep Neural Networks](https://mlsanity.com/paper/1712.09482/) (2017)
- [AutoML: A Survey of the State-of-the-Art](https://mlsanity.com/paper/1908.00709/) (2019)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [Normalized Loss Functions for Deep Learning with Noisy Labels](https://mlsanity.com/paper/2006.13554/) (2020)
- [A General and Adaptive Robust Loss Function](https://mlsanity.com/paper/1701.03077/) (2017)
- [Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Rates](https://mlsanity.com/paper/1910.03231/) (2019)
- [Learning Noise Transition Matrix from Only Noisy Labels via Total Variation Regularization](https://mlsanity.com/paper/2102.02414/) (2021)
- [Robust Bi-Tempered Logistic Loss Based on Bregman Divergences](https://mlsanity.com/paper/1906.03361/) (2019)
- [Data Poisoning Attacks to Deep Learning Based Recommender Systems](https://mlsanity.com/paper/2101.02644/) (2021)
- [Provably Robust Boosted Decision Stumps and Trees against Adversarial Attacks](https://mlsanity.com/paper/1906.03526/) (2019)
- [A Tunable Loss Function for Robust Classification: Calibration, Landscape, and Generalization](https://mlsanity.com/paper/1906.02314/) (2019)
- [Systematic Evaluation of Backdoor Data Poisoning Attacks on Image Classifiers](https://mlsanity.com/paper/2004.11514/) (2020)
- [A One-step Approach to Covariate Shift Adaptation](https://mlsanity.com/paper/2007.04043/) (2020)
- [On Focal Loss for Class-Posterior Probability Estimation: A Theoretical Perspective](https://mlsanity.com/paper/2011.09172/) (2020)
- [Projection Efficient Subgradient Method and Optimal Nonsmooth Frank-Wolfe Method](https://mlsanity.com/paper/2010.01848/) (2020)
- [Boosting Simple Learners](https://mlsanity.com/paper/2001.11704/) (2020)
- [Efficient, Noise-Tolerant, and Private Learning via Boosting](https://mlsanity.com/paper/2002.01100/) (2020)
- [Two-temperature logistic regression based on the Tsallis divergence](https://mlsanity.com/paper/1705.07210/) (2017)
- [A Tunable Loss Function for Binary Classification](https://mlsanity.com/paper/1902.04639/) (2019)
- [Supervised Learning: No Loss No Cry](https://mlsanity.com/paper/2002.03555/) (2020)

## Built on by (most cited first, 11 of 11)

- [Boosting with Tempered Exponential Measures](https://mlsanity.com/paper/2306.05487/) (2023)
- [Generative Trees: Adversarial and Copycat](https://mlsanity.com/paper/2201.11205/) (2022)
- [Fair Wrapping for Black-box Predictions](https://mlsanity.com/paper/2201.12947/) (2022)
- [Proper losses regret at least 1/2-order](https://mlsanity.com/paper/2407.10417/) (2024)
- [LegendreTron: Uprising Proper Multiclass Loss Learning](https://mlsanity.com/paper/2301.11695/) (2023)
- [What killed the Convex Booster ?](https://mlsanity.com/paper/2205.09628/) (2022)
- [AugLoss: A Robust Augmentation-based Fine Tuning Methodology](https://mlsanity.com/paper/2206.02286/) (2022)
- [Smoothly Giving up: Robustness for Simple Models](https://mlsanity.com/paper/2302.09114/) (2023)
- [Label Noise Robustness for Domain-Agnostic Fair Corrections via Nearest Neighbors Label Spreading](https://mlsanity.com/paper/2406.09561/) (2024)
- [DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)](https://mlsanity.com/paper/2507.07855/) (2025)
- [Minimax Generalized Cross-Entropy](https://mlsanity.com/paper/2603.19874/) (2026)

## Abstract (arXiv)

Properness for supervised losses stipulates that the loss function shapes the learning algorithm towards the true posterior of the data generating distribution. Unfortunately, data in modern machine learning can be corrupted or twisted in many ways. Hence, optimizing a proper loss function on twisted data could perilously lead the learning algorithm towards the twisted posterior, rather than to the desired clean posterior. Many papers cope with specific twists (e.g., label/feature/adversarial noise), but there is a growing need for a unified and actionable understanding atop properness. Our chief theoretical contribution is a generalization of the properness framework with a notion called twist-properness, which delineates loss functions with the ability to "untwist" the twisted posterior into the clean posterior. Notably, we show that a nontrivial extension of a loss function called α-loss, which was first introduced in information theory, is twist-proper. We study the twist-proper α-loss under a novel boosting algorithm, called PILBoost, and provide formal and experimental results for this algorithm. Our overarching practical conclusion is that the twist-proper α-loss outperforms the proper log-loss on several variants of twisted data.