# Controlled abstention neural networks for identifying skillful predictions for classification problems

arXiv: https://arxiv.org/abs/2104.08281 · HTML: https://mlsanity.com/paper/2104.08281/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 34 references that are in the map.
- First posted on arXiv 16 Apr 2021.

## Builds on (most cited first, 8 of 8)

- [SelectiveNet: A Deep Neural Network with an Integrated Reject Option](https://mlsanity.com/paper/1901.09192/) (2019)
- [Combating Label Noise in Deep Learning Using Abstention](https://mlsanity.com/paper/1905.10964/) (2019)
- [Stochastic Super-Resolution for Downscaling Time-Evolving Atmospheric Fields with a Generative Adversarial Network](https://mlsanity.com/paper/2005.10374/) (2020)
- [Machine Learning for Stochastic Parameterization: Generative Adversarial Networks in the Lorenz '96 Model](https://mlsanity.com/paper/1909.04711/) (2019)
- [Physically Interpretable Neural Networks for the Geosciences: Applications to Earth System Variability](https://mlsanity.com/paper/1912.01752/) (2019)
- [Neural Network Attribution Methods for Problems in Geoscience: A Novel Synthetic Benchmark Dataset](https://mlsanity.com/paper/2103.10005/) (2021)
- [Resampling with neural networks for stochastic parameterization in multiscale systems](https://mlsanity.com/paper/2004.01457/) (2020)
- [A Unified Plug-and-Play Framework for Effective Data Denoising and Robust Abstention](https://mlsanity.com/paper/2009.12027/) (2020)

## Built on by (most cited first, 3 of 3)

- [Distributional neural networks for electricity price forecasting](https://mlsanity.com/paper/2207.02832/) (2022)
- [CIRA Guide to Custom Loss Functions for Neural Networks in Environmental Sciences -- Version 1](https://mlsanity.com/paper/2106.09757/) (2021)
- [Survey on Leveraging Uncertainty Estimation Towards Trustworthy Deep Neural Networks: The Case of Reject Option and Post-training Processing](https://mlsanity.com/paper/2304.04906/) (2023)

## Abstract (arXiv)

The earth system is exceedingly complex and often chaotic in nature, making prediction incredibly challenging: we cannot expect to make perfect predictions all of the time. Instead, we look for specific states of the system that lead to more predictable behavior than others, often termed "forecasts of opportunity." When these opportunities are not present, scientists need prediction systems that are capable of saying "I don't know." We introduce a novel loss function, termed the "NotWrong loss", that allows neural networks to identify forecasts of opportunity for classification problems. The NotWrong loss introduces an abstention class that allows the network to identify the more confident samples and abstain (say "I don't know") on the less confident samples. The abstention loss is designed to abstain on a user-defined fraction of the samples via a PID controller. Unlike many machine learning methods used to reject samples post-training, the NotWrong loss is applied during training to preferentially learn from the more confident samples. We show that the NotWrong loss outperforms other existing loss functions for multiple climate use cases. The implementation of the proposed loss function is straightforward in most network architectures designed for classification as it only requires the addition of an abstention class to the output layer and modification of the loss function.