# Can we Defend Against the Unknown? An Empirical Study About Threshold Selection for Neural Network Monitoring

arXiv: https://arxiv.org/abs/2405.08654 · HTML: https://mlsanity.com/paper/2405.08654/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 30 references that are in the map.
- First posted on arXiv 14 May 2024.

## Builds on (most cited first, 14 of 14)

- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop](https://mlsanity.com/paper/1506.03365/) (2015)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [ReAct: Out-of-distribution Detection With Rectified Activations](https://mlsanity.com/paper/2111.12797/) (2021)
- [Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data](https://mlsanity.com/paper/2002.11297/) (2020)
- [ViM: Out-Of-Distribution with Virtual-logit Matching](https://mlsanity.com/paper/2203.10807/) (2022)
- [Torchattacks: A PyTorch Repository for Adversarial Attacks](https://mlsanity.com/paper/2010.01950/) (2020)
- [Outside the Box: Abstraction-Based Monitoring of Neural Networks](https://mlsanity.com/paper/1911.09032/) (2019)
- [AugLy: Data Augmentations for Robustness](https://mlsanity.com/paper/2201.06494/) (2022)
- [Out-Of-Distribution Detection Is Not All You Need](https://mlsanity.com/paper/2211.16158/) (2022)
- [Unifying Evaluation of Machine Learning Safety Monitors](https://mlsanity.com/paper/2208.14660/) (2022)
- [Evaluation of Runtime Monitoring for UAV Emergency Landing](https://mlsanity.com/paper/2202.03059/) (2022)

## Built on by (most cited first, 2 of 2)

- [Unveiling AI's Blind Spots: An Oracle for In-Domain, Out-of-Domain, and Adversarial Errors](https://mlsanity.com/paper/2410.02384/) (2024)
- [FailSAE: Towards Interpretable Failure Prediction for Vision-Language Models via Sparse Autoencoders](https://mlsanity.com/paper/2609.04276/) (2026)

## Abstract (arXiv)

With the increasing use of neural networks in critical systems, runtime monitoring becomes essential to reject unsafe predictions during inference. Various techniques have emerged to establish rejection scores that maximize the separability between the distributions of safe and unsafe predictions. The efficacy of these approaches is mostly evaluated using threshold-agnostic metrics, such as the area under the receiver operating characteristic curve. However, in real-world applications, an effective monitor also requires identifying a good threshold to transform these scores into meaningful binary decisions. Despite the pivotal importance of threshold optimization, this problem has received little attention. A few studies touch upon this question, but they typically assume that the runtime data distribution mirrors the training distribution, which is a strong assumption as monitors are supposed to safeguard a system against potentially unforeseen threats. In this work, we present rigorous experiments on various image datasets to investigate: 1. The effectiveness of monitors in handling unforeseen threats, which are not available during threshold adjustments. 2. Whether integrating generic threats into the threshold optimization scheme can enhance the robustness of monitors.