Evaluating Out-of-Distribution Detectors Through Adversarial Generation of Outliers
Not on the map. Why?
Abstract (arXiv)
A reliable evaluation method is essential for building a robust out-of-distribution (OOD) detector. Current robustness evaluation protocols for OOD detectors rely on injecting perturbations to outlier data. However, the perturbations are unlikely to occur naturally or not relevant to the content of data, providing a limited assessment of robustness. In this paper, we propose Evaluation-via-Generation for OOD detectors (EvG), a new protocol for investigating the robustness of OOD detectors under more realistic modes of variation in outliers. EvG utilizes a generative model to synthesize plausible outliers, and employs MCMC sampling to find outliers misclassified as in-distribution with the highest confidence by a detector. We perform a comprehensive benchmark comparison of the performance of state-of-the-art OOD detectors using EvG, uncovering previously overlooked weaknesses.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 36 of its 59 references that are in the map. First posted on arXiv 20 Aug 2022.
Builds on
36 of 59 references.
References · 36
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2013Intriguing properties of neural networkscited by 5,155
- 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
- 2014Deep Learning Face Attributes in the Wildcited by 3,693
- 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
- 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
- 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
- 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
- 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,485
- 2018Deep Anomaly Detection with Outlier Exposurecited by 942
- 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 666
- 2020RobustBench: a standardized adversarial robustness benchmarkcited by 661
24 more
- 2019Likelihood Ratios for Out-of-Distribution Detectioncited by 453
- 2018Do Deep Generative Models Know What They Don't Know?cited by 443
- 2020CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instancescited by 441
- 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 417
- 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
- 2018On the Effectiveness of Interval Bound Propagation for Training Verifiably Robust Modelscited by 328
- 2018Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problemcited by 323
- 2015A note on the evaluation of generative modelscited by 292
- 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
- 2021SSD: A Unified Framework for Self-Supervised Outlier Detectioncited by 264
- 2019Input complexity and out-of-distribution detection with likelihood-based generative modelscited by 186
- 2020Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encodercited by 145
- 2018Constructing Unrestricted Adversarial Examples with Generative Modelscited by 135
- 2019From Variational to Deterministic Autoencoderscited by 133
- 2020ATOM: Robustifying Out-of-distribution Detection Using Outlier Miningcited by 116
- 2019Towards neural networks that provably know when they don't knowcited by 69
- 2018Sampling Can Be Faster Than Optimizationcited by 68
- 2019Semantic Adversarial Attacks: Parametric Transformations That Fool Deep Classifierscited by 57
- 2019ToyADMOS: A Dataset of Miniature-Machine Operating Sounds for Anomalous Sound Detectioncited by 44
- 2020Perfect density models cannot guarantee anomaly detectioncited by 39
- 2021Autoencoding Under Normalization Constraintscited by 26
- 2022Adversarial vulnerability of powerful near out-of-distribution detectioncited by 16
- 2022RODD: A Self-Supervised Approach for Robust Out-of-Distribution Detectioncited by 13
- 2021Random Transformation of Image Brightness for Adversarial Attackcited by 3
Built on by
Citing papers per year: 2024: 1.
1 paper.