Window-Based Distribution Shift Detection for Deep Neural Networks
Not on the map. Why?
Abstract (arXiv)
To deploy and operate deep neural models in production, the quality of their predictions, which might be contaminated benignly or manipulated maliciously by input distributional deviations, must be monitored and assessed. Specifically, we study the case of monitoring the healthy operation of a deep neural network (DNN) receiving a stream of data, with the aim of detecting input distributional deviations over which the quality of the network's predictions is potentially damaged. Using selective prediction principles, we propose a distribution deviation detection method for DNNs. The proposed method is derived from a tight coverage generalization bound computed over a sample of instances drawn from the true underlying distribution. Based on this bound, our detector continuously monitors the operation of the network out-of-sample over a test window and fires off an alarm whenever a deviation is detected. Our novel detection method performs on-par or better than the state-of-the-art, while consuming substantially lower computation time (five orders of magnitude reduction) and space complexities. Unlike previous methods, which require at least linear dependence on the size of the source distribution for each detection, rendering them inapplicable to ``Google-Scale'' datasets, our approach eliminates this dependence, making it suitable for real-world applications.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 28 of its 61 references that are in the map. First posted on arXiv 19 Oct 2022.
Builds on
28 of 61 references.
References · 28
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2014Explaining and Harnessing Adversarial Examplescited by 7,364
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
- 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
- 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
- 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,877
- 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
- 2019Searching for MobileNetV3cited by 1,994
- 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,485
- 2019Natural Adversarial Examplescited by 1,408
16 more
- 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,328
- 2021Scaling Vision Transformerscited by 999
- 2018Deep Anomaly Detection with Outlier Exposurecited by 942
- 2017Selective Classification for Deep Neural Networkscited by 852
- 2019Likelihood Ratios for Out-of-Distribution Detectioncited by 453
- 2018Detecting and Correcting for Label Shift with Black Box Predictorscited by 366
- 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
- 2018Deep Anomaly Detection Using Geometric Transformationscited by 269
- 2018Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shiftcited by 243
- 2016Variational Autoencoder for Deep Learning of Images, Labels and Captionscited by 137
- 2021Predicting with Confidence on Unseen Distributionscited by 116
- 2018Bias-Reduced Uncertainty Estimation for Deep Neural Classifierscited by 113
- 2016Generative Models and Model Criticism via Optimized Maximum Mean Discrepancycited by 100
- 2021Active Testing: Sample-Efficient Model Evaluationcited by 82
- 2021Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learningcited by 82
- 2017Choosing Between Methods of Combining p-valuescited by 17
Built on by
Citing papers per year: 2025: 2.
2 papers.