# Simple Techniques Work Surprisingly Well for Neural Network Test Prioritization and Active Learning (Replicability Study)

arXiv: https://arxiv.org/abs/2205.00664 · HTML: https://mlsanity.com/paper/2205.00664/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 47 references that are in the map.
- First posted on arXiv 2 May 2022.

## Builds on (most cited first, 19 of 19)

- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Deep Bayesian Active Learning with Image Data](https://mlsanity.com/paper/1703.02910/) (2017)
- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://mlsanity.com/paper/1705.06640/) (2017)
- [DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems](https://mlsanity.com/paper/1803.07519/) (2018)
- [MNIST-C: A Robustness Benchmark for Computer Vision](https://mlsanity.com/paper/1906.02337/) (2019)
- [Guiding Deep Learning System Testing using Surprise Adequacy](https://mlsanity.com/paper/1808.08444/) (2018)
- [Misbehaviour Prediction for Autonomous Driving Systems](https://mlsanity.com/paper/1910.04443/) (2019)
- [Test Selection for Deep Learning Systems](https://mlsanity.com/paper/1904.13195/) (2019)
- [Towards Characterizing Adversarial Defects of Deep Learning Software from the Lens of Uncertainty](https://mlsanity.com/paper/2004.11573/) (2020)
- [Reducing DNN Labelling Cost using Surprise Adequacy: An Industrial Case Study for Autonomous Driving](https://mlsanity.com/paper/2006.00894/) (2020)
- [Fail-Safe Execution of Deep Learning based Systems through Uncertainty Monitoring](https://mlsanity.com/paper/2102.00902/) (2021)
- [Uncertainty-Wizard: Fast and User-Friendly Neural Network Uncertainty Quantification](https://mlsanity.com/paper/2101.00982/) (2020)
- [A Review and Refinement of Surprise Adequacy](https://mlsanity.com/paper/2103.05939/) (2021)

## Built on by (most cited first, 27 of 27)

- [FedRC: Tackling Diverse Distribution Shifts Challenge in Federated Learning by Robust Clustering](https://mlsanity.com/paper/2301.12379/) (2023)
- [FAN: Fourier Analysis Networks](https://mlsanity.com/paper/2410.02675/) (2024)
- [DeepGD: A Multi-Objective Black-Box Test Selection Approach for Deep Neural Networks](https://mlsanity.com/paper/2303.04878/) (2023)
- [Hierarchical Distribution-Aware Testing of Deep Learning](https://mlsanity.com/paper/2205.08589/) (2022)
- [Generating and Detecting True Ambiguity: A Forgotten Danger in DNN Supervision Testing](https://mlsanity.com/paper/2207.10495/) (2022)
- [Evaluating the Robustness of Test Selection Methods for Deep Neural Networks](https://mlsanity.com/paper/2308.01314/) (2023)
- [Benchmarking Image Perturbations for Testing Automated Driving Assistance Systems](https://mlsanity.com/paper/2501.12269/) (2025)
- [To Softmax, or not to Softmax: that is the question when applying Active Learning for Transformer Models](https://mlsanity.com/paper/2210.03005/) (2022)
- [Neuron Sensitivity Guided Test Case Selection for Deep Learning Testing](https://mlsanity.com/paper/2307.11011/) (2023)
- [FAST: Boosting Uncertainty-based Test Prioritization Methods for Neural Networks via Feature Selection](https://mlsanity.com/paper/2409.09130/) (2024)
- [MetaSel: A Test Selection Approach for Fine-tuned DNN Models](https://mlsanity.com/paper/2503.17534/) (2025)
- [Contrastive inverse regression for dimension reduction](https://mlsanity.com/paper/2305.12287/) (2023)
- [Guiding the retraining of convolutional neural networks against adversarial inputs](https://mlsanity.com/paper/2207.03689/) (2022)
- [CheapET-3: Cost-Efficient Use of Remote DNN Models](https://mlsanity.com/paper/2208.11552/) (2022)
- [VisAlign: Dataset for Measuring the Degree of Alignment between AI and Humans in Visual Perception](https://mlsanity.com/paper/2308.01525/) (2023)
- [A Comprehensive Assessment Benchmark for Rigorously Evaluating Deep Learning Image Classifiers](https://mlsanity.com/paper/2308.04137/) (2023)
- [Inferring Data Preconditions from Deep Learning Models for Trustworthy Prediction in Deployment](https://mlsanity.com/paper/2401.14628/) (2024)
- [Evaluation and Improvement of Fault Detection for Large Language Models](https://mlsanity.com/paper/2404.14419/) (2024)
- [Toward Efficient Testing of Graph Neural Networks via Test Input Prioritization](https://mlsanity.com/paper/2512.18228/) (2025)
- [A Highly Efficient Diversity-based Input Selection for DNN Improvement Using VLMs](https://mlsanity.com/paper/2601.08024/) (2026)
- [TestifAI: Tomography-Based Testing for Deep Learning Systems](https://mlsanity.com/paper/2608.18900/) (2026)
- [Neuroevolutionary algorithms driven by neuron coverage metrics for semi-supervised classification](https://mlsanity.com/paper/2303.02801/) (2023)
- [Adopting Two Supervisors for Efficient Use of Large-Scale Remote Deep Neural Networks](https://mlsanity.com/paper/2304.02654/) (2023)
- [Hazards in Deep Learning Testing: Prevalence, Impact and Recommendations](https://mlsanity.com/paper/2309.05381/) (2023)
- [Learning-Based Testing for Deep Learning: Enhancing Model Robustness with Adversarial Input Prioritization](https://mlsanity.com/paper/2509.23961/) (2025)
- [CAFD: Concept-Aware DNN Fault Detection using VLMs](https://mlsanity.com/paper/2605.24008/) (2026)
- [Test Case Prioritization for DNNs via Neural Collapse Instability](https://mlsanity.com/paper/2607.20046/) (2026)

## Abstract (arXiv)

Test Input Prioritizers (TIP) for Deep Neural Networks (DNN) are an important technique to handle the typically very large test datasets efficiently, saving computation and labeling costs. This is particularly true for large-scale, deployed systems, where inputs observed in production are recorded to serve as potential test or training data for the next versions of the system. Feng et. al. propose DeepGini, a very fast and simple TIP, and show that it outperforms more elaborate techniques such as neuron- and surprise coverage. In a large-scale study (4 case studies, 8 test datasets, 32'200 trained models) we verify their findings. However, we also find that other comparable or even simpler baselines from the field of uncertainty quantification, such as the predicted softmax likelihood or the entropy of the predicted softmax likelihoods perform equally well as DeepGini.