# Test Selection for Deep Learning Systems

arXiv: https://arxiv.org/abs/1904.13195 · HTML: https://mlsanity.com/paper/1904.13195/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Apr 2019.

## Built on by (most cited first, 27 of 27)

- [Simple Techniques Work Surprisingly Well for Neural Network Test Prioritization and Active Learning (Replicability Study)](https://mlsanity.com/paper/2205.00664/) (2022)
- [NPC: Neuron Path Coverage via Characterizing Decision Logic of Deep Neural Networks](https://mlsanity.com/paper/2203.12915/) (2022)
- [Evaluating Uncertainty and Quality of Vision-Language-Action-enabled Robots](https://mlsanity.com/paper/2507.17049/) (2025)
- [PatchCensor: Patch Robustness Certification for Transformers via Exhaustive Testing](https://mlsanity.com/paper/2111.10481/) (2021)
- [DeepGD: A Multi-Objective Black-Box Test Selection Approach for Deep Neural Networks](https://mlsanity.com/paper/2303.04878/) (2023)
- [A Review and Refinement of Surprise Adequacy](https://mlsanity.com/paper/2103.05939/) (2021)
- [Aries: Efficient Testing of Deep Neural Networks via Labeling-Free Accuracy Estimation](https://mlsanity.com/paper/2207.10942/) (2022)
- [Unveiling Code Pre-Trained Models: Investigating Syntax and Semantics Capacities](https://mlsanity.com/paper/2212.10017/) (2022)
- [CertPri: Certifiable Prioritization for Deep Neural Networks via Movement Cost in Feature Space](https://mlsanity.com/paper/2307.09375/) (2023)
- [Evaluating the Robustness of Test Selection Methods for Deep Neural Networks](https://mlsanity.com/paper/2308.01314/) (2023)
- [An Exploratory Study of AI System Risk Assessment from the Lens of Data Distribution and Uncertainty](https://mlsanity.com/paper/2212.06828/) (2022)
- [FAST: Boosting Uncertainty-based Test Prioritization Methods for Neural Networks via Feature Selection](https://mlsanity.com/paper/2409.09130/) (2024)
- [MetaSel: A Test Selection Approach for Fine-tuned DNN Models](https://mlsanity.com/paper/2503.17534/) (2025)
- [Robust Active Learning: Sample-Efficient Training of Robust Deep Learning Models](https://mlsanity.com/paper/2112.02542/) (2021)
- [Characterizing and Understanding the Behavior of Quantized Models for Reliable Deployment](https://mlsanity.com/paper/2204.04220/) (2022)
- [LaF: Labeling-Free Model Selection for Automated Deep Neural Network Reusing](https://mlsanity.com/paper/2204.03994/) (2022)
- [Guiding the retraining of convolutional neural networks against adversarial inputs](https://mlsanity.com/paper/2207.03689/) (2022)
- [GIST: Generated Inputs Sets Transferability in Deep Learning](https://mlsanity.com/paper/2311.00801/) (2023)
- [Evaluation and Improvement of Fault Detection for Large Language Models](https://mlsanity.com/paper/2404.14419/) (2024)
- [A3Rank: Augmentation Alignment Analysis for Prioritizing Overconfident Failing Samples for Deep Learning Models](https://mlsanity.com/paper/2407.14114/) (2024)
- [DONUT: Physics-aware Machine Learning for Real-time X-ray Nanodiffraction Analysis](https://mlsanity.com/paper/2507.14038/) (2025)
- [TestifAI: Tomography-Based Testing for Deep Learning Systems](https://mlsanity.com/paper/2608.18900/) (2026)
- [Hazards in Deep Learning Testing: Prevalence, Impact and Recommendations](https://mlsanity.com/paper/2309.05381/) (2023)
- [DeepKnowledge: Generalisation-Driven Deep Learning Testing](https://mlsanity.com/paper/2403.16768/) (2024)
- [LeCov: Multi-level Testing Criteria for Large Language Models](https://mlsanity.com/paper/2408.10474/) (2024)
- [TigAug: Data Augmentation for Testing Traffic Light Detection in Autonomous Driving Systems](https://mlsanity.com/paper/2507.05932/) (2025)
- [Learning-Based Testing for Deep Learning: Enhancing Model Robustness with Adversarial Input Prioritization](https://mlsanity.com/paper/2509.23961/) (2025)

## Abstract (arXiv)

Testing of deep learning models is challenging due to the excessive number and complexity of computations involved. As a result, test data selection is performed manually and in an ad hoc way. This raises the question of how we can automatically select candidate test data to test deep learning models. Recent research has focused on adapting test selection metrics from code-based software testing (such as coverage) to deep learning. However, deep learning models have different attributes from code such as spread of computations across the entire network reflecting training data properties, balance of neuron weights and redundancy (use of many more neurons than needed). Such differences make code-based metrics inappropriate to select data that can challenge the models (can trigger misclassification). We thus propose a set of test selection metrics based on the notion of model uncertainty (model confidence on specific inputs). Intuitively, the more uncertain we are about a candidate sample, the more likely it is that this sample triggers a misclassification. Similarly, the samples for which we are the most uncertain, are the most informative and should be used to improve the model by retraining. We evaluate these metrics on two widely-used image classification problems involving real and artificial (adversarial) data. We show that uncertainty-based metrics have a strong ability to select data that are misclassified and lead to major improvement in classification accuracy during retraining: up to 80% more gain than random selection and other state-of-the-art metrics on one dataset and up to 29% on the other.