# Input Prioritization for Testing Neural Networks

arXiv: https://arxiv.org/abs/1901.03768 · HTML: https://mlsanity.com/paper/1901.03768/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jan 2019.

## Built on by (most cited first, 11 of 11)

- [Software Engineering for AI-Based Systems: A Survey](https://mlsanity.com/paper/2105.01984/) (2021)
- [DeepMetis: Augmenting a Deep Learning Test Set to Increase its Mutation Score](https://mlsanity.com/paper/2109.07514/) (2021)
- [A Review and Refinement of Surprise Adequacy](https://mlsanity.com/paper/2103.05939/) (2021)
- [TestRank: Bringing Order into Unlabeled Test Instances for Deep Learning Tasks](https://mlsanity.com/paper/2105.10113/) (2021)
- [CertPri: Certifiable Prioritization for Deep Neural Networks via Movement Cost in Feature Space](https://mlsanity.com/paper/2307.09375/) (2023)
- [Exploring ML testing in practice -- Lessons learned from an interactive rapid review with Axis Communications](https://mlsanity.com/paper/2203.16225/) (2022)
- [FAST: Boosting Uncertainty-based Test Prioritization Methods for Neural Networks via Feature Selection](https://mlsanity.com/paper/2409.09130/) (2024)
- [Uncertainty Measurement of Deep Learning System based on the Convex Hull of Training Sets](https://mlsanity.com/paper/2405.16082/) (2024)
- [Toward Efficient Testing of Graph Neural Networks via Test Input Prioritization](https://mlsanity.com/paper/2512.18228/) (2025)
- [Test Case Prioritization for DNNs via Neural Collapse Instability](https://mlsanity.com/paper/2607.20046/) (2026)
- [RiskBlend: A Multi-Signal Framework for Test Input Prioritization in Machine Learning Regression Testing](https://mlsanity.com/paper/2608.27704/) (2026)

## Abstract (arXiv)

Deep neural networks (DNNs) are increasingly being adopted for sensing and control functions in a variety of safety and mission-critical systems such as self-driving cars, autonomous air vehicles, medical diagnostics, and industrial robotics. Failures of such systems can lead to loss of life or property, which necessitates stringent verification and validation for providing high assurance. Though formal verification approaches are being investigated, testing remains the primary technique for assessing the dependability of such systems. Due to the nature of the tasks handled by DNNs, the cost of obtaining test oracle data---the expected output, a.k.a. label, for a given input---is high, which significantly impacts the amount and quality of testing that can be performed. Thus, prioritizing input data for testing DNNs in meaningful ways to reduce the cost of labeling can go a long way in increasing testing efficacy. This paper proposes using gauges of the DNN's sentiment derived from the computation performed by the model, as a means to identify inputs that are likely to reveal weaknesses. We empirically assessed the efficacy of three such sentiment measures for prioritization---confidence, uncertainty, and surprise---and compare their effectiveness in terms of their fault-revealing capability and retraining effectiveness. The results indicate that sentiment measures can effectively flag inputs that expose unacceptable DNN behavior. For MNIST models, the average percentage of inputs correctly flagged ranged from 88% to 94.8%.