# A Probabilistic Framework for Mutation Testing in Deep Neural Networks

arXiv: https://arxiv.org/abs/2208.06018 · HTML: https://mlsanity.com/paper/2208.06018/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 44 references that are in the map.
- First posted on arXiv 11 Aug 2022.

## Builds on (most cited first, 5 of 5)

- [DeepMutation: Mutation Testing of Deep Learning Systems](https://mlsanity.com/paper/1805.05206/) (2018)
- [How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review](https://mlsanity.com/paper/2107.12045/) (2021)
- [Using Machine Learning Safely in Automotive Software: An Assessment and Adaption of Software Process Requirements in ISO 26262](https://mlsanity.com/paper/1808.01614/) (2018)
- [What Are We Really Testing in Mutation Testing for Machine Learning? A Critical Reflection](https://mlsanity.com/paper/2103.01341/) (2021)
- [Discussion of Big Bayes Stories and BayesBag](https://mlsanity.com/paper/1405.4977/) (2014)

## Built on by (most cited first, 3 of 3)

- [Mutation Testing of Deep Reinforcement Learning Based on Real Faults](https://mlsanity.com/paper/2301.05651/) (2023)
- [Machine Learning Data Suitability and Performance Testing Using Fault Injection Testing Framework](https://mlsanity.com/paper/2309.11274/) (2023)
- [DeepCover: Advancing RNN Test Coverage and Online Error Prediction using State Machine Extraction](https://mlsanity.com/paper/2402.06966/) (2024)

## Abstract (arXiv)

Context: Mutation Testing (MT) is an important tool in traditional Software Engineering (SE) white-box testing. It aims to artificially inject faults in a system to evaluate a test suite's capability to detect them, assuming that the test suite defects finding capability will then translate to real faults. If MT has long been used in SE, it is only recently that it started gaining the attention of the Deep Learning (DL) community, with researchers adapting it to improve the testability of DL models and improve the trustworthiness of DL systems. Objective: If several techniques have been proposed for MT, most of them neglected the stochasticity inherent to DL resulting from the training phase. Even the latest MT approaches in DL, which propose to tackle MT through a statistical approach, might give inconsistent results. Indeed, as their statistic is based on a fixed set of sampled training instances, it can lead to different results across instances set when results should be consistent for any instance. Methods: In this work, we propose a Probabilistic Mutation Testing (PMT) approach that alleviates the inconsistency problem and allows for a more consistent decision on whether a mutant is killed or not. Results: We show that PMT effectively allows a more consistent and informed decision on mutations through evaluation using three models and eight mutation operators used in previously proposed MT methods. We also analyze the trade-off between the approximation error and the cost of our method, showing that relatively small error can be achieved for a manageable cost. Conclusion: Our results showed the limitation of current MT practices in DNN and the need to rethink them. We believe PMT is the first step in that direction which effectively removes the lack of consistency across test executions of previous methods caused by the stochasticity of DNN training.