# Quality-Driven Selective Mutation for Deep Learning

arXiv: https://arxiv.org/abs/2604.22640 · HTML: https://mlsanity.com/paper/2604.22640/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 42 references that are in the map.
- First posted on arXiv 24 Apr 2026.

## Builds on (most cited first, 13 of 13)

- [DeepMutation: Mutation Testing of Deep Learning Systems](https://mlsanity.com/paper/1805.05206/) (2018)
- [Adversarial Sample Detection for Deep Neural Network through Model Mutation Testing](https://mlsanity.com/paper/1812.05793/) (2018)
- [DeepLocalize: Fault Localization for Deep Neural Networks](https://mlsanity.com/paper/2103.03376/) (2021)
- [DeepFD: Automated Fault Diagnosis and Localization for Deep Learning Programs](https://mlsanity.com/paper/2205.01938/) (2022)
- [Bugs in Machine Learning-based Systems: A Faultload Benchmark](https://mlsanity.com/paper/2206.12311/) (2022)
- [A Search-Based Testing Framework for Deep Neural Networks of Source Code Embedding](https://mlsanity.com/paper/2101.07910/) (2021)
- [DeepMetis: Augmenting a Deep Learning Test Set to Increase its Mutation Score](https://mlsanity.com/paper/2109.07514/) (2021)
- [REIN: A Comprehensive Benchmark Framework for Data Cleaning Methods in ML Pipelines](https://mlsanity.com/paper/2302.04702/) (2023)
- [Tailored Mutants Fit Bugs Better](https://mlsanity.com/paper/1611.02516/) (2016)
- [Practical Mutation Testing at Scale](https://mlsanity.com/paper/2102.11378/) (2021)
- [MuFF: Stable and Sensitive Post-training Mutation Testing for Deep Learning](https://mlsanity.com/paper/2501.09846/) (2025)
- [An Empirical Study of the Realism of Mutants in Deep Learning](https://mlsanity.com/paper/2512.16741/) (2025)
- [On Accelerating Deep Neural Network Mutation Analysis by Neuron and Mutant Clustering](https://mlsanity.com/paper/2501.12598/) (2025)

## Abstract (arXiv)

Mutants support testing and debugging in two roles: (i) as test goals and (ii) as substitutes for real faults. Hard-to-kill mutants provide better guidance for test improvement, while realism is essential when mutants are used to simulate real bugs. Building on these roles, selective mutation for deep learning (DL) aims to reduce the cost of mutant generation and execution by choosing operator configurations that yield resistant and realistic mutants. However, the DL literature lacks a unified measure that captures both aspects. This study presents a probabilistic framework to quantify mutant quality along two complementary axes: resistance and realism. Resistance adapts the classical notion of hard-to-kill mutants to the DL setting using statistical killing probabilities, while realism is measured via the generalized Jaccard similarity between mutant and real-fault detectability patterns. The framework enables ranking and filtering of low-quality mutation-operator configurations without assuming a specific use case. We empirically evaluate the approach on four datasets of real DL faults. Three datasets (CleanML, DeepFD, and DeepLocalize) are used to estimate and select high-quality operator configurations, and the held-out defect4ML dataset is used for validation. Results show that quality-driven selection reduces the number of generated mutants by up to 55.6% while preserving typical levels of resistance and realism under baseline-aligned selection thresholds. These findings confirm that dual-objective selection can lower cost without compromising the usefulness of mutants for either role.