# D-Score: A White-Box Diagnosis Score for CNNs Based on Mutation Operators

arXiv: https://arxiv.org/abs/2304.00697 · HTML: https://mlsanity.com/paper/2304.00697/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 40 references that are in the map.
- First posted on arXiv 3 Apr 2023.

## Builds on (most cited first, 14 of 14)

- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [An Introduction to Convolutional Neural Networks](https://mlsanity.com/paper/1511.08458/) (2015)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [Making Convolutional Networks Shift-Invariant Again](https://mlsanity.com/paper/1904.11486/) (2019)
- [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://mlsanity.com/paper/1705.06640/) (2017)
- [Generating Adversarial Examples with Adversarial Networks](https://mlsanity.com/paper/1801.02610/) (2018)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [Why do deep convolutional networks generalize so poorly to small image transformations?](https://mlsanity.com/paper/1805.12177/) (2018)
- [DeepMutation: Mutation Testing of Deep Learning Systems](https://mlsanity.com/paper/1805.05206/) (2018)
- [Adversarial Sample Detection for Deep Neural Network through Model Mutation Testing](https://mlsanity.com/paper/1812.05793/) (2018)
- [Testing Deep Neural Networks](https://mlsanity.com/paper/1803.04792/) (2018)
- [What Are We Really Testing in Mutation Testing for Machine Learning? A Critical Reflection](https://mlsanity.com/paper/2103.01341/) (2021)

## Built on by (most cited first, 1 of 1)

- [Revising the Problem of Partial Labels from the Perspective of CNNs' Robustness](https://mlsanity.com/paper/2407.17630/) (2024)

## Abstract (arXiv)

Convolutional neural networks (CNNs) have been widely applied in many safety-critical domains, such as autonomous driving and medical diagnosis. However, concerns have been raised with respect to the trustworthiness of these models: The standard testing method evaluates the performance of a model on a test set, while low-quality and insufficient test sets can lead to unreliable evaluation results, which can have unforeseeable consequences. Therefore, how to comprehensively evaluate CNNs and, based on the evaluation results, how to enhance their trustworthiness are the key problems to be urgently addressed. Prior work has used mutation tests to evaluate the test sets of CNNs. However, the evaluation scores are black boxes and not explicit enough for what is being tested. In this paper, we propose a white-box diagnostic approach that uses mutation operators and image transformation to calculate the feature and attention distribution of the model and further present a diagnosis score, namely D-Score, to reflect the model's robustness and fitness to a dataset. We also propose a D-Score based data augmentation method to enhance the CNN's performance to translations and rescalings. Comprehensive experiments on two widely used datasets and three commonly adopted CNNs demonstrate the effectiveness of our approach.