# There is Limited Correlation between Coverage and Robustness for Deep Neural Networks

arXiv: https://arxiv.org/abs/1911.05904 · HTML: https://mlsanity.com/paper/1911.05904/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Nov 2019.

## Built on by (most cited first, 8 of 8)

- [Black-Box Testing of Deep Neural Networks Through Test Case Diversity](https://mlsanity.com/paper/2112.12591/) (2021)
- [NPC: Neuron Path Coverage via Characterizing Decision Logic of Deep Neural Networks](https://mlsanity.com/paper/2203.12915/) (2022)
- [A Review and Refinement of Surprise Adequacy](https://mlsanity.com/paper/2103.05939/) (2021)
- [A Survey of Neural Network Robustness Assessment in Image Recognition](https://mlsanity.com/paper/2404.08285/) (2024)
- [Statistical Challenges with Dataset Construction: Why You Will Never Have Enough Images](https://mlsanity.com/paper/2408.11160/) (2024)
- [Is Neuron Coverage Needed to Make Person Detection More Robust?](https://mlsanity.com/paper/2204.10027/) (2022)
- [Excitement Surfeited Turns to Errors: Deep Learning Testing Framework Based on Excitable Neurons](https://mlsanity.com/paper/2202.07464/) (2022)
- [VLAMotor: Test-Guided Enhancement of Vision-Language-Action Models via Agent-BasedData Synthesis](https://mlsanity.com/paper/2606.00053/) (2026)

## Abstract (arXiv)

Deep neural networks (DNN) are increasingly applied in safety-critical systems, e.g., for face recognition, autonomous car control and malware detection. It is also shown that DNNs are subject to attacks such as adversarial perturbation and thus must be properly tested. Many coverage criteria for DNN since have been proposed, inspired by the success of code coverage criteria for software programs. The expectation is that if a DNN is a well tested (and retrained) according to such coverage criteria, it is more likely to be robust. In this work, we conduct an empirical study to evaluate the relationship between coverage, robustness and attack/defense metrics for DNN. Our study is the largest to date and systematically done based on 100 DNN models and 25 metrics. One of our findings is that there is limited correlation between coverage and robustness, i.e., improving coverage does not help improve the robustness. Our dataset and implementation have been made available to serve as a benchmark for future studies on testing DNN.