Testing Deep Learning Models: A First Comparative Study of Multiple Testing Techniques
Not on the map. Why?
Abstract (arXiv)
Deep Learning (DL) has revolutionized the capabilities of vision-based systems (VBS) in critical applications such as autonomous driving, robotic surgery, critical infrastructure surveillance, air and maritime traffic control, etc. By analyzing images, voice, videos, or any type of complex signals, DL has considerably increased the situation awareness of these systems. At the same time, while relying more and more on trained DL models, the reliability and robustness of VBS have been challenged and it has become crucial to test thoroughly these models to assess their capabilities and potential errors. To discover faults in DL models, existing software testing methods have been adapted and refined accordingly. In this article, we provide an overview of these software testing methods, namely differential, metamorphic, mutation, and combinatorial testing, as well as adversarial perturbation testing and review some challenges in their deployment for boosting perception systems used in VBS. We also provide a first experimental comparative study on a classical benchmark used in VBS and discuss its results.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 17 of its 35 references that are in the map. First posted on arXiv 24 Feb 2022.
Builds on
17 of 35 references.
References · 17
- 2013Intriguing properties of neural networkscited by 5,155
- 2015The Limitations of Deep Learning in Adversarial Settingscited by 790
- 2014Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Imagescited by 776
- 2017DeepXplore: Automated Whitebox Testing of Deep Learning Systemscited by 311
- 2017DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Carscited by 251
- 2019Machine Learning Testing: Survey, Landscapes and Horizonscited by 218
- 2018DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systemscited by 147
- 2016Technical Report on the CleverHans v2.1.0 Adversarial Examples Librarycited by 124
- 2018Guiding Deep Learning System Testing using Surprise Adequacycited by 124
- 2018DeepMutation: Mutation Testing of Deep Learning Systemscited by 67
- 2018The Art, Science, and Engineering of Fuzzing: A Surveycited by 62
- 2018On Testing Machine Learning Programscited by 48
5 more
- 2018Adversarial Sample Detection for Deep Neural Network through Model Mutation Testingcited by 48
- 2018Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testingcited by 31
- 2020Deep Learning-Based Anomaly Detection in Cyber-Physical Systems: Progress and Opportunitiescited by 27
- 2019Adaptive Metamorphic Testing with Contextual Banditscited by 7
- 2019Towards Testing of Deep Learning Systems with Training Set Reductioncited by 3
Built on by
Citing papers per year: 2022: 1; 2023: 1.
2 papers.