COMET: Coverage-guided Model Generation For Deep Learning Library Testing
Not on the map. Why?
Abstract (arXiv)
Recent deep learning (DL) applications are mostly built on top of DL libraries. The quality assurance of these libraries is critical to the dependable deployment of DL applications. Techniques have been proposed to generate various DL models and apply them to test these libraries. However, their test effectiveness is constrained by the diversity of layer API calls in their generated DL models. Our study reveals that these techniques can cover at most 34.1% layer inputs, 25.9% layer parameter values, and 15.6% layer sequences. As a result, we find that many bugs arising from specific layer API calls (i.e., specific layer inputs, parameter values, or layer sequences) can be missed by existing techniques. Because of this limitation, we propose COMET to effectively generate DL models with diverse layer API calls for DL library testing. COMET: (1) designs a set of mutation operators and a coverage-based search algorithm to diversify layer inputs, layer parameter values, and layer sequences in DL models. (2) proposes a model synthesis method to boost the test efficiency without compromising the layer API call diversity. Our evaluation result shows that COMET outperforms baselines by covering twice as many layer inputs (69.7% vs. 34.1%), layer parameter values (50.2% vs. 25.9%), and layer sequences (39.0% vs. 15.6%) as those by the state-of-the-art. Moreover, COMET covers 3.4% more library branches than those by existing techniques. Finally, COMET detects 32 new bugs in the latest version of eight popular DL libraries, including TensorFlow and MXNet, with 21 of them confirmed by DL library developers and 7 of those confirmed bugs have been fixed by developers.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 20 of its 61 references that are in the map. First posted on arXiv 2 Aug 2022.
Builds on
20 of 61 references.
References · 20
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2014Going Deeper with Convolutionscited by 4,231
- 2016Identity Mappings in Deep Residual Networkscited by 1,960
- 2017A Survey on Deep Learning in Medical Image Analysiscited by 1,182
- 2017DeepXplore: Automated Whitebox Testing of Deep Learning Systemscited by 307
- 2016Safe, Multi-Agent, Reinforcement Learning for Autonomous Drivingcited by 301
- 2015DeepDriving: Learning Affordance for Direct Perception in Autonomous Drivingcited by 229
- 2017Deep Reinforcement Learning framework for Autonomous Drivingcited by 224
- 2015MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systemscited by 195
- 2018TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzingcited by 67
- 2018DeepMutation: Mutation Testing of Deep Learning Systemscited by 66
- 2018Adversarial Sample Detection for Deep Neural Network through Model Mutation Testingcited by 48
8 more
- 2021DeepLocalize: Fault Localization for Deep Neural Networkscited by 37
- 2022Free Lunch for Testing: Fuzzing Deep-Learning Libraries from Open Sourcecited by 27
- 2022Toward Understanding Deep Learning Framework Bugscited by 26
- 2021Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlowcited by 25
- 2021DocTer: Documentation Guided Fuzzing for Testing Deep Learning API Functionscited by 21
- 2022Muffin: Testing Deep Learning Libraries via Neural Architecture Fuzzingcited by 15
- 2020Graph-Based Fuzz Testing for Deep Learning Inference Enginecited by 7
- 2018Deep Learning for Signal Authentication and Security in Massive Internet of Things Systemscited by 6
Built on by
Citing papers per year: 2024: 2; 2025: 1.
3 papers.