mlsanityData to 7 Oct 2026

Beyond Accuracy: An Empirical Study on Unit Testing in Open-source Deep Learning Projects

arXiv 2402.16546 ↗First posted 26 Feb 2024cs.SE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Deep Learning (DL) models have rapidly advanced, focusing on achieving high performance through testing model accuracy and robustness. However, it is unclear whether DL projects, as software systems, are tested thoroughly or functionally correct when there is a need to treat and test them like other software systems. Therefore, we empirically study the unit tests in open-source DL projects, analyzing 9,129 projects from GitHub. We find that: 1) unit tested DL projects have positive correlation with the open-source project metrics and have a higher acceptance rate of pull requests, 2) 68% of the sampled DL projects are not unit tested at all, 3) the layer and utilities (utils) of DL models have the most unit tests. Based on these findings and previous research outcomes, we built a mapping taxonomy between unit tests and faults in DL projects. We discuss the implications of our findings for developers and researchers and highlight the need for unit testing in open-source DL projects to ensure their reliability and stability. The study contributes to this community by raising awareness of the importance of unit testing in DL projects and encouraging further research in this area.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 23 of its 83 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

23 of 83 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  3. 2017DeepXplore: Automated Whitebox Testing of Deep Learning Systemscited by 307
  4. 2017DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Carscited by 249
  5. 2019Machine Learning Testing: Survey, Landscapes and Horizonscited by 213
  6. 2018DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systemscited by 144
  7. 2018Activation Functions: Comparison of trends in Practice and Research for Deep Learningcited by 130
  8. 2019On Empirical Comparisons of Optimizers for Deep Learningcited by 99
  9. 2019Taxonomy of Real Faults in Deep Learning Systemscited by 90
  10. 2019A Comprehensive Study on Deep Learning Bug Characteristicscited by 65
  11. 2018DLFuzz: Differential Fuzzing Testing of Deep Learning Systemscited by 64
  12. 2018Software Engineering Challenges of Deep Learningcited by 41
11 more
  1. 2020Model-based Exploration of the Frontier of Behaviours for Deep Learning System Testingcited by 40
  2. 2018Testing Deep Neural Networkscited by 36
  3. 2020Repairing Deep Neural Networks: Fix Patterns and Challengescited by 33
  4. 2016Understanding the Factors that Impact the Popularity of GitHub Repositoriescited by 27
  5. 2020Automated Unit Test Generation for Pythoncited by 18
  6. 2020DeText: A Deep Text Ranking Framework with BERTcited by 13
  7. 2023What Do Users Ask in Open-Source AI Repositories? An Empirical Study of GitHub Issuescited by 8
  8. 2019Towards Improved Testing For Deep Learningcited by 6
  9. 2019Software Engineering Practice in the Development of Deep Learning Applicationscited by 5
  10. 2021PyNose: A Test Smell Detector For Pythoncited by 4
  11. 2022Exploring ML testing in practice -- Lessons learned from an interactive rapid review with Axis Communicationscited by 3

Built on by

Not cited yet.