Testing Feedforward Neural Networks Training Programs
Not on the map. Why?
Abstract (arXiv)
Nowadays, we are witnessing an increasing effort to improve the performance and trustworthiness of Deep Neural Networks (DNNs), with the aim to enable their adoption in safety critical systems such as self-driving cars. Multiple testing techniques are proposed to generate test cases that can expose inconsistencies in the behavior of DNN models. These techniques assume implicitly that the training program is bug-free and appropriately configured. However, satisfying this assumption for a novel problem requires significant engineering work to prepare the data, design the DNN, implement the training program, and tune the hyperparameters in order to produce the model for which current automated test data generators search for corner-case behaviors. All these model training steps can be error-prone. Therefore, it is crucial to detect and correct errors throughout all the engineering steps of DNN-based software systems and not only on the resulting DNN model. In this paper, we gather a catalog of training issues and based on their symptoms and their effects on the behavior of the training program, we propose practical verification routines to detect the aforementioned issues, automatically, by continuously validating that some important properties of the learning dynamics hold during the training. Then, we design, TheDeepChecker, an end-to-end property-based debugging approach for DNN training programs. We assess the effectiveness of TheDeepChecker on synthetic and real-world buggy DL programs and compare it with Amazon SageMaker Debugger (SMD). Results show that TheDeepChecker's on-execution validation of DNN-based program's properties succeeds in revealing several coding bugs and system misconfigurations, early on and at a low cost. Moreover, TheDeepChecker outperforms the SMD's offline rules verification on training logs in terms of detection accuracy and DL bugs coverage.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 44 of its 89 references that are in the map. First posted on arXiv 1 Apr 2022.
Builds on
44 of 89 references.
References · 44
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
- 2014Going Deeper with Convolutionscited by 4,231
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
- 2019Similarity of Neural Network Representations Revisitedcited by 1,695
- 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
- 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 660
- 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
- 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 488
- 2017Towards Principled Methods for Training Generative Adversarial Networkscited by 460
- 2017The Effectiveness of Data Augmentation in Image Classification using Deep Learningcited by 368
32 more
- 2018How Does Batch Normalization Help Optimization?cited by 355
- 2017DeepXplore: Automated Whitebox Testing of Deep Learning Systemscited by 307
- 2017DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Carscited by 249
- 2015Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inferencecited by 159
- 2018Understanding Batch Normalizationcited by 157
- 2018DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systemscited by 144
- 2017Stabilizing Training of Generative Adversarial Networks through Regularizationcited by 118
- 2018Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontierscited by 117
- 2016An Analysis of Deep Neural Network Models for Practical Applicationscited by 114
- 2019Taxonomy of Real Faults in Deep Learning Systemscited by 90
- 2018TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzingcited by 67
- 2018DeepMutation: Mutation Testing of Deep Learning Systemscited by 66
- 2019A Comprehensive Study on Deep Learning Bug Characteristicscited by 65
- 2018DLFuzz: Differential Fuzzing Testing of Deep Learning Systemscited by 64
- 2016Towards Better Analysis of Deep Convolutional Neural Networkscited by 51
- 2018Manifold: A Model-Agnostic Framework for Interpretation and Diagnosis of Machine Learning Modelscited by 50
- 2018On Testing Machine Learning Programscited by 48
- 2018Collapse of Deep and Narrow Neural Netscited by 37
- 2018Understanding the Disharmony between Dropout and Batch Normalization by Variance Shiftcited by 36
- 2020Repairing Deep Neural Networks: Fix Patterns and Challengescited by 33
- 2016Survey of resampling techniques for improving classification performance in unbalanced datasetscited by 32
- 2020Analyzing the Noise Robustness of Deep Neural Networkscited by 31
- 2018Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testingcited by 30
- 2017Towards Proving the Adversarial Robustness of Deep Neural Networkscited by 26
- 2018Security and Privacy Issues in Deep Learningcited by 19
- 2019DeepEvolution: A Search-Based Testing Approach for Deep Neural Networkscited by 15
- 2017Developing Bug-Free Machine Learning Systems With Formal Mathematicscited by 13
- 2018Towards Principled Design of Deep Convolutional Networks: Introducing SimpNetcited by 11
- 2018Detecting Dead Weights and Units in Neural Networkscited by 7
- 2016Deep Convolutional Neural Network Design Patternscited by 5
- 2018DeepHunter: Hunting Deep Neural Network Defects via Coverage-Guided Fuzzingcited by 4
- 2018Gradient-Coherent Strong Regularization for Deep Neural Networkscited by 1
Built on by
Citing papers per year: 2024: 1; 2025: 0; 2026: 1.
2 papers.