# An Empirical Study of Fault Localisation Techniques for Deep Learning

arXiv: https://arxiv.org/abs/2412.11304 · HTML: https://mlsanity.com/paper/2412.11304/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 28 references that are in the map.
- First posted on arXiv 15 Dec 2024.

## Builds on (most cited first, 11 of 11)

- [BOHB: Robust and Efficient Hyperparameter Optimization at Scale](https://mlsanity.com/paper/1807.01774/) (2018)
- [Taxonomy of Real Faults in Deep Learning Systems](https://mlsanity.com/paper/1910.11015/) (2019)
- [HEBO Pushing The Limits of Sample-Efficient Hyperparameter Optimisation](https://mlsanity.com/paper/2012.03826/) (2020)
- [Causality-based Neural Network Repair](https://mlsanity.com/paper/2204.09274/) (2022)
- [Provable Repair of Deep Neural Networks](https://mlsanity.com/paper/2104.04413/) (2021)
- [DeepLocalize: Fault Localization for Deep Neural Networks](https://mlsanity.com/paper/2103.03376/) (2021)
- [Arachne: Search Based Repair of Deep Neural Networks](https://mlsanity.com/paper/1912.12463/) (2019)
- [DeepDiagnosis: Automatically Diagnosing Faults and Recommending Actionable Fixes in Deep Learning Programs](https://mlsanity.com/paper/2112.04036/) (2021)
- [DeepFD: Automated Fault Diagnosis and Localization for Deep Learning Programs](https://mlsanity.com/paper/2205.01938/) (2022)
- [Automatic Fault Detection for Deep Learning Programs Using Graph Transformations](https://mlsanity.com/paper/2105.08095/) (2021)
- [Repairing DNN Architecture: Are We There Yet?](https://mlsanity.com/paper/2301.11568/) (2023)

## Abstract (arXiv)

With the increased popularity of Deep Neural Networks (DNNs), increases also the need for tools to assist developers in the DNN implementation, testing and debugging process. Several approaches have been proposed that automatically analyse and localise potential faults in DNNs under test. In this work, we evaluate and compare existing state-of-the-art fault localisation techniques, which operate based on both dynamic and static analysis of the DNN. The evaluation is performed on a benchmark consisting of both real faults obtained from bug reporting platforms and faulty models produced by a mutation tool. Our findings indicate that the usage of a single, specific ground truth (e.g., the human defined one) for the evaluation of DNN fault localisation tools results in pretty low performance (maximum average recall of 0.31 and precision of 0.23). However, such figures increase when considering alternative, equivalent patches that exist for a given faulty DNN. Results indicate that dfd is the most effective tool, achieving an average recall of 0.61 and precision of 0.41 on our benchmark.