mlsanityData to 7 Oct 2026

Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlow

arXiv 2112.13314 ↗First posted 26 Dec 2021cs.SE cs.LGcited by 25

Not on the map. Why?

Abstract (arXiv)

Deep Learning (DL) frameworks are now widely used, simplifying the creation of complex models as well as their integration to various applications even to non DL experts. However, like any other programs, they are prone to bugs. This paper deals with the subcategory of bugs named silent bugs: they lead to wrong behavior but they do not cause system crashes or hangs, nor show an error message to the user. Such bugs are even more dangerous in DL applications and frameworks due to the "black-box" and stochastic nature of the systems (the end user can not understand how the model makes decisions). This paper presents the first empirical study of Keras and TensorFlow silent bugs, and their impact on users' programs. We extracted closed issues related to Keras from the TensorFlow GitHub repository. Out of the 1,168 issues that we gathered, 77 were reproducible silent bugs affecting users' programs. We categorized the bugs based on the effects on the users' programs and the components where the issues occurred, using information from the issue reports. We then derived a threat level for each of the issues, based on the impact they had on the users' programs. To assess the relevance of identified categories and the impact scale, we conducted an online survey with 103 DL developers. The participants generally agreed with the significant impact of silent bugs in DL libraries and acknowledged our findings (i.e., categories of silent bugs and the proposed impact scale). Finally, leveraging our analysis, we provide a set of guidelines to facilitate safeguarding against such bugs in DL frameworks.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 4 of its 62 references that are in the map. First posted on arXiv 26 Dec 2021.

Builds on

4 of 62 references.

References · 4

  1. 2019Machine Learning Testing: Survey, Landscapes and Horizonscited by 213
  2. 2019Taxonomy of Real Faults in Deep Learning Systemscited by 90
  3. 2019A Comprehensive Study on Deep Learning Bug Characteristicscited by 65
  4. 2021Faults in Deep Reinforcement Learning Programs: A Taxonomy and A Detection Approachcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 4; 2024: 8; 2025: 5; 2026: 4.

25 papers.

Sort:

By area:

  1. 2024Bugs in Large Language Models Generated Code: An Empirical Studycited by 59
  2. 2024Impacts of floating-point non-associativity on reproducibility for HPC and deep learning applicationscited by 24
  3. 2022Bugs in Machine Learning-based Systems: A Faultload Benchmarkcited by 15
  4. 2024LLM-Powered Test Case Generation for Detecting Bugs in Plausible Programscited by 14
  5. 2023Bug Characterization in Machine Learning-based Systemscited by 10
  6. 2024A Survey on Failure Analysis and Fault Injection in AI Systemscited by 7
  7. 2024Towards Enhancing the Reproducibility of Deep Learning Bugs: An Empirical Studycited by 6
  8. 2023To Build Our Future, We Must Know Our Past: Contextualizing Paradigm Shifts in Natural Language Processingcited by 4
  9. 2024Deep Learning Library Testing: Definition, Methods and Challengescited by 4
  10. 2024Enhancing Differential Testing With LLMs For Testing Deep Learning Librariescited by 4
  11. 2022COMET: Coverage-guided Model Generation For Deep Learning Library Testingcited by 3
  12. 2023Reliability Assurance for Deep Neural Network Architectures Against Numerical Defectscited by 3
13 more
  1. 2022Challenges in Migrating Imperative Deep Learning Programs to Graph Execution: An Empirical Studycited by 2
  2. 2024Unraveling Code Clone Dynamics in Deep Learning Frameworkscited by 2
  3. 2023An Empirical Study on Bugs Inside PyTorch: A Replication Studycited by 1
  4. 2025PyResBugs: A Dataset of Residual Python Bugs for Natural Language-Driven Fault Injectioncited by 1
  5. 2025TTrace: Lightweight Error Checking and Diagnosis for Distributed Trainingcited by 1
  6. 2025Why Attention Fails: A Taxonomy of Faults in Attention-Based Neural Networkscited by 1
  7. 2025Imitation Game: Reproducing Deep Learning Bugs Leveraging an Intelligent Agentcited by 1
  8. 2026Understanding Agent-Reactive Bugs at the Model-Harness Boundary: An Empirical Study of LLM Agent Issue Reportscited by 1
  9. 2022Comparative analysis of real bugs in open-source Machine Learning projects -- A Registered Reportcited by 0
  10. 2025Verifying Computational Graphs in Production-Grade Distributed Machine Learning Frameworkscited by 0
  11. 2026Demystifying the Silence of Correctness Bugs in PyTorch Compilercited by 0
  12. 2026Relaxed activation analysis of dataflow networks - A clock calculus for machine learning and real-time schedulingcited by 0
  13. 2026Silent Failures Beyond the 32-Bit Index Range: A Differential Characterization of Large-Tensor Matrix Multiplication in PyTorch's MPS Backendcited by 0