mlsanityData to 14 Oct 2026

Understanding Performance Problems in Deep Learning Systems

arXiv 2112.01771 ↗First posted 3 Dec 2021cs.SE cs.AI cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Deep learning (DL) has been widely applied to many domains. Unique challenges in engineering DL systems are posed by the programming paradigm shift from traditional systems to DL systems, and performance is one of the challenges. Performance problems (PPs) in DL systems can cause severe consequences such as excessive resource consumption and financial loss. While bugs in DL systems have been extensively investigated, PPs in DL systems have hardly been explored. To bridge this gap, we present the first comprehensive study to i) characterize symptoms, root causes, and introducing and exposing stages of PPs in DL systems developed in TensorFLow and Keras, with 224 PPs collected from 210 StackOverflow posts, and to ii) assess the capability of existing performance analysis approaches in tackling PPs, with a constructed benchmark of 58 PPs in DL systems. Our findings shed light on the implications on developing high-performance DL systems, and detecting and localizing PPs in DL systems. To demonstrate the usefulness of our findings, we develop a static checker Deep-Perf to detect three types of PPs. It has detected 488 new PPs in 130 GitHub projects. 105 and 27 PPs have been confirmed and fixed.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 22 of its 90 references that are in the map. First posted on arXiv 3 Dec 2021.

Builds on

22 of 90 references.

References · 22

  1. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  2. 2017DeepXplore: Automated Whitebox Testing of Deep Learning Systemscited by 311
  3. 2017DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Carscited by 251
  4. 2019Machine Learning Testing: Survey, Landscapes and Horizonscited by 218
  5. 2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 154
  6. 2018DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systemscited by 147
  7. 2018Guiding Deep Learning System Testing using Surprise Adequacycited by 124
  8. 2019Taxonomy of Real Faults in Deep Learning Systemscited by 90
  9. 2018TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzingcited by 68
  10. 2019A Comprehensive Study on Deep Learning Bug Characteristicscited by 65
  11. 2018SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networkscited by 55
  12. 2018Concolic Testing for Deep Neural Networkscited by 51
10 more
  1. 2020The Deep Learning Compiler: A Comprehensive Surveycited by 46
  2. 2021DeepLocalize: Fault Localization for Deep Neural Networkscited by 38
  3. 2019An Empirical Study towards Characterizing Deep Learning Development and Deployment across Different Frameworks and Platformscited by 35
  4. 2020Repairing Deep Neural Networks: Fix Patterns and Challengescited by 33
  5. 2021An Empirical Study on Deployment Faults of Deep Learning Based Mobile Applicationscited by 20
  6. 2019Moving Deep Learning into Web Browser: How Far Can We Go?cited by 14
  7. 2020Detecting and Understanding Real-World Differential Performance Bugs in Machine Learning Librariescited by 7
  8. 2017SlowFuzz: Automated Domain-Independent Detection of Algorithmic Complexity Vulnerabilitiescited by 6
  9. 2020Interpreting Cloud Computer Vision Pain-Points: A Mining Study of Stack Overflowcited by 6
  10. 2020ShapeFlow: Dynamic Shape Interpreter for TensorFlowcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 6; 2025: 2; 2026: 1.

12 papers.

  1. 2023Bug Characterization in Machine Learning-based Systemscited by 10
  2. 2024A Survey on Failure Analysis and Fault Injection in AI Systemscited by 10
  3. 2024Understanding the Fundamental Design Decisions of Retrieval-Augmented Generation Systemscited by 7
  4. 2024Anatomizing Deep Learning Inference in Web Browserscited by 6
  5. 2026Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agentscited by 6
  6. 2024Deep Learning Library Testing: Definition, Methods and Challengescited by 5
  7. 2024The Impact of Hyperparameters on Large Language Model Inference Performance: An Evaluation of vLLM and HuggingFace Pipelinescited by 5
  8. 2023CaRE: Finding Root Causes of Configuration Issues in Highly-Configurable Robotscited by 4
  9. 2024Efficient Surgical Tool Recognition via HMM-Stabilized Deep Learningcited by 1
  10. 2025Why Attention Fails: A Taxonomy of Faults in Attention-Based Neural Networkscited by 1
  11. 2023Understanding Bugs in Multi-Language Deep Learning Frameworkscited by 0
  12. 2025Speculative Automated Refactoring of Imperative Deep Learning Programs to Graph Executioncited by 0