mlsanityData to 7 Oct 2026

Deep Learning Library Testing: Definition, Methods and Challenges

arXiv 2404.17871 ↗First posted 27 Apr 2024cs.SE cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

In recent years, software systems powered by deep learning (DL) techniques have significantly facilitated people's lives in many aspects. As the backbone of these DL systems, various DL libraries undertake the underlying optimization and computation. However, like traditional software, DL libraries are not immune to bugs, which can pose serious threats to users' personal property and safety. Studying the characteristics of DL libraries, their associated bugs, and the corresponding testing methods is crucial for enhancing the security of DL systems and advancing the widespread application of DL technology. This paper provides an overview of the testing research related to various DL libraries, discusses the strengths and weaknesses of existing methods, and provides guidance and reference for the application of the DL library. This paper first introduces the workflow of DL underlying libraries and the characteristics of three kinds of DL libraries involved, namely DL framework, DL compiler, and DL hardware library. It then provides definitions for DL underlying library bugs and testing. Additionally, this paper summarizes the existing testing methods and tools tailored to these DL libraries separately and analyzes their effectiveness and limitations. It also discusses the existing challenges of DL library testing and outlines potential directions for future research.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 46 of its 150 references that are in the map. First posted on arXiv 27 Apr 2024.

Builds on

46 of 150 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  4. 2019Quantifying the Carbon Emissions of Machine Learningcited by 516
  5. 2019A Survey of Deep Learning Techniques for Autonomous Drivingcited by 401
  6. 2014cuDNN: Efficient Primitives for Deep Learningcited by 240
  7. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 216
  8. 2019Machine Learning Testing: Survey, Landscapes and Horizonscited by 213
  9. 2018Auto-Keras: An Efficient Neural Architecture Search Systemcited by 169
  10. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 113
  11. 2022Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Modelscited by 102
  12. 2018Glow: Graph Lowering Compiler Techniques for Neural Networkscited by 73
34 more
  1. 2020Metamorphic Testing: A New Approach for Generating Next Test Casescited by 71
  2. 2021Software Engineering for AI-Based Systems: A Surveycited by 68
  3. 2019A Comprehensive Study on Deep Learning Bug Characteristicscited by 65
  4. 2018On Testing Machine Learning Programscited by 48
  5. 2020The Deep Learning Compiler: A Comprehensive Surveycited by 46
  6. 2019Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirectcited by 41
  7. 2021DeepLocalize: Fault Localization for Deep Neural Networkscited by 37
  8. 2019An Empirical Study towards Characterizing Deep Learning Development and Deployment across Different Frameworks and Platformscited by 35
  9. 2022NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilerscited by 29
  10. 2022Free Lunch for Testing: Fuzzing Deep-Learning Libraries from Open Sourcecited by 27
  11. 2022Toward Understanding Deep Learning Framework Bugscited by 26
  12. 2021Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlowcited by 25
  13. 2021DocTer: Documentation Guided Fuzzing for Testing Deep Learning API Functionscited by 21
  14. 2022Fuzzing Deep-Learning Libraries via Automated Relational API Inferencecited by 16
  15. 2023Fuzzing Automatic Differentiation in Deep-Learning Librariescited by 16
  16. 2022Muffin: Testing Deep Learning Libraries via Neural Architecture Fuzzingcited by 15
  17. 2021Understanding Performance Problems in Deep Learning Systemscited by 12
  18. 2022CodeEditor: Learning to Edit Source Code with Pre-trained Modelscited by 12
  19. 2022DeepStability: A Study of Unstable Numerical Methods and Their Solutions in Deep Learningcited by 11
  20. 2019A Modular Benchmarking Infrastructure for High-Performance and Reproducible Deep Learningcited by 9
  21. 2022Demystifying Dependency Bugs in Deep Learning Stackcited by 9
  22. 2022Fuzzing Deep Learning Compilers with HirGencited by 6
  23. 2022Towards Understanding the Faults of JavaScript-Based Deep Learning Systemscited by 6
  24. 2023NeuRI: Diversifying DNN Generation via Inductive Rule Inferencecited by 6
  25. 2020Smoke Testing for Machine Learning: Simple Tests to Discover Severe Defectscited by 5
  26. 2021Rise of Distributed Deep Learning Training in the Big Model Era: From a Software Engineering Perspectivecited by 4
  27. 2019Benanza: Automatic μBenchmark Generation to Compute "Lower-bound" Latency and Inform Optimizations of Deep Learning Models on GPUscited by 3
  28. 2022Characterizing and Understanding Software Security Vulnerabilities in Machine Learning Librariescited by 3
  29. 2022COMET: Coverage-guided Model Generation For Deep Learning Library Testingcited by 3
  30. 2023DeltaNN: Assessing the Impact of Computational Environment Parameters on the Performance of Image Recognition Modelscited by 3
  31. 2022Challenges in Migrating Imperative Deep Learning Programs to Graph Execution: An Empirical Studycited by 2
  32. 2023ACETest: Automated Constraint Extraction for Testing Deep Learning Operatorscited by 2
  33. 2020Comparing the costs of abstraction for DL frameworkscited by 1
  34. 2023Effective Random Test Generation for Deep Learning Compilerscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2026Where Agent Frameworks Fall Short: Examining Functional Challenges and Usability Concernscited by 3
  2. 2025Leveraging Neural Graph Compilers in Machine Learning Research for Edge-Cloud Systemscited by 2
  3. 2025TTrace: Lightweight Error Checking and Diagnosis for Distributed Trainingcited by 1
  4. 2026Demystifying the Silence of Correctness Bugs in PyTorch Compilercited by 0