# Deep Learning Library Testing: Definition, Methods and Challenges

arXiv: https://arxiv.org/abs/2404.17871 · HTML: https://mlsanity.com/paper/2404.17871/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 150 references that are in the map.
- First posted on arXiv 27 Apr 2024.

## Builds on (most cited first, 30 of 46)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Carbon Emissions and Large Neural Network Training](https://mlsanity.com/paper/2104.10350/) (2021)
- [Quantifying the Carbon Emissions of Machine Learning](https://mlsanity.com/paper/1910.09700/) (2019)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [cuDNN: Efficient Primitives for Deep Learning](https://mlsanity.com/paper/1410.0759/) (2014)
- [MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases](https://mlsanity.com/paper/2402.14905/) (2024)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [Auto-Keras: An Efficient Neural Architecture Search System](https://mlsanity.com/paper/1806.10282/) (2018)
- [Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark](https://mlsanity.com/paper/2402.11592/) (2024)
- [Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models](https://mlsanity.com/paper/2212.14834/) (2022)
- [Glow: Graph Lowering Compiler Techniques for Neural Networks](https://mlsanity.com/paper/1805.00907/) (2018)
- [Metamorphic Testing: A New Approach for Generating Next Test Cases](https://mlsanity.com/paper/2002.12543/) (2020)
- [Software Engineering for AI-Based Systems: A Survey](https://mlsanity.com/paper/2105.01984/) (2021)
- [A Comprehensive Study on Deep Learning Bug Characteristics](https://mlsanity.com/paper/1906.01388/) (2019)
- [On Testing Machine Learning Programs](https://mlsanity.com/paper/1812.02257/) (2018)
- [The Deep Learning Compiler: A Comprehensive Survey](https://mlsanity.com/paper/2002.03794/) (2020)
- [Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect](https://mlsanity.com/paper/1903.04611/) (2019)
- [DeepLocalize: Fault Localization for Deep Neural Networks](https://mlsanity.com/paper/2103.03376/) (2021)
- [An Empirical Study towards Characterizing Deep Learning Development and Deployment across Different Frameworks and Platforms](https://mlsanity.com/paper/1909.06727/) (2019)
- [NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers](https://mlsanity.com/paper/2207.13066/) (2022)
- [Free Lunch for Testing: Fuzzing Deep-Learning Libraries from Open Source](https://mlsanity.com/paper/2201.06589/) (2022)
- [Toward Understanding Deep Learning Framework Bugs](https://mlsanity.com/paper/2203.04026/) (2022)
- [Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlow](https://mlsanity.com/paper/2112.13314/) (2021)
- [DocTer: Documentation Guided Fuzzing for Testing Deep Learning API Functions](https://mlsanity.com/paper/2109.01002/) (2021)
- [Fuzzing Automatic Differentiation in Deep-Learning Libraries](https://mlsanity.com/paper/2302.04351/) (2023)
- [Muffin: Testing Deep Learning Libraries via Neural Architecture Fuzzing](https://mlsanity.com/paper/2204.08734/) (2022)
- [Fuzzing Deep-Learning Libraries via Automated Relational API Inference](https://mlsanity.com/paper/2207.05531/) (2022)
- [Understanding Performance Problems in Deep Learning Systems](https://mlsanity.com/paper/2112.01771/) (2021)
- [CodeEditor: Learning to Edit Source Code with Pre-trained Models](https://mlsanity.com/paper/2210.17040/) (2022)

## Built on by (most cited first, 5 of 5)

- [Where Agent Frameworks Fall Short: Examining Functional Challenges and Usability Concerns](https://mlsanity.com/paper/2602.21806/) (2026)
- [Leveraging Neural Graph Compilers in Machine Learning Research for Edge-Cloud Systems](https://mlsanity.com/paper/2504.20198/) (2025)
- [TTrace: Lightweight Error Checking and Diagnosis for Distributed Training](https://mlsanity.com/paper/2506.09280/) (2025)
- [Demystifying the Silence of Correctness Bugs in PyTorch Compiler](https://mlsanity.com/paper/2604.08720/) (2026)
- [Grounding Deep Learning Framework Testing in Data Flow](https://mlsanity.com/paper/2610.15808/) (2026)

## Abstract (arXiv)

In recent years, software systems powered by deep learning (DL) techniques have significantly facilitated people's lives in many aspects. As the backbone of these DL systems, various DL libraries undertake the underlying optimization and computation. However, like traditional software, DL libraries are not immune to bugs, which can pose serious threats to users' personal property and safety. Studying the characteristics of DL libraries, their associated bugs, and the corresponding testing methods is crucial for enhancing the security of DL systems and advancing the widespread application of DL technology. This paper provides an overview of the testing research related to various DL libraries, discusses the strengths and weaknesses of existing methods, and provides guidance and reference for the application of the DL library. This paper first introduces the workflow of DL underlying libraries and the characteristics of three kinds of DL libraries involved, namely DL framework, DL compiler, and DL hardware library. It then provides definitions for DL underlying library bugs and testing. Additionally, this paper summarizes the existing testing methods and tools tailored to these DL libraries separately and analyzes their effectiveness and limitations. It also discusses the existing challenges of DL library testing and outlines potential directions for future research.