# Fuzzing Deep-Learning Libraries via Automated Relational API Inference

arXiv: https://arxiv.org/abs/2207.05531 · HTML: https://mlsanity.com/paper/2207.05531/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jul 2022.

## Built on by (most cited first, 16 of 16)

- [NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers](https://mlsanity.com/paper/2207.13066/) (2022)
- [NeuRI: Diversifying DNN Generation via Inductive Rule Inference](https://mlsanity.com/paper/2302.02261/) (2023)
- [The Correctness Illusion in LLM-Generated GPU Kernels](https://mlsanity.com/paper/2606.20128/) (2026)
- [Deep Learning Library Testing: Definition, Methods and Challenges](https://mlsanity.com/paper/2404.17871/) (2024)
- [Enhancing Differential Testing With LLMs For Testing Deep Learning Libraries](https://mlsanity.com/paper/2406.07944/) (2024)
- [Test & Evaluation Best Practices for Machine Learning-Enabled Systems](https://mlsanity.com/paper/2310.06800/) (2023)
- [On the Challenges of Fuzzing Techniques via Large Language Models](https://mlsanity.com/paper/2402.00350/) (2024)
- [Where Agent Frameworks Fall Short: Examining Functional Challenges and Usability Concerns](https://mlsanity.com/paper/2602.21806/) (2026)
- [Productively Deploying Emerging Models on Emerging Platforms: A Top-Down Approach for Testing and Debugging](https://mlsanity.com/paper/2404.09151/) (2024)
- [Operator-Aware Mixed-Precision Tolerance Calibration for Tensor Kernels](https://mlsanity.com/paper/2607.16228/) (2026)
- [ConFL: Constraint-guided Fuzzing for Machine Learning Framework](https://mlsanity.com/paper/2307.05642/) (2023)
- [Towards Reliable LLM-Driven Fuzz Testing: Vision and Road Ahead](https://mlsanity.com/paper/2503.00795/) (2025)
- [On the Mistaken Assumption of Interchangeable Deep Reinforcement Learning Implementations](https://mlsanity.com/paper/2503.22575/) (2025)
- [GPU Temperature Simulation-Based Testing for In-Vehicle Deep Learning Frameworks](https://mlsanity.com/paper/2509.15815/) (2025)
- [Test-Input Generation for Tensor Programs: What Actually Finds Kernel Bugs](https://mlsanity.com/paper/2606.27396/) (2026)
- [Directed Neuro-Symbolic Stochastic Execution for Verification of Distributed Parallel AI Programs](https://mlsanity.com/paper/2608.07947/) (2026)

## Abstract (arXiv)

A growing body of research has been dedicated to DL model testing. However, there is still limited work on testing DL libraries, which serve as the foundations for building, training, and running DL models. Prior work on fuzzing DL libraries can only generate tests for APIs which have been invoked by documentation examples, developer tests, or DL models, leaving a large number of APIs untested. In this paper, we propose DeepREL, the first approach to automatically inferring relational APIs for more effective DL library fuzzing. Our basic hypothesis is that for a DL library under test, there may exist a number of APIs sharing similar input parameters and outputs; in this way, we can easily "borrow" test inputs from invoked APIs to test other relational APIs. Furthermore, we formalize the notion of value equivalence and status equivalence for relational APIs to serve as the oracle for effective bug finding. We have implemented DeepREL as a fully automated end-to-end relational API inference and fuzzing technique for DL libraries, which 1) automatically infers potential API relations based on API syntactic or semantic information, 2) synthesizes concrete test programs for invoking relational APIs, 3) validates the inferred relational APIs via representative test inputs, and finally 4) performs fuzzing on the verified relational APIs to find potential inconsistencies. Our evaluation on two of the most popular DL libraries, PyTorch and TensorFlow, demonstrates that DeepREL can cover 157% more APIs than state-of-the-art FreeFuzz. To date, DeepREL has detected 162 bugs in total, with 106 already confirmed by the developers as previously unknown bugs. Surprisingly, DeepREL has detected 13.5% of the high-priority bugs for the entire PyTorch issue-tracking system in a three-month period. Also, besides the 162 code bugs, we have also detected 14 documentation bugs (all confirmed).