# Fuzzing Automatic Differentiation in Deep-Learning Libraries

arXiv: https://arxiv.org/abs/2302.04351 · HTML: https://mlsanity.com/paper/2302.04351/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Feb 2023.

## Built on by (most cited first, 17 of 17)

- [WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models](https://mlsanity.com/paper/2310.15991/) (2023)
- [KNighter: Transforming Static Analysis with LLM-Synthesized Checkers](https://mlsanity.com/paper/2503.09002/) (2025)
- [KernelGPT: Enhanced Kernel Fuzzing via Large Language Models](https://mlsanity.com/paper/2401.00563/) (2023)
- [Retrieval-Augmented Test Generation: How Far Are We?](https://mlsanity.com/paper/2409.12682/) (2024)
- [The Correctness Illusion in LLM-Generated GPU Kernels](https://mlsanity.com/paper/2606.20128/) (2026)
- [Deep Learning Library Testing: Definition, Methods and Challenges](https://mlsanity.com/paper/2404.17871/) (2024)
- [Enhancing Differential Testing With LLMs For Testing Deep Learning Libraries](https://mlsanity.com/paper/2406.07944/) (2024)
- [Test & Evaluation Best Practices for Machine Learning-Enabled Systems](https://mlsanity.com/paper/2310.06800/) (2023)
- [Retromorphic Testing: A New Approach to the Test Oracle Problem](https://mlsanity.com/paper/2310.06433/) (2023)
- [Productively Deploying Emerging Models on Emerging Platforms: A Top-Down Approach for Testing and Debugging](https://mlsanity.com/paper/2404.09151/) (2024)
- [TTrace: Lightweight Error Checking and Diagnosis for Distributed Training](https://mlsanity.com/paper/2506.09280/) (2025)
- [HFuzzer: Testing Large Language Models for Package Hallucinations via Phrase-based Fuzzing](https://mlsanity.com/paper/2509.23835/) (2025)
- [Operator-Aware Mixed-Precision Tolerance Calibration for Tensor Kernels](https://mlsanity.com/paper/2607.16228/) (2026)
- [Zero Coordinate Shift: Whetted Automatic Differentiation for Physics-informed Operator Learning](https://mlsanity.com/paper/2311.00860/) (2023)
- [Test-Input Generation for Tensor Programs: What Actually Finds Kernel Bugs](https://mlsanity.com/paper/2606.27396/) (2026)
- [Silent Failures Beyond the 32-Bit Index Range: A Differential Characterization of Large-Tensor Matrix Multiplication in PyTorch's MPS Backend](https://mlsanity.com/paper/2609.22991/) (2026)
- [Grounding Deep Learning Framework Testing in Data Flow](https://mlsanity.com/paper/2610.15808/) (2026)

## Abstract (arXiv)

Deep learning (DL) has attracted wide attention and has been widely deployed in recent years. As a result, more and more research efforts have been dedicated to testing DL libraries and frameworks. However, existing work largely overlooked one crucial component of any DL system, automatic differentiation (AD), which is the basis for the recent development of DL. To this end, we propose nablaFuzz, the first general and practical approach specifically targeting the critical AD component in DL libraries. Our key insight is that each DL library API can be abstracted into a function processing tensors/vectors, which can be differentially tested under various execution scenarios (for computing outputs/gradients with different implementations). We have implemented nablaFuzz as a fully automated API-level fuzzer targeting AD in DL libraries, which utilizes differential testing on different execution scenarios to test both first-order and high-order gradients, and also includes automated filtering strategies to remove false positives caused by numerical instability. We have performed an extensive study on four of the most popular and actively-maintained DL libraries, PyTorch, TensorFlow, JAX, and OneFlow. The result shows that nablaFuzz substantially outperforms state-of-the-art fuzzers in terms of both code coverage and bug detection. To date, nablaFuzz has detected 173 bugs for the studied DL libraries, with 144 already confirmed by developers (117 of which are previously unknown bugs and 107 are related to AD). Remarkably, nablaFuzz contributed 58.3% (7/12) of all high-priority AD bugs for PyTorch and JAX during a two-month period. None of the confirmed AD bugs were detected by existing fuzzers.