# DocTer: Documentation Guided Fuzzing for Testing Deep Learning API Functions

arXiv: https://arxiv.org/abs/2109.01002 · HTML: https://mlsanity.com/paper/2109.01002/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Sep 2021.

## Built on by (most cited first, 22 of 22)

- [WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models](https://mlsanity.com/paper/2310.15991/) (2023)
- [Are LLMs Correctly Integrated into Software Systems?](https://mlsanity.com/paper/2407.05138/) (2024)
- [NeuRI: Diversifying DNN Generation via Inductive Rule Inference](https://mlsanity.com/paper/2302.02261/) (2023)
- [The Correctness Illusion in LLM-Generated GPU Kernels](https://mlsanity.com/paper/2606.20128/) (2026)
- [Deep Learning Library Testing: Definition, Methods and Challenges](https://mlsanity.com/paper/2404.17871/) (2024)
- [Enhancing Differential Testing With LLMs For Testing Deep Learning Libraries](https://mlsanity.com/paper/2406.07944/) (2024)
- [No Attack Required: Semantic Fuzzing for Specification Violations in Agent Skills](https://mlsanity.com/paper/2605.13044/) (2026)
- [COMET: Coverage-guided Model Generation For Deep Learning Library Testing](https://mlsanity.com/paper/2208.01508/) (2022)
- [Test & Evaluation Best Practices for Machine Learning-Enabled Systems](https://mlsanity.com/paper/2310.06800/) (2023)
- [Semantic-Based Neural Network Repair](https://mlsanity.com/paper/2306.07995/) (2023)
- [Where Agent Frameworks Fall Short: Examining Functional Challenges and Usability Concerns](https://mlsanity.com/paper/2602.21806/) (2026)
- [What Kinds of Contracts Do ML APIs Need?](https://mlsanity.com/paper/2307.14465/) (2023)
- [Productively Deploying Emerging Models on Emerging Platforms: A Top-Down Approach for Testing and Debugging](https://mlsanity.com/paper/2404.09151/) (2024)
- [FuzzCoder: Byte-level Fuzzing Test via Large Language Model](https://mlsanity.com/paper/2409.01944/) (2024)
- [ExAIS: Executable AI Semantics](https://mlsanity.com/paper/2202.09868/) (2022)
- [An Empirical Study of Bugs in Data Visualization Libraries](https://mlsanity.com/paper/2506.15084/) (2025)
- [Operator-Aware Mixed-Precision Tolerance Calibration for Tensor Kernels](https://mlsanity.com/paper/2607.16228/) (2026)
- [ConFL: Constraint-guided Fuzzing for Machine Learning Framework](https://mlsanity.com/paper/2307.05642/) (2023)
- [Improving Performance of Commercially Available AI Products in a Multi-Agent Configuration](https://mlsanity.com/paper/2410.22129/) (2024)
- [ATTest: Agent-Driven Tensor Testing for Deep Learning Library Modules](https://mlsanity.com/paper/2602.13987/) (2026)
- [Test-Input Generation for Tensor Programs: What Actually Finds Kernel Bugs](https://mlsanity.com/paper/2606.27396/) (2026)
- [Grounding Deep Learning Framework Testing in Data Flow](https://mlsanity.com/paper/2610.15808/) (2026)

## Abstract (arXiv)

Input constraints are useful for many software development tasks. For example, input constraints of a function enable the generation of valid inputs, i.e., inputs that follow these constraints, to test the function deeper. API functions of deep learning (DL) libraries have DL specific input constraints, which are described informally in the free form API documentation. Existing constraint extraction techniques are ineffective for extracting DL specific input constraints. To fill this gap, we design and implement a new technique, DocTer, to analyze API documentation to extract DL specific input constraints for DL API functions. DocTer features a novel algorithm that automatically constructs rules to extract API parameter constraints from syntactic patterns in the form of dependency parse trees of API descriptions. These rules are then applied to a large volume of API documents in popular DL libraries to extract their input parameter constraints. To demonstrate the effectiveness of the extracted constraints, DocTer uses the constraints to enable the automatic generation of valid and invalid inputs to test DL API functions. Our evaluation on three popular DL libraries (TensorFlow, PyTorch, and MXNet) shows that the precision of DocTer in extracting input constraints is 85.4%. DocTer detects 94 bugs from 174 API functions, including one previously unknown security vulnerability that is now documented in the CVE database, while a baseline technique without input constraints detects only 59 bugs. Most (63) of the 94 bugs are previously unknown, 54 of which have been fixed or confirmed by developers after we report them. In addition, DocTer detects 43 inconsistencies in documents, 39 of which are fixed or confirmed.