# NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers

arXiv: https://arxiv.org/abs/2207.13066 · HTML: https://mlsanity.com/paper/2207.13066/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 79 references that are in the map.
- First posted on arXiv 26 Jul 2022.

## Builds on (most cited first, 20 of 20)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Empirical Evaluation of Rectified Activations in Convolutional Network](https://mlsanity.com/paper/1505.00853/) (2015)
- [Ansor: Generating High-Performance Tensor Programs for Deep Learning](https://mlsanity.com/paper/2006.06762/) (2020)
- [Learning to Optimize Tensor Programs](https://mlsanity.com/paper/1805.08166/) (2018)
- [The Art, Science, and Engineering of Fuzzing: A Survey](https://mlsanity.com/paper/1812.00140/) (2018)
- [Free Lunch for Testing: Fuzzing Deep-Learning Libraries from Open Source](https://mlsanity.com/paper/2201.06589/) (2022)
- [Muffin: Testing Deep Learning Libraries via Neural Architecture Fuzzing](https://mlsanity.com/paper/2204.08734/) (2022)
- [Fuzzing Deep-Learning Libraries via Automated Relational API Inference](https://mlsanity.com/paper/2207.05531/) (2022)
- [Automated Conformance Testing for JavaScript Engines via Deep Compiler Fuzzing](https://mlsanity.com/paper/2104.07460/) (2021)
- [Coverage-Guided Tensor Compiler Fuzzing with Joint IR-Pass Mutation](https://mlsanity.com/paper/2202.09947/) (2022)
- [Fuzzing Hardware Like Software](https://mlsanity.com/paper/2102.02308/) (2021)
- [Montage: A Neural Network Language Model-Guided JavaScript Engine Fuzzer](https://mlsanity.com/paper/2001.04107/) (2020)
- [Graph-Based Fuzz Testing for Deep Learning Inference Engine](https://mlsanity.com/paper/2008.05933/) (2020)
- [Skeletal Program Enumeration for Rigorous Compiler Testing](https://mlsanity.com/paper/1610.03148/) (2016)
- [A Systematic Impact Study for Fuzzer-Found Compiler Bugs](https://mlsanity.com/paper/1902.09334/) (2019)

## Built on by (most cited first, 30 of 30)

- [Fuzz4All: Universal Fuzzing with Large Language Models](https://mlsanity.com/paper/2308.04748/) (2023)
- [WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models](https://mlsanity.com/paper/2310.15991/) (2023)
- [Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection](https://mlsanity.com/paper/2404.09894/) (2024)
- [Are LLMs Correctly Integrated into Software Systems?](https://mlsanity.com/paper/2407.05138/) (2024)
- [Challenges and Practices of Deep Learning Model Reengineering: A Case Study on Computer Vision](https://mlsanity.com/paper/2303.07476/) (2023)
- [Training with Confidence: Catching Silent Errors in Deep Learning Training with Automated Proactive Checks](https://mlsanity.com/paper/2506.14813/) (2025)
- [DyCL: Dynamic Neural Network Compilation Via Program Rewriting and Graph Optimization](https://mlsanity.com/paper/2307.04963/) (2023)
- [NeuRI: Diversifying DNN Generation via Inductive Rule Inference](https://mlsanity.com/paper/2302.02261/) (2023)
- [Your Compiler is Backdooring Your Model: Understanding and Exploiting Compilation Inconsistency Vulnerabilities in Deep Learning Compilers](https://mlsanity.com/paper/2509.11173/) (2025)
- [Deep Learning Library Testing: Definition, Methods and Challenges](https://mlsanity.com/paper/2404.17871/) (2024)
- [TrainVerify: Equivalence-Based Verification for Distributed LLM Training](https://mlsanity.com/paper/2506.15961/) (2025)
- [Where Agent Frameworks Fall Short: Examining Functional Challenges and Usability Concerns](https://mlsanity.com/paper/2602.21806/) (2026)
- [Productively Deploying Emerging Models on Emerging Platforms: A Top-Down Approach for Testing and Debugging](https://mlsanity.com/paper/2404.09151/) (2024)
- [Evaluating LLMs on Sequential API Call Through Automated Test Generation](https://mlsanity.com/paper/2507.09481/) (2025)
- [DiTOX: Fault Detection and Localization in the ONNX Optimizer](https://mlsanity.com/paper/2505.01892/) (2025)
- [TTrace: Lightweight Error Checking and Diagnosis for Distributed Training](https://mlsanity.com/paper/2506.09280/) (2025)
- [GPU-Fuzz: Finding Memory Errors in Deep Learning Frameworks](https://mlsanity.com/paper/2602.10478/) (2026)
- [Exploring Robustness of Image Recognition Models on Hardware Accelerators](https://mlsanity.com/paper/2306.01697/) (2023)
- [ML2Tuner: Efficient Code Tuning via Multi-Level Machine Learning Models](https://mlsanity.com/paper/2411.10764/) (2024)
- [Verify Distributed Deep Learning Model Implementation Refinement with Iterative Relation Inference](https://mlsanity.com/paper/2508.09505/) (2025)
- [Once4All: Skeleton-Guided SMT Solver Fuzzing with LLM-Synthesized Generators](https://mlsanity.com/paper/2508.20340/) (2025)
- [M2K: Making the Model-Kernel Interface Explicit for Reliable CUDA Kernel Verification](https://mlsanity.com/paper/2603.24595/) (2026)
- [Demystifying the Silence of Correctness Bugs in PyTorch Compiler](https://mlsanity.com/paper/2604.08720/) (2026)
- [PPO guided Agentic Pipeline for Adaptive Prompt Selection and Test Case Generation](https://mlsanity.com/paper/2605.00942/) (2026)
- [FeedbackLLM: Metadata driven Multi-Agentic Language Agnostic Test Case Generator with Evolving prompt and Coverage Feedback](https://mlsanity.com/paper/2605.01264/) (2026)
- [Detecting and Understanding Vulnerabilities in Fully Homomorphic Encryption Frameworks](https://mlsanity.com/paper/2606.22519/) (2026)
- [From Expert Reduction to Behavioral Divergence: Tracing Numerical State through Sparse MoE Inference](https://mlsanity.com/paper/2607.28097/) (2026)
- [Measuring the Checker: Mutation Analysis for GPU-Kernel Benchmark Oracles](https://mlsanity.com/paper/2609.22220/) (2026)
- [Silent Failures Beyond the 32-Bit Index Range: A Differential Characterization of Large-Tensor Matrix Multiplication in PyTorch's MPS Backend](https://mlsanity.com/paper/2609.22991/) (2026)
- [Grounding Deep Learning Framework Testing in Data Flow](https://mlsanity.com/paper/2610.15808/) (2026)

## Abstract (arXiv)

Deep-learning (DL) compilers such as TVM and TensorRT are increasingly being used to optimize deep neural network (DNN) models to meet performance, resource utilization and other requirements. Bugs in these compilers can result in models whose semantics differ from the original ones, producing incorrect results that corrupt the correctness of downstream applications. However, finding bugs in these compilers is challenging due to their complexity. In this work, we propose a new fuzz testing approach for finding bugs in deep-learning compilers. Our core approach consists of (i) generating diverse yet valid DNN test models that can exercise a large part of the compiler's transformation logic using light-weight operator specifications; (ii) performing gradient-based search to find model inputs that avoid any floating-point exceptional values during model execution, reducing the chance of missed bugs or false alarms; and (iii) using differential testing to identify bugs. We implemented this approach in NNSmith which has found 72 new bugs for TVM, TensorRT, ONNXRuntime, and PyTorch to date. Of these 58 have been confirmed and 51 have been fixed by their respective project maintainers.