# Muffin: Testing Deep Learning Libraries via Neural Architecture Fuzzing

arXiv: https://arxiv.org/abs/2204.08734 · HTML: https://mlsanity.com/paper/2204.08734/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Apr 2022.

## Built on by (most cited first, 16 of 16)

- [WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models](https://mlsanity.com/paper/2310.15991/) (2023)
- [NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers](https://mlsanity.com/paper/2207.13066/) (2022)
- [Large Language Models Synergize with Automated Machine Learning](https://mlsanity.com/paper/2405.03727/) (2024)
- [NeuRI: Diversifying DNN Generation via Inductive Rule Inference](https://mlsanity.com/paper/2302.02261/) (2023)
- [Deep Learning Library Testing: Definition, Methods and Challenges](https://mlsanity.com/paper/2404.17871/) (2024)
- [Enhancing Differential Testing With LLMs For Testing Deep Learning Libraries](https://mlsanity.com/paper/2406.07944/) (2024)
- [Hidden Reliability Risks in Large Language Models: Systematic Identification of Precision-Induced Output Disagreements](https://mlsanity.com/paper/2604.19790/) (2026)
- [COMET: Coverage-guided Model Generation For Deep Learning Library Testing](https://mlsanity.com/paper/2208.01508/) (2022)
- [On the Challenges of Fuzzing Techniques via Large Language Models](https://mlsanity.com/paper/2402.00350/) (2024)
- [Test2Vec: An Execution Trace Embedding for Test Case Prioritization](https://mlsanity.com/paper/2206.15428/) (2022)
- [Generative transformations and patterns in LLM-native approaches for software verification and falsification](https://mlsanity.com/paper/2404.09384/) (2024)
- [Analysis of Failures and Risks in Deep Learning Model Converters: A Case Study in the ONNX Ecosystem](https://mlsanity.com/paper/2303.17708/) (2023)
- [Fuzzy Logic Guided Reward Function Variation: An Oracle for Testing Reinforcement Learning Programs](https://mlsanity.com/paper/2406.19812/) (2024)
- [CAP: Detecting Unauthorized Data Usage in Generative Models via Prompt Generation](https://mlsanity.com/paper/2410.05819/) (2024)
- [GPU Temperature Simulation-Based Testing for In-Vehicle Deep Learning Frameworks](https://mlsanity.com/paper/2509.15815/) (2025)
- [Grounding Deep Learning Framework Testing in Data Flow](https://mlsanity.com/paper/2610.15808/) (2026)

## Abstract (arXiv)

Deep learning (DL) techniques are proven effective in many challenging tasks, and become widely-adopted in practice. However, previous work has shown that DL libraries, the basis of building and executing DL models, contain bugs and can cause severe consequences. Unfortunately, existing testing approaches still cannot comprehensively exercise DL libraries. They utilize existing trained models and only detect bugs in model inference phase. In this work we propose Muffin to address these issues. To this end, Muffin applies a specifically-designed model fuzzing approach, which allows it to generate diverse DL models to explore the target library, instead of relying only on existing trained models. Muffin makes differential testing feasible in the model training phase by tailoring a set of metrics to measure the inconsistencies between different DL libraries. In this way, Muffin can best exercise the library code to detect more bugs. To evaluate the effectiveness of Muffin, we conduct experiments on three widely-used DL libraries. The results demonstrate that Muffin can detect 39 new bugs in the latest release versions of popular DL libraries, including Tensorflow, CNTK, and Theano.