# Fuzz4All: Universal Fuzzing with Large Language Models

arXiv: https://arxiv.org/abs/2308.04748 · HTML: https://mlsanity.com/paper/2308.04748/

## Key facts

- Cited by 88 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 84 references that are in the map.
- First posted on arXiv 9 Aug 2023.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [Large Language Models Are Human-Level Prompt Engineers](https://mlsanity.com/paper/2211.01910/) (2022)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [StarCoder: may the source be with you!](https://mlsanity.com/paper/2305.06161/) (2023)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [A Survey on Automated Fact-Checking](https://mlsanity.com/paper/2108.11896/) (2021)
- [Learning How to Ask: Querying LMs with Mixtures of Soft Prompts](https://mlsanity.com/paper/2104.06599/) (2021)
- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [Keep the Conversation Going: Fixing 162 out of 337 bugs for 0.42 each using ChatGPT](https://mlsanity.com/paper/2304.00385/) (2023)
- [Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Models](https://mlsanity.com/paper/2212.14834/) (2022)
- [No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation](https://mlsanity.com/paper/2305.04207/) (2023)
- [No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligence](https://mlsanity.com/paper/2207.11680/) (2022)
- [Code Generation Tools (Almost) for Free? A Study of Few-Shot, Pre-Trained Language Models on Code](https://mlsanity.com/paper/2206.01335/) (2022)
- [Evaluating Fuzz Testing](https://mlsanity.com/paper/1808.09700/) (2018)
- [Can Large Language Models Write Good Property-Based Tests?](https://mlsanity.com/paper/2307.04346/) (2023)
- [NNSmith: Generating Diverse and Valid Test Cases for Deep Learning Compilers](https://mlsanity.com/paper/2207.13066/) (2022)
- [TF-Coder: Program Synthesis for Tensor Manipulations](https://mlsanity.com/paper/2003.09040/) (2020)
- [Learn&Fuzz: Machine Learning for Input Fuzzing](https://mlsanity.com/paper/1701.07232/) (2017)
- [Montage: A Neural Network Language Model-Guided JavaScript Engine Fuzzer](https://mlsanity.com/paper/2001.04107/) (2020)

## Built on by (most cited first, 30 of 88)

- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [RepairAgent: An Autonomous, LLM-Based Agent for Program Repair](https://mlsanity.com/paper/2403.17134/) (2024)
- [From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future](https://mlsanity.com/paper/2408.02479/) (2024)
- [SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks](https://mlsanity.com/paper/2506.11791/) (2025)
- [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024)
- [SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Manner](https://mlsanity.com/paper/2406.05498/) (2024)
- [Unifying the Perspectives of NLP and Software Engineering: A Survey on Language Models for Code](https://mlsanity.com/paper/2311.07989/) (2023)
- [LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoning](https://mlsanity.com/paper/2401.16185/) (2024)
- [Large Language Models for Compiler Optimization](https://mlsanity.com/paper/2309.07062/) (2023)
- [You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projects](https://mlsanity.com/paper/2412.10133/) (2024)
- [Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMs](https://mlsanity.com/paper/2308.09895/) (2023)
- [Can LLMs Replace Manual Annotation of Software Engineering Artifacts?](https://mlsanity.com/paper/2408.05534/) (2024)
- [Beyond pip install: Evaluating LLM Agents for the Automated Installation of Python Projects](https://mlsanity.com/paper/2412.06294/) (2024)
- [When Large Language Models Meet Evolutionary Algorithms: Potential Enhancements and Challenges](https://mlsanity.com/paper/2401.10510/) (2024)
- [Moderating New Waves of Online Hate with Chain-of-Thought Reasoning in Large Language Models](https://mlsanity.com/paper/2312.15099/) (2023)
- [Are LLMs Correctly Integrated into Software Systems?](https://mlsanity.com/paper/2407.05138/) (2024)
- [CTINexus: Automatic Cyber Threat Intelligence Knowledge Graph Construction Using Large Language Models](https://mlsanity.com/paper/2410.21060/) (2024)
- [Enhancing Genetic Improvement Mutations Using Large Language Models](https://mlsanity.com/paper/2310.19813/) (2023)
- [A Contemporary Survey of Large Language Model Assisted Program Analysis](https://mlsanity.com/paper/2502.18474/) (2025)
- [VulnLLM-R: Specialized Reasoning LLM with Agent Scaffold for Vulnerability Detection](https://mlsanity.com/paper/2512.07533/) (2025)
- [LLMs Can Defend Themselves Against Jailbreaking in a Practical Manner: A Vision Paper](https://mlsanity.com/paper/2402.15727/) (2024)
- [RepoMasterEval: Evaluating Code Completion via Real-World Repositories](https://mlsanity.com/paper/2408.03519/) (2024)
- [Incident Response Planning Using a Lightweight Large Language Model with Reduced Hallucination](https://mlsanity.com/paper/2508.05188/) (2025)
- [Locus: Agentic Predicate Synthesis for Directed Fuzzing](https://mlsanity.com/paper/2508.21302/) (2025)
- [A System for Automated Unit Test Generation Using Large Language Models and Assessment of Generated Test Suites](https://mlsanity.com/paper/2408.07846/) (2024)
- [`From Prompt to Perturbation': An Adaptive Framework for Voice-Based Jailbreaks on Audio LLMs](https://mlsanity.com/paper/2502.00735/) (2025)
- [Validating Network Protocol Parsers with Traceable RFC Document Interpretation](https://mlsanity.com/paper/2504.18050/) (2025)
- [Hybrid Fuzzing with LLM-Guided Input Mutation and Semantic Feedback](https://mlsanity.com/paper/2511.03995/) (2025)
- [Agentic Verification of Software Systems](https://mlsanity.com/paper/2511.17330/) (2025)
- [Identify Critical Nodes in Complex Network with Large Language Models](https://mlsanity.com/paper/2403.03962/) (2024)

## Abstract (arXiv)

Fuzzing has achieved tremendous success in discovering bugs and vulnerabilities in various software systems. Systems under test (SUTs) that take in programming or formal language as inputs, e.g., compilers, runtime engines, constraint solvers, and software libraries with accessible APIs, are especially important as they are fundamental building blocks of software development. However, existing fuzzers for such systems often target a specific language, and thus cannot be easily applied to other languages or even other versions of the same language. Moreover, the inputs generated by existing fuzzers are often limited to specific features of the input language, and thus can hardly reveal bugs related to other or new features. This paper presents Fuzz4All, the first fuzzer that is universal in the sense that it can target many different input languages and many different features of these languages. The key idea behind Fuzz4All is to leverage large language models (LLMs) as an input generation and mutation engine, which enables the approach to produce diverse and realistic inputs for any practically relevant language. To realize this potential, we present a novel autoprompting technique, which creates LLM prompts that are wellsuited for fuzzing, and a novel LLM-powered fuzzing loop, which iteratively updates the prompt to create new fuzzing inputs. We evaluate Fuzz4All on nine systems under test that take in six different languages (C, C++, Go, SMT2, Java and Python) as inputs. The evaluation shows, across all six languages, that universal fuzzing achieves higher coverage than existing, language-specific fuzzers. Furthermore, Fuzz4All has identified 98 bugs in widely used systems, such as GCC, Clang, Z3, CVC5, OpenJDK, and the Qiskit quantum computing platform, with 64 bugs already confirmed by developers as previously unknown.