# How Effective Are They? Exploring Large Language Model Based Fuzz Driver Generation

arXiv: https://arxiv.org/abs/2307.12469 · HTML: https://mlsanity.com/paper/2307.12469/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Jul 2023.

## Built on by (most cited first, 21 of 21)

- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks](https://mlsanity.com/paper/2506.11791/) (2025)
- [LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights](https://mlsanity.com/paper/2502.07049/) (2025)
- [Locus: Agentic Predicate Synthesis for Directed Fuzzing](https://mlsanity.com/paper/2508.21302/) (2025)
- [When Fuzzing Meets LLMs: Challenges and Opportunities](https://mlsanity.com/paper/2404.16297/) (2024)
- [AutoPT: How Far Are We from the End2End Automated Web Penetration Testing?](https://mlsanity.com/paper/2411.01236/) (2024)
- [Testing learning-enabled cyber-physical systems with Large-Language Models: A Formal Approach](https://mlsanity.com/paper/2311.07377/) (2023)
- [Open-Source AI-based SE Tools: Opportunities and Challenges of Collaborative Software Learning](https://mlsanity.com/paper/2404.06201/) (2024)
- [HarnessAgent: Scaling Automatic Fuzzing Harness Construction with Tool-Augmented LLM Pipelines](https://mlsanity.com/paper/2512.03420/) (2025)
- [Generative transformations and patterns in LLM-native approaches for software verification and falsification](https://mlsanity.com/paper/2404.09384/) (2024)
- [FalseCrashReducer: Mitigating False Positive Crashes in OSS-Fuzz-Gen Using Agentic AI](https://mlsanity.com/paper/2510.02185/) (2025)
- [deepSURF: Detecting Memory Safety Vulnerabilities in Rust Through Fuzzing LLM-Augmented Harnesses](https://mlsanity.com/paper/2506.15648/) (2025)
- [AgentExecutor: Partial Code Execution via Agentic Context Generation](https://mlsanity.com/paper/2608.05959/) (2026)
- [Towards Reliable LLM-Driven Fuzz Testing: Vision and Road Ahead](https://mlsanity.com/paper/2503.00795/) (2025)
- [Large Language Models in Cybersecurity: Applications, Vulnerabilities, and Defense Techniques](https://mlsanity.com/paper/2507.13629/) (2025)
- [Orion: Fuzzing Workflow Automation](https://mlsanity.com/paper/2509.15195/) (2025)
- [Coverage-Guided Multi-Agent Harness Generation for Java Library Fuzzing](https://mlsanity.com/paper/2603.08616/) (2026)
- [LLM-Assisted Dynamic Threat Analysis for Attacker-Reachable Software Weaknesses in Autonomous Vehicles](https://mlsanity.com/paper/2608.13450/) (2026)
- [AI Slop and Hallucinations in Vulnerability Assessment: A Survey on Reasoning Failures and Trustworthy Mitigation](https://mlsanity.com/paper/2608.25667/) (2026)
- [CIT-CAD: Constraint Intent Tree-based CAD Code Generation and Verification](https://mlsanity.com/paper/2609.07434/) (2026)
- [SyzHarness: Patch-Based Kernel Bug Reproduction with LLM-Synthesized Fuzzing Harnesses](https://mlsanity.com/paper/2609.23889/) (2026)

## Abstract (arXiv)

LLM-based (Large Language Model) fuzz driver generation is a promising research area. Unlike traditional program analysis-based method, this text-based approach is more general and capable of harnessing a variety of API usage information, resulting in code that is friendly for human readers. However, there is still a lack of understanding regarding the fundamental issues on this direction, such as its effectiveness and potential challenges. To bridge this gap, we conducted the first in-depth study targeting the important issues of using LLMs to generate effective fuzz drivers. Our study features a curated dataset with 86 fuzz driver generation questions from 30 widely-used C projects. Six prompting strategies are designed and tested across five state-of-the-art LLMs with five different temperature settings. In total, our study evaluated 736,430 generated fuzz drivers, with 0.85 billion token costs (8,000+ charged tokens). Additionally, we compared the LLM-generated drivers against those utilized in industry, conducting extensive fuzzing experiments (3.75 CPU-year). Our study uncovered that: - While LLM-based fuzz driver generation is a promising direction, it still encounters several obstacles towards practical applications; - LLMs face difficulties in generating effective fuzz drivers for APIs with intricate specifics. Three featured design choices of prompt strategies can be beneficial: issuing repeat queries, querying with examples, and employing an iterative querying process; - While LLM-generated drivers can yield fuzzing outcomes that are on par with those used in the industry, there are substantial opportunities for enhancement, such as extending contained API usage, or integrating semantic oracles to facilitate logical bug detection. Our insights have been implemented to improve the OSS-Fuzz-Gen project, facilitating practical fuzz driver generation in industry.