# Evaluating Fuzz Testing

arXiv: https://arxiv.org/abs/1808.09700 · HTML: https://mlsanity.com/paper/1808.09700/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Aug 2018.

## Built on by (most cited first, 30 of 32)

- [Fuzz4All: Universal Fuzzing with Large Language Models](https://mlsanity.com/paper/2308.04748/) (2023)
- [CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale](https://mlsanity.com/paper/2506.02548/) (2025)
- [WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models](https://mlsanity.com/paper/2310.15991/) (2023)
- [PAPILLON: Efficient and Stealthy Fuzz Testing-Powered Jailbreaks for LLMs](https://mlsanity.com/paper/2409.14866/) (2024)
- [LLAMAFUZZ: Large Language Model Enhanced Greybox Fuzzing](https://mlsanity.com/paper/2406.07714/) (2024)
- [Constrained Sampling for Language Models Should Be Easy: An MCMC Perspective](https://mlsanity.com/paper/2506.05754/) (2025)
- [MDPFuzz: Testing Models Solving Markov Decision Processes](https://mlsanity.com/paper/2112.02807/) (2021)
- [Comprehensive Evaluation of ChatGPT Reliability Through Multilingual Inquiries](https://mlsanity.com/paper/2312.10524/) (2023)
- [Coverage-Guided Tensor Compiler Fuzzing with Joint IR-Pass Mutation](https://mlsanity.com/paper/2202.09947/) (2022)
- [VulnLLMEval: A Framework for Evaluating Large Language Models in Software Vulnerability Detection and Patching](https://mlsanity.com/paper/2409.10756/) (2024)
- [Locus: Agentic Predicate Synthesis for Directed Fuzzing](https://mlsanity.com/paper/2508.21302/) (2025)
- [ARVO: Atlas of Reproducible Vulnerabilities for Open-Source Software](https://mlsanity.com/paper/2606.17283/) (2026)
- [LProtector: An LLM-driven Vulnerability Detection System](https://mlsanity.com/paper/2411.06493/) (2024)
- [On the Challenges of Fuzzing Techniques via Large Language Models](https://mlsanity.com/paper/2402.00350/) (2024)
- [MatchFixAgent: Language-Agnostic Autonomous Repository-Level Code Translation Validation and Repair](https://mlsanity.com/paper/2509.16187/) (2025)
- [SoK: The Pitfalls of Deep Reinforcement Learning for Cybersecurity](https://mlsanity.com/paper/2602.08690/) (2026)
- [AnyPoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detection](https://mlsanity.com/paper/2604.11950/) (2026)
- [State Selection Algorithms and Their Impact on The Performance of Stateful Network Protocol Fuzzing](https://mlsanity.com/paper/2112.15498/) (2021)
- [ARVO: Atlas of Reproducible Vulnerabilities for Open-Source Software](https://mlsanity.com/paper/2408.02153/) (2024)
- [Fuzzing: Randomness? Reasoning! Efficient Directed Fuzzing via Large Language Models](https://mlsanity.com/paper/2507.22065/) (2025)
- [SeedAIchemy: LLM-Driven Seed Corpus Generation for Fuzzing](https://mlsanity.com/paper/2511.12448/) (2025)
- [The Greatness of Science Cannot Be Planned: Agentic Auto-Research is Fuzz Testing](https://mlsanity.com/paper/2608.09855/) (2026)
- [Revisiting Neural Program Smoothing for Fuzzing](https://mlsanity.com/paper/2309.16618/) (2023)
- [LLMs are All You Need? Improving Fuzz Testing for MOJO with Large Language Models](https://mlsanity.com/paper/2510.10179/) (2025)
- [Fuzzwise: Intelligent Initial Corpus Generation for Fuzzing](https://mlsanity.com/paper/2512.21440/) (2025)
- [SmartOracle -- An Agentic Approach to Mitigate Noise in Differential Oracles](https://mlsanity.com/paper/2601.15074/) (2026)
- [Fuzzing with Agents? Generators Are All You Need](https://mlsanity.com/paper/2604.01442/) (2026)
- [FuzzAgent: Multi-Agent System for Evolutionary Library Fuzzing](https://mlsanity.com/paper/2605.14431/) (2026)
- [Understanding Implicit Trust Errors in Core Carrier Networks through Multi-Agent Flaw Discovery and Analysis](https://mlsanity.com/paper/2607.10315/) (2026)
- [CAROL: Context-Aware Online Learning for Fuzzer Scheduling](https://mlsanity.com/paper/2609.06772/) (2026)

## Abstract (arXiv)

Fuzz testing has enjoyed great success at discovering security critical bugs in real software. Recently, researchers have devoted significant effort to devising new fuzzing techniques, strategies, and algorithms. Such new ideas are primarily evaluated experimentally so an important question is: What experimental setup is needed to produce trustworthy results? We surveyed the recent research literature and assessed the experimental evaluations carried out by 32 fuzzing papers. We found problems in every evaluation we considered. We then performed our own extensive experimental evaluation using an existing fuzzer. Our results showed that the general problems we found in existing experimental evaluations can indeed translate to actual wrong or misleading assessments. We conclude with some guidelines that we hope will help improve experimental evaluations of fuzz testing algorithms, making reported results more robust.