# SoK: Prudent Evaluation Practices for Fuzzing

arXiv: https://arxiv.org/abs/2405.10220 · HTML: https://mlsanity.com/paper/2405.10220/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 May 2024.

## Built on by (most cited first, 8 of 8)

- [ARVO: Atlas of Reproducible Vulnerabilities for Open-Source Software](https://mlsanity.com/paper/2606.17283/) (2026)
- [SoK: The Pitfalls of Deep Reinforcement Learning for Cybersecurity](https://mlsanity.com/paper/2602.08690/) (2026)
- [ARVO: Atlas of Reproducible Vulnerabilities for Open-Source Software](https://mlsanity.com/paper/2408.02153/) (2024)
- [SeedAIchemy: LLM-Driven Seed Corpus Generation for Fuzzing](https://mlsanity.com/paper/2511.12448/) (2025)
- [Direction for Detection: A Survey of Automated Vulnerability Detection and all of its Pain Points](https://mlsanity.com/paper/2412.11194/) (2024)
- [Orion: Fuzzing Workflow Automation](https://mlsanity.com/paper/2509.15195/) (2025)
- [Clutch Control: An Attention-based Combinatorial Bandit for Efficient Mutation in JavaScript Engine Fuzzing](https://mlsanity.com/paper/2510.12732/) (2025)
- [CAROL: Context-Aware Online Learning for Fuzzer Scheduling](https://mlsanity.com/paper/2609.06772/) (2026)

## Abstract (arXiv)

Fuzzing has proven to be a highly effective approach to uncover software bugs over the past decade. After AFL popularized the groundbreaking concept of lightweight coverage feedback, the field of fuzzing has seen a vast amount of scientific work proposing new techniques, improving methodological aspects of existing strategies, or porting existing methods to new domains. All such work must demonstrate its merit by showing its applicability to a problem, measuring its performance, and often showing its superiority over existing works in a thorough, empirical evaluation. Yet, fuzzing is highly sensitive to its target, environment, and circumstances, e.g., randomness in the testing process. After all, relying on randomness is one of the core principles of fuzzing, governing many aspects of a fuzzer's behavior. Combined with the often highly difficult to control environment, the reproducibility of experiments is a crucial concern and requires a prudent evaluation setup. To address these threats to validity, several works, most notably Evaluating Fuzz Testing by Klees et al., have outlined how a carefully designed evaluation setup should be implemented, but it remains unknown to what extent their recommendations have been adopted in practice. In this work, we systematically analyze the evaluation of 150 fuzzing papers published at the top venues between 2018 and 2023. We study how existing guidelines are implemented and observe potential shortcomings and pitfalls. We find a surprising disregard of the existing guidelines regarding statistical tests and systematic errors in fuzzing evaluations. For example, when investigating reported bugs, ...