mlsanityData to 7 Oct 2026

Evaluating Fuzz Testing

arXiv 1808.09700 ↗First posted 29 Aug 2018cs.CRcited by 29

Not on the map. Why?

Abstract (arXiv)

Fuzz testing has enjoyed great success at discovering security critical bugs in real software. Recently, researchers have devoted significant effort to devising new fuzzing techniques, strategies, and algorithms. Such new ideas are primarily evaluated experimentally so an important question is: What experimental setup is needed to produce trustworthy results? We surveyed the recent research literature and assessed the experimental evaluations carried out by 32 fuzzing papers. We found problems in every evaluation we considered. We then performed our own extensive experimental evaluation using an existing fuzzer. Our results showed that the general problems we found in existing experimental evaluations can indeed translate to actual wrong or misleading assessments. We conclude with some guidelines that we hope will help improve experimental evaluations of fuzz testing algorithms, making reported results more robust.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Aug 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 1; 2023: 3; 2024: 6; 2025: 8; 2026: 9.

29 papers.

Sort:

By area:

  1. 2023Fuzz4All: Universal Fuzzing with Large Language Modelscited by 88
  2. 2025CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scalecited by 71
  3. 2023WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Modelscited by 37
  4. 2024PAPILLON: Efficient and Stealthy Fuzz Testing-Powered Jailbreaks for LLMscited by 20
  5. 2024LLAMAFUZZ: Large Language Model Enhanced Greybox Fuzzingcited by 14
  6. 2025Constrained Sampling for Language Models Should Be Easy: An MCMC Perspectivecited by 14
  7. 2021MDPFuzz: Testing Models Solving Markov Decision Processescited by 11
  8. 2022Coverage-Guided Tensor Compiler Fuzzing with Joint IR-Pass Mutationcited by 9
  9. 2024VulnLLMEval: A Framework for Evaluating Large Language Models in Software Vulnerability Detection and Patchingcited by 9
  10. 2024LProtector: An LLM-driven Vulnerability Detection Systemcited by 8
  11. 2025Locus: Agentic Predicate Synthesis for Directed Fuzzingcited by 8
  12. 2026ARVO: Atlas of Reproducible Vulnerabilities for Open-Source Softwarecited by 8
17 more
  1. 2024On the Challenges of Fuzzing Techniques via Large Language Modelscited by 4
  2. 2026SoK: The Pitfalls of Deep Reinforcement Learning for Cybersecuritycited by 4
  3. 2026AnyPoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detectioncited by 4
  4. 2021State Selection Algorithms and Their Impact on The Performance of Stateful Network Protocol Fuzzingcited by 3
  5. 2025MatchFixAgent: Language-Agnostic Autonomous Repository-Level Code Translation Validation and Repaircited by 3
  6. 2025Fuzzing: Randomness? Reasoning! Efficient Directed Fuzzing via Large Language Modelscited by 2
  7. 2024ARVO: Atlas of Reproducible Vulnerabilities for Open-Source Softwarecited by 1
  8. 2025SeedAIchemy: LLM-Driven Seed Corpus Generation for Fuzzingcited by 1
  9. 2026The Greatness of Science Cannot Be Planned: Agentic Auto-Research is Fuzz Testingcited by 1
  10. 2023Revisiting Neural Program Smoothing for Fuzzingcited by 0
  11. 2025LLMs are All You Need? Improving Fuzz Testing for MOJO with Large Language Modelscited by 0
  12. 2025Fuzzwise: Intelligent Initial Corpus Generation for Fuzzingcited by 0
  13. 2026SmartOracle -- An Agentic Approach to Mitigate Noise in Differential Oraclescited by 0
  14. 2026Fuzzing with Agents? Generators Are All You Needcited by 0
  15. 2026FuzzAgent: Multi-Agent System for Evolutionary Library Fuzzingcited by 0
  16. 2026Understanding Implicit Trust Errors in Core Carrier Networks through Multi-Agent Flaw Discovery and Analysiscited by 0
  17. 2026CAROL: Context-Aware Online Learning for Fuzzer Schedulingcited by 0