# Magma: A Ground-Truth Fuzzing Benchmark

arXiv: https://arxiv.org/abs/2009.01120 · HTML: https://mlsanity.com/paper/2009.01120/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Sep 2020.

## Built on by (most cited first, 16 of 16)

- [How Far Have We Gone in Vulnerability Detection Using Large Language Models](https://mlsanity.com/paper/2311.12420/) (2023)
- [LLAMAFUZZ: Large Language Model Enhanced Greybox Fuzzing](https://mlsanity.com/paper/2406.07714/) (2024)
- [Locus: Agentic Predicate Synthesis for Directed Fuzzing](https://mlsanity.com/paper/2508.21302/) (2025)
- [Large Language Models for Security Operations Centers: A Comprehensive Survey](https://mlsanity.com/paper/2509.10858/) (2025)
- [ARVO: Atlas of Reproducible Vulnerabilities for Open-Source Software](https://mlsanity.com/paper/2606.17283/) (2026)
- [Vulnerability Detection: From Formal Verification to Large Language Models and Hybrid Approaches: A Comprehensive Overview](https://mlsanity.com/paper/2503.10784/) (2025)
- [State Selection Algorithms and Their Impact on The Performance of Stateful Network Protocol Fuzzing](https://mlsanity.com/paper/2112.15498/) (2021)
- [Adversarial Attacks against Binary Similarity Systems](https://mlsanity.com/paper/2303.11143/) (2023)
- [FuzzTheREST: An Intelligent Automated Black-box RESTful API Fuzzer](https://mlsanity.com/paper/2407.14361/) (2024)
- [ARVO: Atlas of Reproducible Vulnerabilities for Open-Source Software](https://mlsanity.com/paper/2408.02153/) (2024)
- [Cybersecurity is the True Frontier for Generative AI Success or Failure](https://mlsanity.com/paper/2606.28929/) (2026)
- [SLOPT: Bandit Optimization Framework for Mutation-Based Fuzzing](https://mlsanity.com/paper/2211.03285/) (2022)
- [Another Generic Setting for Entity Resolution: Basic Theory](https://mlsanity.com/paper/2303.06629/) (2023)
- [ANVIL: Anomaly-based Vulnerability Identification without Labelled Training Data](https://mlsanity.com/paper/2408.16028/) (2024)
- [C2RUST-BENCH: A Minimized, Representative Dataset for C-to-Rust Transpilation Evaluation](https://mlsanity.com/paper/2504.15144/) (2025)
- [SeedAIchemy: LLM-Driven Seed Corpus Generation for Fuzzing](https://mlsanity.com/paper/2511.12448/) (2025)

## Abstract (arXiv)

High scalability and low running costs have made fuzz testing the de facto standard for discovering software bugs. Fuzzing techniques are constantly being improved in a race to build the ultimate bug-finding tool. However, while fuzzing excels at finding bugs in the wild, evaluating and comparing fuzzer performance is challenging due to the lack of metrics and benchmarks. For example, crash count, perhaps the most commonly-used performance metric, is inaccurate due to imperfections in deduplication techniques. Additionally, the lack of a unified set of targets results in ad hoc evaluations that hinder fair comparison. We tackle these problems by developing Magma, a ground-truth fuzzing benchmark that enables uniform fuzzer evaluation and comparison. By introducing real bugs into real software, Magma allows for the realistic evaluation of fuzzers against a broad set of targets. By instrumenting these bugs, Magma also enables the collection of bug-centric performance metrics independent of the fuzzer. Magma is an open benchmark consisting of seven targets that perform a variety of input manipulations and complex computations, presenting a challenge to state-of-the-art fuzzers. We evaluate seven widely-used mutation-based fuzzers (AFL, AFLFast, AFL++, FairFuzz, MOpt-AFL, honggfuzz, and SymCC-AFL) against Magma over 200,000 CPU-hours. Based on the number of bugs reached, triggered, and detected, we draw conclusions about the fuzzers' exploration and detection capabilities. This provides insight into fuzzer performance evaluation, highlighting the importance of ground truth in performing more accurate and meaningful evaluations.