# An Extensive Comparison of Static Application Security Testing Tools

arXiv: https://arxiv.org/abs/2403.09219 · HTML: https://mlsanity.com/paper/2403.09219/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Mar 2024.

## Built on by (most cited first, 10 of 10)

- [SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code](https://mlsanity.com/paper/2506.05692/) (2025)
- [Beyond Words: On Large Language Models Actionability in Mission-Critical Risk Analysis](https://mlsanity.com/paper/2406.10273/) (2024)
- [6GSoft: Software for Edge-to-Cloud Continuum](https://mlsanity.com/paper/2407.05963/) (2024)
- [REDO: Execution-Free Runtime Error Detection for COding Agents](https://mlsanity.com/paper/2410.09117/) (2024)
- [A Call for Critically Rethinking and Reforming Data Analysis in Empirical Software Engineering](https://mlsanity.com/paper/2501.12728/) (2025)
- [SQuaD: The Software Quality Dataset](https://mlsanity.com/paper/2511.11265/) (2025)
- [Classi|Qrangle Towards a Translation Framework To Bridge The Classical-Quantum Programming Gap](https://mlsanity.com/paper/2406.06764/) (2024)
- [Descriptor: C++ Self-Admitted Technical Debt Dataset (CppSATD)](https://mlsanity.com/paper/2505.01136/) (2025)
- [On the Effectiveness of Instruction-Tuning Local LLMs for Identifying Software Vulnerabilities](https://mlsanity.com/paper/2512.20062/) (2025)
- [The Invisible Hand of AI Libraries Shaping Open Source Projects and Communities](https://mlsanity.com/paper/2601.01944/) (2026)

## Abstract (arXiv)

Context: Static Application Security Testing Tools (SASTTs) identify software vulnerabilities to support the security and reliability of software applications. Interestingly, several studies have suggested that alternative solutions may be more effective than SASTTs due to their tendency to generate false alarms, commonly referred to as low Precision. Aim: We aim to comprehensively evaluate SASTTs, setting a reliable benchmark for assessing and finding gaps in vulnerability identification mechanisms based on SASTTs or alternatives. Method: Our SASTTs evaluation is based on a controlled, though synthetic, Java codebase. It involves an assessment of 1.5 million test executions, and it features innovative methodological features such as effort-aware accuracy metrics and method-level analysis. Results: Our findings reveal that SASTTs detect a tiny range of vulnerabilities. In contrast to prevailing wisdom, SASTTs exhibit high Precision while falling short in Recall. Conclusions: The paper suggests that enhancing Recall, alongside expanding the spectrum of detected vulnerability types, should be the primary focus for improving SASTTs or alternative approaches, such as machine learning-based vulnerability identification solutions.