# Detecting False Alarms from Automatic Static Analysis Tools: How Far are We?

arXiv: https://arxiv.org/abs/2202.05982 · HTML: https://mlsanity.com/paper/2202.05982/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Feb 2022.

## Built on by (most cited first, 17 of 17)

- [Can LLMs Replace Manual Annotation of Software Engineering Artifacts?](https://mlsanity.com/paper/2408.05534/) (2024)
- [Comparison of Static Application Security Testing Tools and Large Language Models for Repo-level Vulnerability Detection](https://mlsanity.com/paper/2407.16235/) (2024)
- [Black-Box Adversarial Attacks on LLM-Based Code Completion](https://mlsanity.com/paper/2408.02509/) (2024)
- [SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios](https://mlsanity.com/paper/2509.22097/) (2025)
- [Sifting the Noise: A Comparative Study of LLM Agents in Vulnerability False Positive Filtering](https://mlsanity.com/paper/2601.22952/) (2026)
- [Can LLM Prompting Serve as a Proxy for Static Analysis in Vulnerability Detection](https://mlsanity.com/paper/2412.12039/) (2024)
- [XOXO: Stealthy Cross-Origin Context Poisoning Attacks against AI Coding Assistants](https://mlsanity.com/paper/2503.14281/) (2025)
- [REDO: Execution-Free Runtime Error Detection for COding Agents](https://mlsanity.com/paper/2410.09117/) (2024)
- [How to Find Actionable Static Analysis Warnings: A Case Study with FindBugs](https://mlsanity.com/paper/2205.10504/) (2022)
- [Is Hyper-Parameter Optimization Different for Software Analytics?](https://mlsanity.com/paper/2401.09622/) (2024)
- [Argus: Reorchestrating Static Analysis via a Multi-Agent Ensemble for Full-Chain Security Vulnerability Detection](https://mlsanity.com/paper/2604.06633/) (2026)
- [Less, but Stronger: On the Value of Strong Heuristics in Semi-supervised Learning for Software Analytics](https://mlsanity.com/paper/2302.01997/) (2023)
- [Argus: A Multi-Agent Sensitive Information Leakage Detection Framework Based on Hierarchical Reference Relationships](https://mlsanity.com/paper/2512.08326/) (2025)
- [Deep Learning-based Binary Analysis for Vulnerability Detection in x86-64 Machine Code](https://mlsanity.com/paper/2601.09157/) (2026)
- [Beyond Reproducibility: Towards Security-Aware Evaluation of Research Artifacts](https://mlsanity.com/paper/2605.06508/) (2026)
- [Is Model Instability just Noise to be Tolerated or a Property that can be Managed?](https://mlsanity.com/paper/2607.10420/) (2026)
- [P2Patch: Exploit-Guided Vulnerability Repair](https://mlsanity.com/paper/2610.20245/) (2026)

## Abstract (arXiv)

Automatic static analysis tools (ASATs), such as Findbugs, have a high false alarm rate. The large number of false alarms produced poses a barrier to adoption. Researchers have proposed the use of machine learning to prune false alarms and present only actionable warnings to developers. The state-of-the-art study has identified a set of "Golden Features" based on metrics computed over the characteristics and history of the file, code, and warning. Recent studies show that machine learning using these features is extremely effective and that they achieve almost perfect performance. We perform a detailed analysis to better understand the strong performance of the "Golden Features". We found that several studies used an experimental procedure that results in data leakage and data duplication, which are subtle issues with significant implications. Firstly, the ground-truth labels have leaked into features that measure the proportion of actionable warnings in a given context. Secondly, many warnings in the testing dataset appear in the training dataset. Next, we demonstrate limitations in the warning oracle that determines the ground-truth labels, a heuristic comparing warnings in a given revision to a reference revision in the future. We show the choice of reference revision influences the warning distribution. Moreover, the heuristic produces labels that do not agree with human oracles. Hence, the strong performance of these techniques previously seen is overoptimistic of their true performance if adopted in practice. Our results convey several lessons and provide guidelines for evaluating false alarm detectors.