# A Critical Comparison on Six Static Analysis Tools: Detection, Agreement, and Precision

arXiv: https://arxiv.org/abs/2101.08832 · HTML: https://mlsanity.com/paper/2101.08832/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jan 2021.

## Built on by (most cited first, 9 of 9)

- [Security and Quality in LLM-Generated Code: A Multi-Language, Multi-Model Analysis](https://mlsanity.com/paper/2502.01853/) (2025)
- [Combining Large Language Models with Static Analyzers for Code Review Generation](https://mlsanity.com/paper/2502.06633/) (2025)
- [How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench](https://mlsanity.com/paper/2507.02976/) (2025)
- [Sifting the Noise: A Comparative Study of LLM Agents in Vulnerability False Positive Filtering](https://mlsanity.com/paper/2601.22952/) (2026)
- [Follow Your Nose -- Which Code Smells are Worth Chasing?](https://mlsanity.com/paper/2103.01861/) (2021)
- [Do Prompt Patterns Affect Code Quality? A First Empirical Assessment of ChatGPT-Generated Code](https://mlsanity.com/paper/2504.13656/) (2025)
- [What Were You Thinking? An LLM-Driven Large-Scale Study of Refactoring Motivations in Open-Source Projects](https://mlsanity.com/paper/2509.07763/) (2025)
- [RefactorAssist: Agentic Refinement for Reliable Code Refactoring](https://mlsanity.com/paper/2608.00924/) (2026)
- [On the Effectiveness of Instruction-Tuning Local LLMs for Identifying Software Vulnerabilities](https://mlsanity.com/paper/2512.20062/) (2025)

## Abstract (arXiv)

Background. Developers use Automated Static Analysis Tools (ASATs) to control for potential quality issues in source code, including defects and technical debt. Tool vendors have devised quite a number of tools, which makes it harder for practitioners to select the most suitable one for their needs. To better support developers, researchers have been conducting several studies on ASATs to favor the understanding of their actual capabilities. Aims. Despite the work done so far, there is still a lack of knowledge regarding (1) which source quality problems can actually be detected by static analysis tool warnings, (2) what is their agreement, and (3) what is the precision of their recommendations. We aim at bridging this gap by proposing a large-scale comparison of six popular static analysis tools for Java projects: Better Code Hub, CheckStyle, Coverity Scan, Findbugs, PMD, and SonarQube. Method. We analyze 47 Java projects and derive a taxonomy of warnings raised by 6 state-of-the-practice ASATs. To assess their agreement, we compared them by manually analyzing - at line-level - whether they identify the same issues. Finally, we manually evaluate the precision of the tools. Results. The key results report a comprehensive taxonomy of ASATs warnings, show little to no agreement among the tools and a low degree of precision. Conclusions. We provide a taxonomy that can be useful to researchers, practitioners, and tool vendors to map the current capabilities of the tools. Furthermore, our study provides the first overview on the agreement among different tools as well as an extensive analysis of their precision.