# Reducing False Positives in Static Bug Detection with LLMs: An Empirical Study in Industry

arXiv: https://arxiv.org/abs/2601.18844 · HTML: https://mlsanity.com/paper/2601.18844/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 60 references that are in the map.
- First posted on arXiv 26 Jan 2026.

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Long-context LLMs Struggle with Long In-context Learning](https://mlsanity.com/paper/2404.02060/) (2024)
- [Vulnerability Detection with Code Language Models: How Far Are We?](https://mlsanity.com/paper/2403.18624/) (2024)
- [Make Your LLM Fully Utilize the Context](https://mlsanity.com/paper/2404.16811/) (2024)
- [IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities](https://mlsanity.com/paper/2405.17238/) (2024)
- [Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAG](https://mlsanity.com/paper/2406.11147/) (2024)
- [Large Language Models for Test-Free Fault Localization](https://mlsanity.com/paper/2310.01726/) (2023)
- [Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilities](https://mlsanity.com/paper/2311.16169/) (2023)
- [RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing](https://mlsanity.com/paper/2501.18160/) (2025)
- [LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights](https://mlsanity.com/paper/2502.07049/) (2025)
- [Large Language Model for Vulnerability Detection: Emerging Results and Future Directions](https://mlsanity.com/paper/2401.15468/) (2024)
- [Large Language Model for Vulnerability Detection and Repair: Literature Review and the Road Ahead](https://mlsanity.com/paper/2404.02525/) (2024)
- [ReGVD: Revisiting Graph Neural Networks for Vulnerability Detection](https://mlsanity.com/paper/2110.07317/) (2021)
- [Benchmarking LLMs and LLM-based Agents in Practical Vulnerability Detection for Code Repositories](https://mlsanity.com/paper/2503.03586/) (2025)
- [How Far Have We Gone in Vulnerability Detection Using Large Language Models](https://mlsanity.com/paper/2311.12420/) (2023)
- [LLMDFA: Analyzing Dataflow in Code with Large Language Models](https://mlsanity.com/paper/2402.10754/) (2024)
- [Learning to Reduce False Positives in Analytic Bug Detectors](https://mlsanity.com/paper/2203.09907/) (2022)
- ["False negative -- that one is going to kill you": Understanding Industry Perspectives of Static Analysis based Security Testing](https://mlsanity.com/paper/2307.16325/) (2023)
- [Exploring the Limits of ChatGPT in Software Security Applications](https://mlsanity.com/paper/2312.05275/) (2023)
- [Revisiting the Performance of Deep Learning-Based Vulnerability Detection on Realistic Datasets](https://mlsanity.com/paper/2407.03093/) (2024)
- [Minimizing False Positives in Static Bug Detection via LLM-Enhanced Path Feasibility Analysis](https://mlsanity.com/paper/2506.10322/) (2025)
- [Conquering the Extensional Scalability Problem for Value-Flow Analysis Frameworks](https://mlsanity.com/paper/1912.06878/) (2019)

## Built on by (most cited first, 5 of 5)

- [AnyPoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detection](https://mlsanity.com/paper/2604.11950/) (2026)
- [Learning Globally Reusable Skills for Coding Agents](https://mlsanity.com/paper/2608.06153/) (2026)
- [Refute-or-Promote: An Adversarial Stage-Gated Multi-Agent Review Methodology for High-Precision LLM-Assisted Defect Discovery](https://mlsanity.com/paper/2604.19049/) (2026)
- [Activating Latent Security Knowledge through LLM-Guided Risk Analysis for Secure Code Generation](https://mlsanity.com/paper/2606.16244/) (2026)
- [ALIBI: Adaptive Agentic Attacks on LLM-Based Vulnerability Detectors via Adversarial Code Comments](https://mlsanity.com/paper/2607.24964/) (2026)

## Abstract (arXiv)

Static analysis tools (SATs) are widely adopted in both academia and industry for improving software quality, yet their practical use is often hindered by high false positive rates, especially in large-scale enterprise systems. These false alarms demand substantial manual inspection, creating severe inefficiencies in industrial code review. While recent work has demonstrated the potential of large language models (LLMs) for false alarm reduction on open-source benchmarks, their effectiveness in real-world enterprise settings remains unclear. To bridge this gap, we conduct the first comprehensive empirical study of diverse LLM-based false alarm reduction techniques in an industrial context at Tencent, one of the largest IT companies in China. Using data from Tencent's enterprise-customized SAT on its large-scale Advertising and Marketing Services software, we construct a dataset of 433 alarms (328 false positives, 105 true positives) covering three common bug types. Through interviewing developers and analyzing the data, our results highlight the prevalence of false positives, which wastes substantial manual effort (e.g., 10-20 minutes of manual inspection per alarm). Meanwhile, our results show the huge potential of LLMs for reducing false alarms in industrial settings (e.g., hybrid techniques of LLM and static analysis eliminate 94-98% of false positives with high recall). Furthermore, LLM-based techniques are cost-effective, with per-alarm costs as low as 2.1-109.5 seconds and 0.0011-0.12, representing orders-of-magnitude savings compared to manual review. Finally, our case analysis further identifies key limitations of LLM-based false alarm reduction in industrial settings.