# Sifting the Noise: A Comparative Study of LLM Agents in Vulnerability False Positive Filtering arXiv: https://arxiv.org/abs/2601.22952 ยท HTML: https://mlsanity.com/paper/2601.22952/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. - Builds on 20 of its 64 references that are in the map. - First posted on arXiv 30 Jan 2026. ## Placed in - LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/) ## Builds on (most cited first, 20 of 20) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024) - [OpenHands: An Open Platform for AI Software Developers as Generalist Agents](https://mlsanity.com/paper/2407.16741/) (2024) - [RepairAgent: An Autonomous, LLM-Based Agent for Program Repair](https://mlsanity.com/paper/2403.17134/) (2024) - [IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities](https://mlsanity.com/paper/2405.17238/) (2024) - [Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilities](https://mlsanity.com/paper/2311.16169/) (2023) - [D2A: A Dataset Built for AI-Based Vulnerability Detection Methods Using Differential Analysis](https://mlsanity.com/paper/2102.07995/) (2021) - [Large Language Model for Vulnerability Detection: Emerging Results and Future Directions](https://mlsanity.com/paper/2401.15468/) (2024) - [A Manually-Curated Dataset of Fixes to Vulnerabilities of Open-Source Software](https://mlsanity.com/paper/1902.02595/) (2019) - [Detecting False Alarms from Automatic Static Analysis Tools: How Far are We?](https://mlsanity.com/paper/2202.05982/) (2022) - [Learning to Reduce False Positives in Analytic Bug Detectors](https://mlsanity.com/paper/2203.09907/) (2022) - [On the Effectiveness of Function-Level Vulnerability Detectors for Inter-Procedural Vulnerabilities](https://mlsanity.com/paper/2401.09767/) (2024) - [A Critical Comparison on Six Static Analysis Tools: Detection, Agreement, and Precision](https://mlsanity.com/paper/2101.08832/) (2021) - [VERCATION: Precise Vulnerable Open-source Software Version Identification based on Static Analysis and LLM](https://mlsanity.com/paper/2408.07321/) (2024) - [Out of Distribution, Out of Luck: How Well Can LLMs Trained on Vulnerability Datasets Detect Top 25 CWE Weaknesses?](https://mlsanity.com/paper/2507.21817/) (2025) - [Large Language Models Versus Static Code Analysis Tools: A Systematic Benchmark for Vulnerability Detection](https://mlsanity.com/paper/2508.04448/) (2025) - [An LLM-based multi-agent framework for agile effort estimation](https://mlsanity.com/paper/2509.14483/) (2025) - [Software Security Patch Management -- A Systematic Literature Review of Challenges, Approaches, Tools and Practices](https://mlsanity.com/paper/2012.00544/) (2020) - [Validating Static Warnings via Testing Code Fragments](https://mlsanity.com/paper/2106.04735/) (2021) - [Fixseeker: An Empirical Driven Graph-based Approach for Detecting Silent Vulnerability Fixes in Open Source Software](https://mlsanity.com/paper/2503.20265/) (2025) ## Built on by (most cited first, 5 of 5) - [DREA: Decoupled Reasoning and Exploration Agents for Repository-Level Vulnerability Detection](https://mlsanity.com/paper/2607.13439/) (2026) - [Refute-or-Promote: An Adversarial Stage-Gated Multi-Agent Review Methodology for High-Precision LLM-Assisted Defect Discovery](https://mlsanity.com/paper/2604.19049/) (2026) - [Are Frontier LLMs Ready for Cybersecurity? Evidence for Vertical Foundation Models from Dual-Mode Vulnerability Benchmarks](https://mlsanity.com/paper/2605.23243/) (2026) - [ALIBI: Adaptive Agentic Attacks on LLM-Based Vulnerability Detectors via Adversarial Code Comments](https://mlsanity.com/paper/2607.24964/) (2026) - [The History Is the Detector: Executing CVE Patch History, End-to-End](https://mlsanity.com/paper/2609.05335/) (2026) ## Abstract (arXiv) In this paper, we present a comparative study of three state-of-the-art LLM-based agent frameworks, i.e., Aider, OpenHands, and SWE-agent, for vulnerability FP filtering. We evaluate these frameworks using the vulnerabilities from the OWASP Benchmark and real-world open-source Java projects. We further conduct a focused post-cutoff C/C++ study using the strongest configuration to test contamination-free generalization and isolate key agentic capabilities. The experimental results show that LLM-based agents can remove the majority of SAST noise, reducing an initial FP detection rate of over 92% on the OWASP Benchmark to as low as 6.3% in the best configuration. On a real-world Java dataset, the best configuration of LLM-based agents can achieve an FP identification rate of up to 93.3% involving CodeQL alerts. However, the benefits of agents are strongly backbone- and CWE-dependent: agentic frameworks significantly outperform vanilla prompting for stronger models such as Claude Sonnet 4 and GPT-5, but yield limited or inconsistent gains for weaker backbones. On the post-cutoff OSS-Fuzz dataset, SWE-agent with Claude Sonnet 4 identifies 95.5% of FPs while maintaining 95.5% precision, compared with a 36.4% FP identification rate for vanilla prompting. Moreover, aggressive FP reduction can come at the cost of suppressing true vulnerabilities, highlighting important trade-offs. Finally, we observe large disparities in computational cost across agent frameworks. Overall, our study demonstrates that LLM-based agents are a powerful but non-uniform solution for SAST FP filtering, and that their practical deployment requires careful consideration of agent design, backbone model choice, vulnerability category, and operational cost.