mlsanityData to 14 Oct 2026

Can Open-Source LLM Agents Replace Static Application Security Testing Tools? An Empirical Assessment

arXiv 2606.11672 ↗First posted 10 Jun 2026cs.CR cs.AIcited by 0

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

This paper explores the value of agentic AI tools for cybersecurity purposes. We evaluate the efficacy of a general-purpose GenAI Large Language Model- (GenAI-) based agent when powered by three different Ollama-hosted general-purpose open source models. We assess each agent's performance using precision, recall, false positive count, and a calculated composite score based upon the interplay of the captured metrics, against the baseline performance of an existing, vetted Static Application Security Testing (SAST) tool, Bandit. Our findings refute the notion that a modern open-source GenAI LLM-based agent is currently suitable for the specialized task of SAST scanning under realistic conditions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 12 of its 49 references that are in the map. First posted on arXiv 10 Jun 2026.

Builds on

12 of 49 references.

References · 12

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024Large Language Models for Cyber Security: A Systematic Literature Reviewcited by 68
  3. 2025LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insightscited by 47
  4. 2024Prompting Techniques for Secure Code Generation: A Systematic Investigationcited by 34
  5. 2023Navigating the Complexity of Generative AI Adoption in Software Engineeringcited by 32
  6. 2024Using AI Assistants in Software Development: A Qualitative Study on Security Practices and Concernscited by 28
  7. 2025KNighter: Transforming Static Analysis with LLM-Synthesized Checkerscited by 24
  8. 2024The Future of AI-Driven Software Engineeringcited by 23
  9. 2025On the Use of Agentic Coding Manifests: An Empirical Study of Claude Codecited by 14
  10. 2025Decoding the Configuration of AI Coding Agents: Insights from Claude Code Projectscited by 14
  11. 2024Software Security Analysis in 2030 and Beyond: A Research Roadmapcited by 5
  12. 2025Imagining Design Workflows in Agentic AI Futurescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Argus: Reorchestrating Static Analysis via a Multi-Agent Ensemble for Full-Chain Security Vulnerability Detection
  2. 2024HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing
  3. 2025From LLMs to Agents: A Comparative Evaluation of LLMs and LLM-based Agents in Security Patch Detection
  4. 2026Tangent: An Empirical Study of Testing Practices for LLM-Based Agent Applications
  5. 2025AutoPentest: Enhancing Vulnerability Management With Autonomous LLM Agents
  6. 2026A Large-Scale Empirical Study of Quality Assurance Practices and Gaps in AI Agents
  7. 2025Automated Penetration Testing with LLM Agents and Classical Planning
  8. 2026LLM-Based Agents for Software and Systems Security: Approaches, Applications, and Assessment