mlsanityData to 14 Oct 2026

SastBench: A Benchmark for Testing Agentic SAST Triage

arXiv 2601.02941 ↗First posted 6 Jan 2026cs.CR cs.AI cs.CLcited by 3

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

SAST (Static Application Security Testing) tools are among the most widely used techniques in defensive cybersecurity, employed by commercial and non-commercial organizations to identify potential vulnerabilities in software. Despite their great utility, they generate numerous false positives, requiring costly manual filtering (aka triage). While LLM-powered agents show promise for automating cybersecurity tasks, existing benchmarks fail to emulate real-world SAST finding distributions. We introduce SastBench, a benchmark for evaluating SAST triage agents that combines real CVEs as true positives with filtered SAST tool findings as approximate false positives. SastBench features an agent-agnostic design. We evaluate different agents on the benchmark and present a comparative analysis of their performance, provide a detailed analysis of the dataset, and discuss the implications for future development.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 18 of its 44 references that are in the map. First posted on arXiv 6 Jan 2026.

Builds on

18 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Benchmarking LLMs and LLM-based Agents in Practical Vulnerability Detection for Code Repositories5 Mar

Other references · 17

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  3. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  4. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  5. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  6. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  7. 2019Devign: Effective Vulnerability Identification by Learning Comprehensive Program Semantics via Graph Neural Networkscited by 272
  8. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  9. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 158
  10. 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 155
  11. 2024Vulnerability Detection with Code Language Models: How Far Are We?cited by 139
  12. 2018Automated Vulnerability Detection in Source Code Using Deep Representation Learningcited by 114
5 more
  1. 2021CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Softwarecited by 91
  2. 2021D2A: A Dataset Built for AI-Based Vulnerability Detection Methods Using Differential Analysiscited by 49
  3. 2024ReposVul: A Repository-Level High-Quality Vulnerability Datasetcited by 36
  4. 2024VulEval: Towards Repository-Level Evaluation of Software Vulnerability Detectioncited by 19
  5. 2025CASTLE: Benchmarking Dataset for Static Code Analyzers and LLMs towards CWE Detectioncited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Seclens: Role-specific Evaluation of LLM's for security vulnerablity detectioncited by 0
  2. 2026The History Is the Detector: Executing CVE Patch History, End-to-Endcited by 0
  3. 2026Contextualization of Third-Party Cloud Security Findingscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Argus: Reorchestrating Static Analysis via a Multi-Agent Ensemble for Full-Chain Security Vulnerability Detection
  2. 2026Can Open-Source LLM Agents Replace Static Application Security Testing Tools? An Empirical Assessment
  3. 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks
  4. 2026QASecClaw: A Multi-Agent LLM Approach for False Positive Reduction in Static Application Security Testing
  5. 2026Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps
  6. 2026TitanCA: Lessons from Orchestrating LLM Agents to Discover 100+ CVEs
  7. 2025CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities
  8. 2026CWE-bench: Refocusing Agentic Security from Exploitation to Defense