mlsanityData to 14 Oct 2026

Are Frontier LLMs Ready for Cybersecurity? Evidence for Vertical Foundation Models from Dual-Mode Vulnerability Benchmarks

arXiv 2605.23243 ↗First posted 22 May 2026cs.CR cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We evaluate whether frontier LLMs are ready for cybersecurity through a dual-mode benchmark: white-box function-level vulnerability detection (VulnLLM-R, across C/Java/Python) and black-box web application security testing (five production-style applications with 118 ground-truth vulnerabilities across 20+ CWE families, which we will open-source). We test six frontier models (GPT-5.4, Codex~5.3, Claude Opus~4.7, Sonnet~4.6, Gemini~3.1~Pro and Gemini~3~Flash) and two domain-specialized models across four testing paradigms. Our findings are sobering: (1)~every frontier model produces 10-50% false positive rates in white-box detection, systematically over-predicting vulnerabilities; (2)~in black-box testing, frontier models achieve only 4-8% ground-truth coverage, improving to just 10-19% even with external security tools (Playwright MCP, Burp Suite MCP); (3)~structured penetration-testing methodology encoded in domain-specialized agents raises per-family detection above 50%, demonstrating that methodology, not scale, is the primary lever; and (4)~a domain-specialized defense model achieves the highest precision (0.904) and lowest false positive rate (9.7%) among all models, on a single GPU. We identify the absence of structured security testing traces end-to-end request/response sequences, failure-heavy data, and multi-step attack chains as the fundamental training data bottleneck, and propose self-play security testing as a data generation strategy. Our results make the case for vertical foundation models purpose-built for cybersecurity.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 29 references that are in the map. First posted on arXiv 22 May 2026.

Builds on

18 of 29 references.

References · 18

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2023BloombergGPT: A Large Language Model for Financecited by 939
  3. 2023Large Language Models for Software Engineering: Survey and Open Problemscited by 181
  4. 2024Vulnerability Detection with Code Language Models: How Far Are We?cited by 139
  5. 2023PentestGPT: An LLM-empowered Automatic Penetration Testing Toolcited by 97
  6. 2025CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilitiescited by 96
  7. 2024NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Securitycited by 93
  8. 2024CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Modelscited by 89
  9. 2024HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testingcited by 37
  10. 2025Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testingcited by 15
  11. 2025Chasing Shadows: Pitfalls in LLM Security Researchcited by 13
  12. 2025Cybersecurity AI Benchmark (CAIBench): A Meta-Benchmark for Evaluating Cybersecurity AI Agentscited by 12
6 more
  1. 2025VulnLLM-R: Specialized Reasoning LLM with Agent Scaffold for Vulnerability Detectioncited by 11
  2. 2026Sifting the Noise: A Comparative Study of LLM Agents in Vulnerability False Positive Filteringcited by 5
  3. 2025LLM-based Vulnerability Discovery through the Lens of Code Metricscited by 4
  4. 2025LLM Agents for Automated Web Vulnerability Reproduction: Are We There Yet?cited by 4
  5. 2026ZeroDayBench: Evaluating LLM Agents on Unseen Zero-Day Vulnerabilities for Cyberdefensecited by 4
  6. 2026Measuring and Exploiting Contextual Bias in LLM-Assisted Security Code Reviewcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026TitanCA: Lessons from Orchestrating LLM Agents to Discover 100+ CVEs
  2. 2025Analysing Safety Risks in LLMs Fine-Tuned with Pseudo-Malicious Cyber Security Data
  3. 2025Estimating Worst-Case Frontier Risks of Open-Weight LLMs
  4. 2026Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks
  5. 2025Towards Effective Complementary Security Analysis using Large Language Models
  6. 2026Internal Safety Collapse in Frontier Large Language Models
  7. 2025Quantifying CBRN Risk in Frontier Models
  8. 2025Out of Distribution, Out of Luck: How Well Can LLMs Trained on Vulnerability Datasets Detect Top 25 CWE Weaknesses?