mlsanityData to 14 Oct 2026

Last Secure Code Benchmark: Is the Code You Vibe Secure?

arXiv 2610.14766 ↗First posted 10 Oct 2026cs.CR cs.AIcited by 0

New in LLM agents and tool use; no direction yet. LLM agents and tool use

Abstract (arXiv)

AI coding agents now write much of the code that ships, yet whether that code is secure remains unclear. We introduce the Last Secure Code Benchmark, a comprehensive evaluation of secure code generation by coding agents. Built from 150 real vulnerabilities, its 450 tasks span seven widely used languages (Java, C, C++, JavaScript, Python, PHP, and Go), 24 CWEs including 21 of the 2025 CWE Top 25 and 8 of the Top 10 KEV weaknesses, more than any benchmark we compare with, and three vibe-coding scenarios: filling in a function, writing a file, and building a repository's source tree from scratch. Every task asks only for functionality, through a specification that never mentions security, and hidden functional and security tests grade the result. Across six frontier agents, 73.6% to 91.3% of tasks pass functionally, yet even the strongest, Claude Fable 5, produces code that is both functional and secure for only 25.1% of tasks, and the others for 16.4% to 24.7%. The best model only passes 12% of the vulnerability scenarios across all vibe coding setting, posing strong concern on the security of the code written by vibe coding.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 36 references that are in the map. First posted on arXiv 10 Oct 2026.

Builds on

26 of 36 references.

References · 26

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  5. 2021Measuring Coding Challenge Competence With APPScited by 902
  6. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 431
  7. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 336
  8. 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 155
  9. 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 138
  10. 2025SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue Resolutioncited by 90
  11. 2025CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scalecited by 78
  12. 2024Instruction Tuning for Secure Code Generationcited by 68
14 more
  1. 2025NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agentscited by 65
  2. 2025CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generationcited by 59
  3. 2023CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Modelscited by 55
  4. 2025BaxBench: Can LLMs Generate Correct and Secure Backends?cited by 41
  5. 2023SALLM: Security Assessment of Generated Codecited by 39
  6. 2025Vibe Coding in Practice: Motivations, Challenges, and a Future Outlook -- a Grey Literature Reviewcited by 30
  7. 2025Vibe coding: programming through conversation with artificial intelligencecited by 29
  8. 2026ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?cited by 29
  9. 2024ProSec: Fortifying Code LLMs with Proactive Security Alignmentcited by 25
  10. 2025Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Taskscited by 22
  11. 2025PurpCode: Reasoning for Safer Code Generationcited by 18
  12. 2025A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Codecited by 14
  13. 2026CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerabilitycited by 9
  14. 2026Secure Code Generation via Online Reinforcement Learning with Vulnerability Reward Modelcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios
  2. 2025SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositories
  3. 2026Beyond Pass@k: Measuring Reliability and Security of Agentic Code Generation
  4. 2024RedCode: Risky Code Execution and Generation Benchmark for Code Agents
  5. 2026RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories
  6. 2026SoK: AI Secure Code Generation: Progress, Pitfalls, and Paths Forward
  7. 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks
  8. 2026AuraForge: Scaling Security Supervision for Training Coding Agents