mlsanityData to 7 Oct 2026

SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios

arXiv 2509.22097 ↗First posted 26 Sep 2025cs.SE cs.AI cs.CLcited by 11

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Large language model-powered code agents are rapidly transforming software engineering, yet the security risks of their generated code have become a critical concern. Existing benchmarks have provided valuable insights, but they fail to capture scenarios in which vulnerabilities are actually introduced by human developers, making fair comparisons between humans and agents infeasible. We therefore introduce SecureVibeBench, a benchmark of 105 C/C++ secure coding tasks sourced from 41 projects in OSS-Fuzz for code agents. SecureVibeBench has the following features: (i) realistic task settings that require multi-file edits in large repositories, (ii)~aligned contexts based on real-world open-source vulnerabilities with precisely identified vulnerability introduction points, and (iii) comprehensive evaluation that combines functionality testing and security checking with both static and dynamic oracles. We evaluate 5 popular code agents like OpenHands, supported by 5 LLMs (e.g., Claude sonnet 4.5) on SecureVibeBench. Results show that current agents struggle to produce both correct and secure code, as even the best-performing one, produces merely 23.8% correct and secure solutions on SecureVibeBench. Our code and data are on https://github.com/iCSawyer/SecureVibeBench.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 42 of its 77 references that are in the map. First posted on arXiv 26 Sep 2025.

Builds on

42 of 77 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks13 Jun
  2. 2025Let the Trial Begin: A Mock-Court Approach to Vulnerability Detection using LLM-Based Agents16 May
  3. 2025AgentSafe: Safeguarding Large Language Model-based Multi-agent Systems via Hierarchical Data Management6 Mar
  4. 2025Benchmarking LLMs and LLM-based Agents in Practical Vulnerability Detection for Code Repositories5 Mar
  5. 2024EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities24 Sep

Other references · 37

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  3. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  4. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 722
  5. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  6. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 400
  7. 2024SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?cited by 196
  8. 2024Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Modelscited by 156
  9. 2024Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineeringcited by 146
  10. 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 130
  11. 2024SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agentscited by 129
  12. 2024SWE-Bench+: Enhanced Coding Benchmark for LLMscited by 95
25 more
  1. 2025Trae Agent: An LLM-based Agent for Software Engineering with Test-time Scalingcited by 93
  2. 2024NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Securitycited by 90
  3. 2025SWE-bench Goes Live!cited by 74
  4. 2024Instruction Tuning for Secure Code Generationcited by 63
  5. 2023LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluationscited by 58
  6. 2025FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementationcited by 57
  7. 2025CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generationcited by 55
  8. 2023CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Modelscited by 52
  9. 2025BaxBench: Can LLMs Generate Correct and Secure Backends?cited by 38
  10. 2024Constrained Decoding for Secure Code Generationcited by 35
  11. 2025Evaluating Agent-based Program Repair at Googlecited by 35
  12. 2024Prompting Techniques for Secure Code Generation: A Systematic Investigationcited by 33
  13. 2024Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detectioncited by 27
  14. 2024ProSec: Fortifying Code LLMs with Proactive Security Alignmentcited by 24
  15. 2025SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Codecited by 17
  16. 2025PurpCode: Reasoning for Safer Code Generationcited by 17
  17. 2022Detecting False Alarms from Automatic Static Analysis Tools: How Far are We?cited by 16
  18. 2025Doc2Feat-Bench: Evaluating Documentation-Driven Feature Additioncited by 15
  19. 2024HexaCoder: Secure Code Generation via Oracle-Guided Synthetic Training Datacited by 14
  20. 2025A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Codecited by 12
  21. 2023CHRONOS: Time-Aware Zero-Shot Identification of Libraries from Vulnerability Reportscited by 7
  22. 2023Evaluating SZZ Implementations: An Empirical Study on the Linux Kernelcited by 7
  23. 2024Fixing Security Vulnerabilities with AI in OSS-Fuzzcited by 5
  24. 2025Back to the Basics: Rethinking Issue-Commit Linking with LLM-Assisted Retrievalcited by 4
  25. 2025"My productivity is boosted, but ..." Demystifying Users' Perception on AI Coding Assistantscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 11.

11 papers.

  1. 2026CodeTaste: Can LLMs Generate Human-Level Code Refactorings?cited by 8
  2. 2026Towards Secure Logging: Characterizing and Benchmarking Logging Code Security Issues with LLMscited by 1
  3. 2026Refusal Evaluation in Coding LLMs and Code Agents: A Systematic Review of Thirteen Malicious-Code Prompt Corpora (2023-2025)cited by 1
  4. 2026How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Studycited by 1
  5. 2026The Balkanization of Execution-Security Research for AI Coding Agents: Isolation, Access Control, and Time-of-Check-to-Time-of-Use Vulnerabilitiescited by 0
  6. 2026The Patchwork Problem in LLM-Generated Codecited by 0
  7. 2026SWE-NFI: Studying and Benchmarking Coding Agents for Non-Functional Improvementscited by 0
  8. 2026SkillShield: Prompt-Space Security Skills for LLM Coding Agentscited by 0
  9. 2026A Study of the Reliability of Agentic AI-Generated Programscited by 0
  10. 2026SecProbe: Adaptive Evaluation of Coding Agents on Cybersecurity Vulnerabilitiescited by 0
  11. 2026AuraForge: Scaling Security Supervision for Training Coding Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks
  2. 2026MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents
  3. 2025SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositories
  4. 2026SoK: AI Secure Code Generation: Progress, Pitfalls, and Paths Forward
  5. 2025Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks
  6. 2026CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability
  7. 2026CIBER: A Comprehensive Benchmark for Security Evaluation of Code Interpreter Agents
  8. 2026IssueTrojanBench: Benchmarking AI Coding Agents Against Malicious Issue Requests