mlsanityData to 14 Oct 2026

SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks

arXiv 2506.11791 ↗First posted 13 Jun 2025cs.LG cs.CRcited by 75

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Rigorous security-focused evaluation of large language model (LLM) agents is imperative for establishing trust in their safe deployment throughout the software development lifecycle. However, existing benchmarks largely rely on synthetic challenges or simplified vulnerability datasets that fail to capture the complexity and ambiguity encountered by security engineers in practice. We introduce SEC-bench, the first fully automated benchmarking framework for evaluating LLM agents on authentic security engineering tasks. SEC-bench employs a novel multi-agent scaffold that automatically constructs code repositories with harnesses, reproduces vulnerabilities in isolated environments, and generates gold patches for reliable evaluation. Our framework automatically creates high-quality software vulnerability datasets with reproducible artifacts at a cost of only 0.87 per instance. Using SEC-bench, we implement two critical software security tasks to rigorously evaluate LLM agents' capabilities: proof-of-concept (PoC) generation and vulnerability patching. A comprehensive evaluation of state-of-the-art LLM code agents reveals significant performance gaps, achieving at most 18.0% success in PoC generation and 34.0% in vulnerability patching on our complete dataset. These results highlight the crucial steps needed toward developing LLM agents that are more practical, intelligent, and autonomous for security engineering.

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 41 of its 100 references that are in the map. First posted on arXiv 13 Jun 2025.

Builds on

41 of 100 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities21 Mar

Other references · 40

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  4. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  5. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  6. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  7. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  8. 2024Executable Code Actions Elicit Better LLM Agentscited by 564
  9. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 546
  10. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 421
  11. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 336
  12. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 285
28 more
  1. 2025SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolutioncited by 263
  2. 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 178
  3. 2025Multi-SWE-bench: A Multilingual Benchmark for Issue Resolvingcited by 171
  4. 2024Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Modelscited by 164
  5. 2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agentscited by 158
  6. 2024Vulnerability Detection with Code Language Models: How Far Are We?cited by 139
  7. 2022Practical Program Repair in the Era of Large Pre-trained Language Modelscited by 133
  8. 2022Large Language Models are Zero-Shot Fuzzers: Fuzzing Deep-Learning Libraries via Large Language Modelscited by 103
  9. 2024SWE-Bench+: Enhanced Coding Benchmark for LLMscited by 99
  10. 2023LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarkscited by 97
  11. 2024NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Securitycited by 93
  12. 2021CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Softwarecited by 91
  13. 2024CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Modelscited by 89
  14. 2023Fuzz4All: Universal Fuzzing with Large Language Modelscited by 88
  15. 2025SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agentscited by 85
  16. 2024Evaluating Language Models for Efficient Code Generationcited by 73
  17. 2024MarsCode Agent: AI-native Automated Bug Fixingcited by 71
  18. 2023Data Quality for Software Vulnerability Datasetscited by 51
  19. 2024Generalization-Enhanced Code Vulnerability Detection via Multi-Task Instruction Fine-Tuningcited by 30
  20. 2023Prompt Fuzzing for Fuzz Driver Generationcited by 27
  21. 2023How Effective Are They? Exploring Large Language Model Based Fuzz Driver Generationcited by 21
  22. 2024To Err is Machine: Vulnerability Detection Challenges LLM Reasoningcited by 18
  23. 2024KernJC: Automated Vulnerable Environment Generation for Linux Kernel Vulnerabilitiescited by 2
  24. 2025An Empirical Study on the Classification of Bug Reports with Machine Learningcited by 2
  25. 2017EffectiveSan: Type and Memory Error Detection using Dynamically Typed C/C++cited by 1
  26. 2022Beyond the Surface: Investigating Malicious CVE Proof of Concept Exploits on GitHubcited by 1
  27. 2023GWP-ASan: Sampling-Based Detection of Memory-Safety Bugs in Productioncited by 1
  28. 2024Detecting Build Dependency Errors in Incremental Buildscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 63.

75 papers.

Sort:

By area:

  1. 2025Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?cited by 95
  2. 2026ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?cited by 29
  3. 2025Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Taskscited by 22
  4. 2025A Survey on Agentic Security: Applications, Threats and Defensescited by 20
  5. 2026SkillSafetyBench: Evaluating Agent Safety under Skill-Facing Attack Surfacescited by 19
  6. 2026ExploitBench: A Capability Ladder Benchmark for LLM Cybersecurity Agentscited by 19
  7. 2026AgentHazard: A Benchmark for Evaluating Harmful Behavior in Computer-Use Agentscited by 18
  8. 2025Training Language Model Agents to Find Vulnerabilities with CTF-Dojocited by 15
  9. 2025A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic Systemcited by 15
  10. 2025A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Codecited by 14
  11. 2025SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarioscited by 13
  12. 2025It's a TRAP! Task-Redirecting Agent Persuasion Benchmark for Web Agentscited by 10
38 more
  1. 2026CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerabilitycited by 9
  2. 2026SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?cited by 9
  3. 2026Risky-Bench: Probing Agentic Safety Risks under Real-World Deploymentcited by 8
  4. 2026Scaling Coding Agents via Atomic Skillscited by 8
  5. 2026CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilitiescited by 8
  6. 2026Safety Testing LLM Agents at Scale: From Risk Discovery to Evidence-Grounded Verificationcited by 7
  7. 2025CATArena: Evaluating Evolutionary Capabilities of Code Agents via Iterative Tournamentscited by 5
  8. 2025AutoBaxBuilder: Bootstrapping Code Security Benchmarkingcited by 5
  9. 2026LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environmentscited by 5
  10. 2025LLM Agents for Automated Web Vulnerability Reproduction: Are We There Yet?cited by 4
  11. 2026From SFT to RL: Demystifying the Post-Training Pipeline for LLM-based Vulnerability Detectioncited by 4
  12. 2026KLong: Training LLM Agent for Extremely Long-horizon Taskscited by 4
  13. 2026ZeroDayBench: Evaluating LLM Agents on Unseen Zero-Day Vulnerabilities for Cyberdefensecited by 4
  14. 2026AnyPoC: Universal Proof-of-Concept Test Generation for Scalable LLM-Based Bug Detectioncited by 4
  15. 2025When "Correct" Is Not Safe: Can We Trust Functionally Correct Patches Generated by Code Agents?cited by 3
  16. 2026Execution-State-Aware LLM Reasoning for Automated Proof-of-Vulnerability Generationcited by 3
  17. 2026MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agentscited by 3
  18. 2026Sandboxed Coding Agents are Competitive Omni-modal Task Solverscited by 3
  19. 2026VulnResolver: A Hybrid Agent Framework for LLM-Based Automated Vulnerability Issue Resolutioncited by 2
  20. 2026CrackMeBench: Binary Reverse Engineering for Agentscited by 2
  21. 2026HARP: Measuring Harm Amplification in Multi-Agent LLM Systemscited by 2
  22. 2026EvoRepair: Enhancing Vulnerability Repair Agents Through Experience-Based Self-Evolutioncited by 2
  23. 2026Self-Evolving Coding Agentscited by 2
  24. 2026PatchBench: Evaluating AI Agents for Vulnerability Patchingcited by 2
  25. 2026CheckerBench: Can Long-Horizon Agents Synthesize Static-Analysis Checkers?cited by 2
  26. 2026To Defend Against Cyber Attacks, We Must Teach AI Agents to Hackcited by 1
  27. 2026Quality-Driven Agentic Reasoning for LLM-Assisted Software Design: Questions-of-Thoughts (QoT) as a Time-Series Self-QA Chaincited by 1
  28. 2026VulnAgent-R2: Evidence-Calibrated Multi-Agent Auditing for Repository-Level Vulnerability Detectioncited by 1
  29. 2026WebSP-Eval: Evaluating Web Agents on Website Security and Privacy Taskscited by 1
  30. 2026Veritas: Grounding LLM Agents for Reliable Vulnerability Reasoning over Stripped Binariescited by 1
  31. 2026APWA: A Distributed Architecture for Parallelizable Agentic Workflowscited by 1
  32. 2026Refusal Evaluation in Coding LLMs and Code Agents: A Systematic Review of Thirteen Malicious-Code Prompt Corpora (2023-2025)cited by 1
  33. 2026Revelio: Cost-Efficient Agentic Memory Safety Vulnerability Detection For Repository-Scale Codebasescited by 1
  34. 2026RustMizan: A Compilable, Contamination-Aware Benchmarking Framework for Rust Vulnerabilitiescited by 1
  35. 2026LivingArena: Do LLMs Know What Other LLMs Don't? Peer-Probing as Scalable Evaluationcited by 1
  36. 2026Cyber-Capable AI Agents: Vulnerabilities, Evaluation Containment, and Defensive Responsecited by 1
  37. 2026AgenticRepair: Multi-Faceted Program Context Engineering for Agentic Vulnerability Repaircited by 1
  38. 2026The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmarkcited by 1

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024AutoSafeCoder: A Multi-Agent Framework for Securing LLM Code Generation through Static Analysis and Fuzz Testing
  2. 2025From LLMs to Agents: A Comparative Evaluation of LLMs and LLM-based Agents in Security Patch Detection
  3. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents
  4. 2025Benchmarking LLMs and LLM-based Agents in Practical Vulnerability Detection for Code Repositories
  5. 2025SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositories
  6. 2025MCP Security Bench (MSB): Benchmarking Attacks Against Model Context Protocol in LLM Agents
  7. 2026Systematic Capability Benchmarking of Frontier Large Language Models for Offensive Cyber Tasks
  8. 2025How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench