mlsanityData to 14 Oct 2026

SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositories

arXiv 2504.21205 ↗First posted 29 Apr 2025cs.CR cs.AIcited by 14

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

This paper introduces SecRepoBench, a benchmark to evaluate code agents on secure code completion in real-world repositories. SecRepoBench has 318 code completion tasks in 27 C/C++ repositories, covering 15 CWEs. We evaluate 29 standalone LLMs and 15 code agents across 3 state-of-the-art agent frameworks using our benchmark. We find that state-of-the-art LLMs struggle with generating correct and secure code completions. However, code agents significantly outperform standalone LLMs. We show that SecRepoBench is more difficult than the prior state-of-the-art benchmark. Finally, our comprehensive analysis provides insights into potential directions for enhancing the ability of code agents to write correct and secure code in real-world repositories.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 22 of its 37 references that are in the map. First posted on arXiv 29 Apr 2025.

Builds on

22 of 37 references.

References · 22

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  4. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  5. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  6. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  7. 2021Measuring Coding Challenge Competence With APPScited by 902
  8. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 431
  9. 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 353
  10. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  11. 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 183
  12. 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 155
10 more
  1. 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 138
  2. 2023DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detectioncited by 102
  3. 2023Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generationcited by 73
  4. 2025CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generationcited by 59
  5. 2023CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Modelscited by 55
  6. 2024Long Code Arena: a Set of Benchmarks for Long-Context Code Modelscited by 50
  7. 2024Top Leaderboard Ranking = Top Coding Proficiency, Always? EvoEval: Evolving Coding Benchmarks via LLMcited by 40
  8. 2024PromSec: Prompt Optimization for Secure Generation of Functional Source Code with Large Language Models (LLMs)cited by 19
  9. 2023Enhancing Large Language Models for Secure Code Generation: A Dataset-driven Study on Vulnerability Mitigationcited by 15
  10. 2024Enhancing Security of AI-Based Code Synthesis with GitHub Copilot via Cheap and Efficient Prompt-Engineeringcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 10.

14 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2026ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?cited by 29
  3. 2025Cyber-Zero: Training Cybersecurity Agents without Runtimecited by 22
  4. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 15
  5. 2026OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Codingcited by 15
  6. 2026CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilitiescited by 8
  7. 2025DualGauge: Automated Joint Security-Functionality Benchmarking of Specification-Only Code Generation by LLMs and Coding Agentscited by 1
  8. 2026Refusal Evaluation in Coding LLMs and Code Agents: A Systematic Review of Thirteen Malicious-Code Prompt Corpora (2023-2025)cited by 1
  9. 2026SEVRA-BENCH: Social Engineering of Vulnerabilities in Review Agentscited by 1
  10. 2026SoK: AI Secure Code Generation: Progress, Pitfalls, and Paths Forwardcited by 0
  11. 2026The Patchwork Problem in LLM-Generated Codecited by 0
  12. 2026EduPluginBench: Executable Assurance for AI-Generated Educational Pluginscited by 0
2 more
  1. 2026Large Language Models at the Intersection of Software Engineering and Software Security:An Evidence-Centered Structured Survey and Research Agendacited by 0
  2. 2026SkillShield: Prompt-Space Security Skills for LLM Coding Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks
  2. 2025SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios
  3. 2026RealSec-bench: A Benchmark for Evaluating Secure Code Generation in Real-World Repositories
  4. 2025GitTaskBench: A Benchmark for Code Agents Solving Real-World Tasks Through Code Repository Leveraging
  5. 2025Benchmarking LLMs and LLM-based Agents in Practical Vulnerability Detection for Code Repositories
  6. 2026CrackMeBench: Binary Reverse Engineering for Agents
  7. 2026ProcCtrlBench: Evaluating Process-Level Defects and Control Preservation in LLM Coding Agents
  8. 2025SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents