mlsanityData to 7 Oct 2026

SecProbe: Adaptive Evaluation of Coding Agents on Cybersecurity Vulnerabilities

arXiv 2609.33763 ↗First posted 27 Sep 2026cs.CR cs.AIcited by 0

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Assessing cybersecurity vulnerability awareness in coding agents requires evaluations that reveal capability gaps and remain informative as models evolve. Static benchmarks offer fixed coverage and difficulty, while scarce vulnerable repositories and costly expert authoring limit their renewal at scale. We introduce SecProbe, a framework for adaptive evaluation that combines Item Response Theory (IRT) with on-demand synthesis of repository-scale vulnerability-repair tasks. From observed performance, textscSecProbe estimates agent ability and identifies where additional evidence is most informative, selecting existing tasks or synthesizing new ones accordingly. As one use case, we construct 353 tasks spanning six programming languages and 151 CWE types and evaluate nine frontier models with two agent harnesses. Success rates peak at 28.33%, highlighting substantial gaps in vulnerability recognition and repair. Compared with random and one-shot baselines, textscSecProbe achieves comparable agent ability estimates while requiring agents to solve up to 29.5% fewer tasks. These results support adaptive evaluation as an efficient and discriminative approach to assessing cybersecurity vulnerability awareness.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 44 of its 58 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

44 of 58 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark11 Aug
  2. 2026ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?11 May
  3. 2026CVE-Factory: Scaling Expert-Level Agentic Tasks for Code Security Vulnerability3 Feb
  4. 2025SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios26 Sep
  5. 2025When Developer Aid Becomes Security Debt: A Systematic Analysis of Insecure Behaviors in LLM Coding Agents12 Jul
  6. 2025How Safe Are AI-Generated Patches? A Large-scale Study on Security Risks in LLM and Agentic Automated Program Repair on SWE-bench30 Jun
  7. 2025RAS-Eval: A Comprehensive Benchmark for Security Evaluation of LLM Agents in Real-World Environments18 Jun
  8. 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks13 Jun
  9. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents3 Oct

Other references · 35

  1. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  2. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  3. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
  4. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  5. 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 515
  6. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  7. 2024tinyBenchmarks: evaluating LLMs with fewer examplescited by 295
  8. 2025SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?cited by 254
  9. 2021Generating Datasets with Pretrained Language Modelscited by 192
  10. 2023Simple synthetic data reduces sycophancy in large language modelscited by 174
  11. 2023AugGPT: Leveraging ChatGPT for Text Data Augmentationcited by 128
  12. 2023Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventionscited by 121
23 more
  1. 2023Anchor Points: Benchmarking Models with Much Fewer Examplescited by 84
  2. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  3. 2025SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agentscited by 80
  4. 2023Efficient Benchmarking of Language Modelscited by 71
  5. 2025CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scalecited by 71
  6. 2024CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databasescited by 62
  7. 2020Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answeringcited by 41
  8. 2023How Predictable Are Large Language Model Capabilities? A Case Study on BIG-benchcited by 34
  9. 2025SWE-Mirror: Scaling Issue-Resolving Datasets by Mirroring Issues Across Repositoriescited by 30
  10. 2025Reliable and Efficient Amortized Model-based Evaluationcited by 29
  11. 2025A Rosetta Stone for AI Benchmarkscited by 29
  12. 2025Automated Benchmark Generation for Repository-Level Coding Taskscited by 27
  13. 2025SWE-Dev: Building Software Engineering Agents with Training and Inference Scalingcited by 24
  14. 2024DataGen: Unified Synthetic Dataset Generation via Large Language Modelscited by 22
  15. 2026SWE-Explore: Benchmarking How Coding Agents Explore Repositoriescited by 20
  16. 2024SyntheT2C: Generating Synthetic Data for Fine-Tuning Large Language Models on the Text2Cypher Taskcited by 19
  17. 2025SWE-fficiency: Can Language Models Optimize Real-World Repositories on Real Workloads?cited by 15
  18. 2025Oasis: One Image is All You Need for Multimodal Instruction Data Synthesiscited by 13
  19. 2025MetaSynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic Data Generationcited by 10
  20. 2026CyberGym-E2E: Scalable Real-World Benchmark for AI Agents' End-to-End Cybersecurity Capabilitiescited by 6
  21. 2025Beyond Single-Value Metrics: Evaluating and Enhancing LLM Unlearning with Cognitive Diagnosiscited by 5
  22. 2025AutoBaxBuilder: Bootstrapping Code Security Benchmarkingcited by 4
  23. 2025Better Datasets Start From RefineLab: Automatic Optimization for High-Quality Dataset Refinementcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026MOSAIC-Bench: Measuring Compositional Vulnerability Induction in Coding Agents
  2. 2026Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale
  3. 2026Beyond the Payload: How User Invocation Shapes Coding Agent Vulnerability to Repository Poisoning
  4. 2026ExploitBench: A Capability Ladder Benchmark for LLM Cybersecurity Agents
  5. 2025Dynamic Risk Assessments for Offensive Cybersecurity Agents
  6. 2026Cyber-Capable AI Agents: Vulnerabilities, Evaluation Containment, and Defensive Response
  7. 2026Transferable Self-Evolving Playbooks for Agentic Security Auditing
  8. 2026Agent Skills in the Wild: An Empirical Study of Security Vulnerabilities at Scale