mlsanityData to 7 Oct 2026

From Runnable to Verifiable: An Independent Reproducibility Study of LLM/Agent-Driven Vulnerability Validation Artifacts

arXiv 2608.09567 ↗First posted 10 Aug 2026cs.CRcited by 0

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Security research artifacts---repositories, PoC exploits, and validation pipelines---are increasingly produced by LLM/agent-driven vulnerability workflows, yet the gap between publicly available, runnable, signal-producing, and semantically confirmed artifacts is poorly measured. We conduct a pre-registered reproducibility audit of this literature. A search covering 2023--2026 with dual screening yields a 104-paper consensus corpus, of which 59 papers (56.7%) have a publicly reachable artifact. We execute an 18-paper sample at R0/R1 and all 102 cases of the anchor benchmark (arXiv:2509.24037), with patched-counterfactual verdicts on 30 signal-producing cases and matched-negative-control verdicts on 19. Three findings stand out. First, 58/102 (56.9%) anchor cases contain a script-internal CVE identifier that diverges from the declared directory CVE. Second, only 10/18 (55.6%) paper-level artifacts complete their declared workflow at R0, rising to 11/18 (61.1%) after environment-only R1 repair. Third, artifact-embedded oracles prove unreliable: 20/30 patched-counterfactual audits still produce the claimed signal on the patched build, 7/19 matched negative controls still trigger on benign input, and the oracle confusion matrix has sensitivity 60% and specificity 45%. A trigger on the vulnerable build is not evidence of CVE-specific reproduction without a clean patched counterfactual. These are exploratory results from a pre-registered protocol, and our protocol---pre-registered post-conditions, R0/R1 repair ladder, G1--G3 semantic evidence levels, and patched-counterfactual oracles---is a reusable template for the security reproducibility community.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 12 of its 20 references that are in the map. First posted on arXiv 10 Aug 2026.

Builds on

12 of 20 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026PoC-Adapt: Semantic-Aware Automated Vulnerability Reproduction with LLM Multi-Agents and Reinforcement Learning-Driven Adaptive Policy8 Apr
  2. 2025FaultLine: Automated Proof-of-Vulnerability Generation Using LLM Agents21 Jul
  3. 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks13 Jun
  4. 2025CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities21 Mar

Other references · 8

  1. 2024Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Modelscited by 156
  2. 2025PATCHEVAL: A New Benchmark for Evaluating LLMs on Patching Real-World Vulnerabilitiescited by 19
  3. 2024AutoPT: How Far Are We from the End2End Automated Web Penetration Testing?cited by 7
  4. 2025Shell or Nothing: Real-World Benchmarks and Memory-Activated Agents for Automated Penetration Testingcited by 7
  5. 2025Good News for Script Kiddies? Evaluating Large Language Models for Automated Exploit Generationcited by 6
  6. 2024ContractTinker: LLM-Empowered Vulnerability Repair for Real-World Smart Contractscited by 4
  7. 2025Automated Vulnerability Validation and Verification: A Large Language Model Approachcited by 3
  8. 2026Patch Validation in Automated Vulnerability Repaircited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025From CVE Entries to Verifiable Exploits: An Automated Multi-Agent Framework for Reproducing CVEs
  2. 2026Veritas: Grounding LLM Agents for Reliable Vulnerability Reasoning over Stripped Binaries
  3. 2026SINGED: Correct Outputs Do Not Certify Safe Execution in LLM Agents
  4. 2026Cheap to Hypothesize, Costly to Verify: The Defense Surface of Agentic Vulnerability Discovery
  5. 2026Local LLM Agents as Vulnerable Runtimes:A Source-Code Audit of the Agent Runtime Layer
  6. 2026Transferable Self-Evolving Playbooks for Agentic Security Auditing
  7. 2026Context-Fractured Decomposition Attacks on Tool-Using LLM Agents: Exploiting Artifact Provenance Gaps
  8. 2026PACE: Provenance-Aware Capability Enforcement for Tool-Using LLM Agents