mlsanityData to 14 Oct 2026

The Model Proposes, the Code Disposes: A Pre-Registered Ablation of a Verifier-and-Acceptance Stage in an LLM-Orchestrated Offensive-Security Agent

arXiv 2609.15887 ↗First posted 14 Sep 2026cs.CRcited by 1

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

We evaluate whether a verifier-and-acceptance stage - a model verifier whose verdicts are enforced by deterministic code - changes what an LLM-driven offensive-security agent reports. We report a 15-run exploratory pilot, a pre-registered 20-run confirmatory ablation, and a pre-registered 2 x 2 factorial study with 40 runs across two deliberately vulnerable lab targets. In the confirmatory study, removing the stage eliminated pre-report suppression (median 2 versus 0 findings per run; exact one-sided p = 0.00003) and reduced model-blinded shipped precision (median 0.471 versus 0.353; p = 0.0087). Recall against a frozen but incomplete ground-truth list did not differ significantly (two-sided p = 0.158; equivalence was not established). The factorial study attributed suppression to the model verifier (Holm-adjusted p = 0.004); deterministic acceptance rules alone suppressed no false positives, and no interaction was detected (p = 0.72). The full design retained 93.8% of model-adjudicated true candidates but did not meet its pre-registered non-inferiority criterion because the lower one-sided 95% bound was 0.875, below the 0.90 floor. Across the confirmatory and factorial studies, an instrumented canary recorded zero contacts in 60 of 60 runs, with incidental external contacts disclosed separately. Independent human adjudication of the retained blind packets is pending, so precision and sensitivity endpoints are supporting rather than final evidence. Six audit-trail failures, including one in the evaluation tooling, are also disclosed. The results support a narrow conclusion: the verifier changes what the system ships, while deterministic code supplies enforcement and auditability; they do not establish superiority to other agents or generalization beyond lab targets.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 2 of its 13 references that are in the map. First posted on arXiv 14 Sep 2026.

Builds on

2 of 13 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Guided Reasoning in LLM-Driven Penetration Testing Using Structured Attack Trees9 Sep

Other references · 1

  1. 2023PentestGPT: An LLM-empowered Automatic Penetration Testing Toolcited by 97

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2025Recognition Without Mitigation: Ethical Frameworks in Autonomous Offensive-LLM Agent Research
  2. 2026Where Cyber Agents Struggle: Bottleneck Analysis of Multi-Stage LLM Agents
  3. 2026Recognition Without Enforcement: Configuration-Dependent Failures in LLM Agent Instruction Arbitration and External Control
  4. 2026Prompted to Discriminate: Generalizing Malicious-Input Probes in the Wild
  5. 2026How does information access affect LLM monitors' ability to detect sabotage?
  6. 2026Towards a Unified Misuse Monitoring Benchmark
  7. 2026Look Before You Leap: Pre-Action Verification for LLM Agents
  8. 2025Verifiability-First Agents: Provable Observability and Lightweight Audit Agents for Controlling Autonomous LLM Systems