Trust, but Validate the Instrument: Auditing AI-Generated RTL Verification Plans on Authored Security-Regression Proxies
Not on the map. Why?
Abstract (arXiv)
AI-generated RTL verification plans can satisfy a provider schema yet fail at the boundary to trusted execution. We present SecTB-RTL, an auditable framework covering 31 tasks and 124 authored hardware-security regressions. A deterministic non-AI baseline killed 36, 75, and 78 mutants at increasing resource limits. The first confirmatory run (C1-R2) failed before model execution because the provider rejected its response schema. After a schema-only repair made without viewing outcomes, a separately frozen follow-up run (C1-R3) completed 1,860 calls. The provider accepted 1,857 responses, but only nine passed the production semantic validator. The generation and execution rules did not match. We therefore preserve the run as an instrument-validation incident and report no prompt-effect estimate. This incident shows that provider or schema acceptance does not establish execution validity. Compilation and coverage are only diagnostics; the exact saved artifact must pass the full production path. A subsequent follow-up is excluded because it did not satisfy the preregistered evidence-completeness gate and is treated only as future work. We release the benchmark, failure-preserving contract, incident provenance, and governance controls needed to prevent infrastructure behavior from being misreported as model behavior.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 7 of its 10 references that are in the map. First posted on arXiv 17 Sep 2026.
Builds on
7 of 10 references.
References · 7
- 2024AutoBench: Automatic Testbench Generation and Evaluation Using LLMs for HDL Designcited by 41
- 2023(Security) Assertions by Large Language Modelscited by 32
- 2024AssertLLM: Generating Hardware Verification Assertions from Design Specifications via Multi-LLMscited by 31
- 2023LLM4DV: Using Large Language Models for Hardware Test Stimuli Generationcited by 23
- 2024AssertionBench: A Benchmark to Evaluate Large-Language Models for Assertion Generationcited by 16
- 2026HardSecBench: Benchmarking the Security Awareness of LLMs for Hardware Code Generationcited by 5
- 2026GRPO with State Mutations: Improving LLM-Based Hardware Test Plan Generationcited by 4
Built on by
Not cited yet.