Trust, but Validate the Instrument: Auditing AI-Generated RTL Verification Plans on Authored Security-Regression Proxies
Abstract
AI-generated RTL verification plans can satisfy a
provider schema yet fail at the boundary to trusted execution.
We present SecTB-RTL, an auditable framework covering
31 tasks and 124 authored hardware-security regressions. A
deterministic non-AI baseline killed 36, 75, and 78 mutants at
increasing resource limits. C1-R2 then failed at provider schema
prevalidation. After an outcome-blind schema repair, a 1,860-
call run completed with 1,857 provider-accepted responses, but
only nine passed the production semantic validator because
the generation-facing and execution contracts disagreed. We
therefore preserve the run as an instrument-validation incident
and report no prompt-effect estimate. The incident directly shows
that provider/schema acceptance is not evidence of execution
validity; our contract likewise treats compilation and structural
coverage only as diagnostics until the exact persisted artifact
passes the production path. A subsequent follow-up is excluded
because it did not satisfy the preregistered evidence-completeness
gate and is treated only as future work. We release the benchmark,
failure-preserving contract, incident provenance, and governance
controls needed to prevent infrastructure behavior from being
misreported as model behavior.
provider schema yet fail at the boundary to trusted execution.
We present SecTB-RTL, an auditable framework covering
31 tasks and 124 authored hardware-security regressions. A
deterministic non-AI baseline killed 36, 75, and 78 mutants at
increasing resource limits. C1-R2 then failed at provider schema
prevalidation. After an outcome-blind schema repair, a 1,860-
call run completed with 1,857 provider-accepted responses, but
only nine passed the production semantic validator because
the generation-facing and execution contracts disagreed. We
therefore preserve the run as an instrument-validation incident
and report no prompt-effect estimate. The incident directly shows
that provider/schema acceptance is not evidence of execution
validity; our contract likewise treats compilation and structural
coverage only as diagnostics until the exact persisted artifact
passes the production path. A subsequent follow-up is excluded
because it did not satisfy the preregistered evidence-completeness
gate and is treated only as future work. We release the benchmark,
failure-preserving contract, incident provenance, and governance
controls needed to prevent infrastructure behavior from being
misreported as model behavior.
Research Areas
