SPECA: Specification-to-Checklist Agentic Auditing for Multi-Implementation Systems -- A Case Study on Ethereum Clients
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
Multi-implementation systems are increasingly audited against natural-language specifications. Differential testing scales well when implementations disagree, but it provides little signal when all implementations converge on the same incorrect interpretation of an ambiguous requirement. We present SPECA, a Specification-to-Checklist Auditing framework that turns normative requirements into checklists, maps them to implementation locations, and supports cross-implementation reuse. We instantiate SPECA in an in-the-wild security audit contest for the Ethereum Fusaka upgrade, covering 11 production clients. Across 54 submissions, 17 were judged valid by the contest organizers. Cross-implementation checks account for 76.5 percent (13 of 17) of valid findings, suggesting that checklist-derived one-to-many reuse is a practical scaling mechanism in multi-implementation audits. To understand false positives, we manually coded the 37 invalid submissions and find that threat model misalignment explains 56.8 percent (21 of 37): reports that rely on assumptions about trust boundaries or scope that contradict the audit's rules. We detected no High or Medium findings in the V1 deployment; misses concentrated in specification details and implicit assumptions (57.1 percent), timing and concurrency issues (28.6 percent), and external library dependencies (14.3 percent). Our improved agent, evaluated against the ground truth of a competitive audit, achieved a strict recall of 27.3 percent on high-impact vulnerabilities, placing it in the top 4 percent of human auditors and outperforming 49 of 51 contestants on critical issues. These results, though from a single deployment, suggest that early, explicit threat modeling is essential for reducing false positives and focusing agentic auditing effort. The agent-driven process enables expert validation and submission in about 40 minutes on average.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 8 of its 13 references that are in the map. First posted on arXiv 7 Feb 2026.
Builds on
8 of 13 references.
Closest ancestors · 2
Same direction, newest first
- 2025VulAgent: Hypothesis-Validation based Multi-Agent Vulnerability Detection15 Sep
- 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents3 Oct
Other references · 6
- 2024IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilitiescited by 103
- 2025RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditingcited by 49
- 2025QLPro: Automated Code Vulnerability Discovery via LLM and Static Code Analysis Integrationcited by 6
- 2025MAVUL: Multi-Agent Vulnerability Detection via Contextual Reasoning and Interactive Refinementcited by 6
- 2025Synthesizing Precise Protocol Specs from Natural Language for Effective Test Generationcited by 2
- 2025Uncovering Gaps Between RFC Updates and TCP/IP Implementations: LLM-Facilitated Differential Checks on Intermediate Representationscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026LEDGER: Claim-to-Evidence Trace Graphs for Auditing LLM Agents
- 2026An Effective and Cost-Efficient Agentic Framework for Ethereum Smart Contract Auditing
- 2024Combining Fine-Tuning and LLM-based Agents for Intuitive Smart Contract Auditing with Justifications
- 2026Decoupled Smart Contract Audits: Lightweight LLM Framework via Distillation and Aggregation
- 2026Who Tests the Testers? Systematic Enumeration and Coverage Audit of LLM Agent Tool Call Safety
- 2026VulnAgent-R2: Evidence-Calibrated Multi-Agent Auditing for Repository-Level Vulnerability Detection
- 2026IMMACULATE: A Practical LLM Auditing Framework via Verifiable Computation
- 2025TRUST: A Decentralized Framework for Auditing Large Language Model Reasoning