# Do you still need a manual smart contract audit?

arXiv: https://arxiv.org/abs/2306.12338 · HTML: https://mlsanity.com/paper/2306.12338/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jun 2023.

## Built on by (most cited first, 30 of 35)

- [Large Language Models for Cyber Security: A Systematic Literature Review](https://mlsanity.com/paper/2405.04760/) (2024)
- [GPTScan: Detecting Logic Vulnerabilities in Smart Contracts by Combining GPT with Program Analysis](https://mlsanity.com/paper/2308.03314/) (2023)
- [LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoning](https://mlsanity.com/paper/2401.16185/) (2024)
- [Large Language Model-Powered Smart Contract Vulnerability Detection: New Perspectives](https://mlsanity.com/paper/2310.01152/) (2023)
- [Combining Fine-Tuning and LLM-based Agents for Intuitive Smart Contract Auditing with Justifications](https://mlsanity.com/paper/2403.16073/) (2024)
- [Large Language Models in Cybersecurity: State-of-the-Art](https://mlsanity.com/paper/2402.00891/) (2024)
- [AI Agent Smart Contract Exploit Generation](https://mlsanity.com/paper/2507.05558/) (2025)
- [Smart-LLaMA: Two-Stage Post-Training of Large Language Models for Smart Contract Vulnerability Detection and Explanation](https://mlsanity.com/paper/2411.06221/) (2024)
- [Smart-LLaMA-DPO: Reinforced Large Language Model for Explainable Smart Contract Vulnerability Detection](https://mlsanity.com/paper/2506.18245/) (2025)
- [Logic Meets Magic: LLMs Cracking Smart Contract Vulnerabilities](https://mlsanity.com/paper/2501.07058/) (2025)
- [LLMs unlock new paths to monetizing exploits](https://mlsanity.com/paper/2505.11449/) (2025)
- [Retrieval Augmented Generation Integrated Large Language Models in Smart Contract Vulnerability Detection](https://mlsanity.com/paper/2407.14838/) (2024)
- [Prompt to Pwn: Automated Exploit Generation for Smart Contracts](https://mlsanity.com/paper/2508.01371/) (2025)
- [AiRacleX: Automated Detection of Price Oracle Manipulations via LLM-Driven Knowledge Mining and Prompt Generation](https://mlsanity.com/paper/2502.06348/) (2025)
- [Evaluation of ChatGPT's Smart Contract Auditing Capabilities Based on Chain of Thought](https://mlsanity.com/paper/2402.12023/) (2024)
- [Generative transformations and patterns in LLM-native approaches for software verification and falsification](https://mlsanity.com/paper/2404.09384/) (2024)
- [Detection Made Easy: Potentials of Large Language Models for Solidity Vulnerabilities](https://mlsanity.com/paper/2409.10574/) (2024)
- [Blockchain Meets LLMs: A Living Survey on Bidirectional Integration](https://mlsanity.com/paper/2411.16809/) (2024)
- [Adaptive Plan-Execute Framework for Smart Contract Security Auditing](https://mlsanity.com/paper/2505.15242/) (2025)
- [SAEL: Leveraging Large Language Models with Adaptive Mixture-of-Experts for Smart Contract Vulnerability Detection](https://mlsanity.com/paper/2507.22371/) (2025)
- [LISA Technical Report: An Agentic Framework for Smart Contract Auditing](https://mlsanity.com/paper/2509.24698/) (2025)
- [TxRay: Agentic Postmortem of Live Blockchain Attacks](https://mlsanity.com/paper/2602.01317/) (2026)
- [Generative AI-enabled Blockchain Networks: Fundamentals, Applications, and Case Study](https://mlsanity.com/paper/2401.15625/) (2024)
- [DISL: Fueling Research with A Large Dataset of Solidity Smart Contracts](https://mlsanity.com/paper/2403.16861/) (2024)
- [FTSmartAudit: A Knowledge Distillation-Enhanced Framework for Automated Smart Contract Auditing Using Fine-Tuned LLMs](https://mlsanity.com/paper/2410.13918/) (2024)
- [Towards Secure and Explainable Smart Contract Generation with Security-Aware Group Relative Policy Optimization](https://mlsanity.com/paper/2509.09942/) (2025)
- [Large Language Model based Smart Contract Auditing with LLMBugScanner](https://mlsanity.com/paper/2512.02069/) (2025)
- [Cracking IoT Security: Can LLMs Outsmart Static Analysis Tools?](https://mlsanity.com/paper/2601.00559/) (2026)
- [AEGIS: From Clues to Verdicts -- Graph-Guided Deep Vulnerability Reasoning via Dialectics and Meta-Auditing](https://mlsanity.com/paper/2603.20637/) (2026)
- [Crypto x AI, AI x Crypto: A Survey](https://mlsanity.com/paper/2606.13892/) (2026)

## Abstract (arXiv)

We investigate the feasibility of employing large language models (LLMs) for conducting the security audit of smart contracts, a traditionally time-consuming and costly process. Our research focuses on the optimization of prompt engineering for enhanced security analysis, and we evaluate the performance and accuracy of LLMs using a benchmark dataset comprising 52 Decentralized Finance (DeFi) smart contracts that have previously been compromised. Our findings reveal that, when applied to vulnerable contracts, both GPT-4 and Claude models correctly identify the vulnerability type in 40% of the cases. However, these models also demonstrate a high false positive rate, necessitating continued involvement from manual auditors. The LLMs tested outperform a random model by 20% in terms of F1-score. To ensure the integrity of our study, we conduct mutation testing on five newly developed and ostensibly secure smart contracts, into which we manually insert two and 15 vulnerabilities each. This testing yielded a remarkable best-case 78.7% true positive rate for the GPT-4-32k model. We tested both, asking the models to perform a binary classification on whether a contract is vulnerable, and a non-binary prompt. We also examined the influence of model temperature variations and context length on the LLM's performance. Despite the potential for many further enhancements, this work lays the groundwork for a more efficient and economical approach to smart contract security audits.