# Detection Made Easy: Potentials of Large Language Models for Solidity Vulnerabilities

arXiv: https://arxiv.org/abs/2409.10574 · HTML: https://mlsanity.com/paper/2409.10574/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 47 references that are in the map.
- First posted on arXiv 15 Sep 2024.

## Builds on (most cited first, 26 of 26)

- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X](https://mlsanity.com/paper/2303.17568/) (2023)
- [Software Testing with Large Language Models: Survey, Landscape, and Vision](https://mlsanity.com/paper/2307.07221/) (2023)
- [Slither: A Static Analysis Framework For Smart Contracts](https://mlsanity.com/paper/1908.09878/) (2019)
- [GPTScan: Detecting Logic Vulnerabilities in Smart Contracts by Combining GPT with Program Analysis](https://mlsanity.com/paper/2308.03314/) (2023)
- [Securify: Practical Security Analysis of Smart Contracts](https://mlsanity.com/paper/1806.01143/) (2018)
- [A Quantitative and Qualitative Evaluation of LLM-Based Explainable Fault Localization](https://mlsanity.com/paper/2308.05487/) (2023)
- [Large Language Model-Powered Smart Contract Vulnerability Detection: New Perspectives](https://mlsanity.com/paper/2310.01152/) (2023)
- [When ChatGPT Meets Smart Contract Vulnerability Detection: How Far Are We?](https://mlsanity.com/paper/2309.05520/) (2023)
- [Manticore: A User-Friendly Symbolic Execution Framework for Binaries and Smart Contracts](https://mlsanity.com/paper/1907.03890/) (2019)
- [A Systematic Literature Review on Large Language Models for Automated Program Repair](https://mlsanity.com/paper/2405.01466/) (2024)
- [Do you still need a manual smart contract audit?](https://mlsanity.com/paper/2306.12338/) (2023)
- [Combining Graph Neural Networks with Expert Knowledge for Smart Contract Vulnerability Detection](https://mlsanity.com/paper/2107.11598/) (2021)
- [Smart Contract Vulnerability Detection: From Pure Neural Network to Interpretable Graph Feature and Expert Pattern Fusion](https://mlsanity.com/paper/2106.09282/) (2021)
- [How Effective are Smart Contract Analysis Tools? Evaluating Smart Contract Static Analysis Tools Using Bug Injection](https://mlsanity.com/paper/2005.11613/) (2020)
- [SmartBugs: A Framework to Analyze Solidity Smart Contracts](https://mlsanity.com/paper/2007.04771/) (2020)
- [AuditGPT: Auditing Smart Contracts with ChatGPT](https://mlsanity.com/paper/2404.04306/) (2024)
- [Multi-Class classification of vulnerabilities in Smart Contracts using AWD-LSTM, with pre-trained encoder inspired from natural language processing](https://mlsanity.com/paper/2004.00362/) (2020)
- [Defining Smart Contract Defects on Ethereum](https://mlsanity.com/paper/1905.01467/) (2019)
- [ScrawlD: A Dataset of Real World Ethereum Smart Contracts Labelled with Vulnerabilities](https://mlsanity.com/paper/2202.11409/) (2022)
- [Efficiently Detecting Reentrancy Vulnerabilities in Complex Smart Contracts](https://mlsanity.com/paper/2403.11254/) (2024)
- [Vulpedia: Detecting Vulnerable Ethereum Smart Contracts via Abstracted Vulnerability Signatures](https://mlsanity.com/paper/1912.04466/) (2019)
- [Security Defense For Smart Contracts: A Comprehensive Survey](https://mlsanity.com/paper/2302.07347/) (2023)
- [A Comparative Evaluation of Automated Analysis Tools for Solidity Smart Contracts](https://mlsanity.com/paper/2310.20212/) (2023)
- [Soley: Identification and Automated Detection of Logic Vulnerabilities in Ethereum Smart Contracts Using Large Language Models](https://mlsanity.com/paper/2406.16244/) (2024)

## Built on by (most cited first, 2 of 2)

- [LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights](https://mlsanity.com/paper/2502.07049/) (2025)
- [Towards Secure Cloud-Native Computing: Unveiling Kubernetes Misconfigurations with Large Language Models](https://mlsanity.com/paper/2609.20834/) (2026)

## Abstract (arXiv)

The large-scale deployment of Solidity smart contracts on the Ethereum mainnet has increasingly attracted financially-motivated attackers in recent years. A few now-infamous attacks in Ethereum's history includes DAO attack in 2016 (50 million dollars lost), Parity Wallet hack in 2017 (146 million dollars locked), Beautychain's token BEC in 2018 (900 million dollars market value fell to 0), and NFT gaming blockchain breach in 2022 (600 million in Ether stolen). This paper presents a comprehensive investigation of the use of large language models (LLMs) and their capabilities in detecting OWASP Top Ten vulnerabilities in Solidity. We introduce a novel, class-balanced, structured, and labeled dataset named VulSmart, which we use to benchmark and compare the performance of open-source LLMs such as CodeLlama, Llama2, CodeT5 and Falcon, alongside closed-source models like GPT-3.5 Turbo and GPT-4o Mini. Our proposed SmartVD framework is rigorously tested against these models through extensive automated and manual evaluations, utilizing BLEU and ROUGE metrics to assess the effectiveness of vulnerability detection in smart contracts. We also explore three distinct prompting strategies-zero-shot, few-shot, and chain-of-thought-to evaluate the multi-class classification and generative capabilities of the SmartVD framework. Our findings reveal that SmartVD outperforms its open-source counterparts and even exceeds the performance of closed-source base models like GPT-3.5 and GPT-4 Mini. After fine-tuning, the closed-source models, GPT-3.5 Turbo and GPT-4o Mini, achieved remarkable performance with 99% accuracy in detecting vulnerabilities, 94% in identifying their types, and 98% in determining severity. Notably, SmartVD performs best with the `chain-of-thought' prompting technique, whereas the fine-tuned closed-source models excel with the `zero-shot' prompting approach.