# Logic Meets Magic: LLMs Cracking Smart Contract Vulnerabilities

arXiv: https://arxiv.org/abs/2501.07058 · HTML: https://mlsanity.com/paper/2501.07058/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 67 references that are in the map.
- First posted on arXiv 13 Jan 2025.

## Builds on (most cited first, 30 of 31)

- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [LLM Agents can Autonomously Exploit One-day Vulnerabilities](https://mlsanity.com/paper/2404.08144/) (2024)
- [InferFix: End-to-End Program Repair with LLMs](https://mlsanity.com/paper/2303.07263/) (2023)
- [Examining Zero-Shot Vulnerability Repair with Large Language Models](https://mlsanity.com/paper/2112.02125/) (2021)
- [LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks](https://mlsanity.com/paper/2312.12575/) (2023)
- [Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants](https://mlsanity.com/paper/2208.09727/) (2022)
- [Slither: A Static Analysis Framework For Smart Contracts](https://mlsanity.com/paper/1908.09878/) (2019)
- [GPTScan: Detecting Logic Vulnerabilities in Smart Contracts by Combining GPT with Program Analysis](https://mlsanity.com/paper/2308.03314/) (2023)
- [When ChatGPT Meets Smart Contract Vulnerability Detection: How Far Are We?](https://mlsanity.com/paper/2309.05520/) (2023)
- [Combining Fine-Tuning and LLM-based Agents for Intuitive Smart Contract Auditing with Justifications](https://mlsanity.com/paper/2403.16073/) (2024)
- [Transformer-Based Language Models for Software Vulnerability Detection](https://mlsanity.com/paper/2204.03214/) (2022)
- [Empirical Review of Automated Analysis Tools on 47,587 Ethereum Smart Contracts](https://mlsanity.com/paper/1910.10601/) (2019)
- [Do you still need a manual smart contract audit?](https://mlsanity.com/paper/2306.12338/) (2023)
- [Non-Fungible Token (NFT): Overview, Evaluation, Opportunities and Challenges](https://mlsanity.com/paper/2105.07447/) (2021)
- [SoK: Decentralized Finance (DeFi)](https://mlsanity.com/paper/2101.08778/) (2021)
- [An Overview on Smart Contracts: Challenges, Advances and Platforms](https://mlsanity.com/paper/1912.10370/) (2019)
- [Finding The Greedy, Prodigal, and Suicidal Contracts at Scale](https://mlsanity.com/paper/1802.06038/) (2018)
- [sFuzz: An Efficient Adaptive Fuzzer for Solidity Smart Contracts](https://mlsanity.com/paper/2004.08563/) (2020)
- [Vandal: A Scalable Security Analysis Framework for Smart Contracts](https://mlsanity.com/paper/1809.03981/) (2018)
- [SmartBugs: A Framework to Analyze Solidity Smart Contracts](https://mlsanity.com/paper/2007.04771/) (2020)
- [Rethinking Smart Contract Fuzzing: Fuzzing With Invocation Ordering and Important Branch Revisiting](https://mlsanity.com/paper/2301.03943/) (2023)
- [Smart Contract Templates: foundations, design landscape and research directions](https://mlsanity.com/paper/1608.00771/) (2016)
- [LLM4Fuzz: Guided Fuzzing of Smart Contracts with Large Language Models](https://mlsanity.com/paper/2401.11108/) (2024)
- [BlockScope: Detecting and Investigating Propagated Vulnerabilities in Forked Blockchain Projects](https://mlsanity.com/paper/2208.00205/) (2022)
- [SmartBugs 2.0: An Execution Framework for Weakness Detection in Ethereum Smart Contracts](https://mlsanity.com/paper/2306.05057/) (2023)
- [Decentralized Finance (DeFi): A Survey](https://mlsanity.com/paper/2308.05282/) (2023)
- [Exploring Web3 From the View of Blockchain](https://mlsanity.com/paper/2206.08821/) (2022)

## Built on by (most cited first, 5 of 5)

- [Prompt to Pwn: Automated Exploit Generation for Smart Contracts](https://mlsanity.com/paper/2508.01371/) (2025)
- [Domain Specific Benchmarks for Evaluating Multimodal Large Language Models](https://mlsanity.com/paper/2506.12958/) (2025)
- [Validating Solidity Code Defects using Symbolic and Concrete Execution powered by Large Language Models](https://mlsanity.com/paper/2509.13023/) (2025)
- [Benchmarking Zero-Shot Reasoning Approaches for Error Detection in Solidity Smart Contracts](https://mlsanity.com/paper/2603.13239/) (2026)
- [Benchmarking LLM-Based Static Analysis for Secure Smart Contract Development: Reliability, Limitations, and Potential Hybrid Solutions](https://mlsanity.com/paper/2605.11163/) (2026)

## Abstract (arXiv)

Smart contract vulnerabilities caused significant economic losses in blockchain applications. Large Language Models (LLMs) provide new possibilities for addressing this time-consuming task. However, state-of-the-art LLM-based detection solutions are often plagued by high false-positive rates. In this paper, we push the boundaries of existing research in two key ways. First, our evaluation is based on Solidity v0.8, offering the most up-to-date insights compared to prior studies that focus on older versions (v0.4). Second, we leverage the latest five LLM models (across companies), ensuring comprehensive coverage across the most advanced capabilities in the field. We conducted a series of rigorous evaluations. Our experiments demonstrate that a well-designed prompt can reduce the false-positive rate by over 60%. Surprisingly, we also discovered that the recall rate for detecting some specific vulnerabilities in Solidity v0.8 has dropped to just 13% compared to earlier versions (i.e., v0.4). Further analysis reveals the root cause of this decline: the reliance of LLMs on identifying changes in newly introduced libraries and frameworks during detection.