# Empirical Review of Automated Analysis Tools on 47,587 Ethereum Smart Contracts

arXiv: https://arxiv.org/abs/1910.10601 · HTML: https://mlsanity.com/paper/1910.10601/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2019.

## Built on by (most cited first, 30 of 36)

- [Efficient Avoidance of Vulnerabilities in Auto-completed Smart Contract Code Using Vulnerability-constrained Decoding](https://mlsanity.com/paper/2309.09826/) (2023)
- [Exploring the Security Threats of Knowledge Base Poisoning in Retrieval-Augmented Code Generation](https://mlsanity.com/paper/2502.03233/) (2025)
- [SCALM: Detecting Bad Practices in Smart Contracts Through LLMs](https://mlsanity.com/paper/2502.04347/) (2025)
- [Eth2Vec: Learning Contract-Wide Code Representations for Vulnerability Detection on Ethereum Smart Contracts](https://mlsanity.com/paper/2101.02377/) (2021)
- [Logic Meets Magic: LLMs Cracking Smart Contract Vulnerabilities](https://mlsanity.com/paper/2501.07058/) (2025)
- [A Systematic Study of Code Obfuscation Against LLM-based Vulnerability Detection](https://mlsanity.com/paper/2512.16538/) (2025)
- [MANDO: Multi-Level Heterogeneous Graph Embeddings for Fine-Grained Detection of Smart Contract Vulnerabilities](https://mlsanity.com/paper/2208.13252/) (2022)
- [VulnSense: Efficient Vulnerability Detection in Ethereum Smart Contracts by Multimodal Learning with Graph Neural Network and Language Model](https://mlsanity.com/paper/2309.08474/) (2023)
- [Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Quality](https://mlsanity.com/paper/2503.05860/) (2025)
- [Prompt to Pwn: Automated Exploit Generation for Smart Contracts](https://mlsanity.com/paper/2508.01371/) (2025)
- [SoK: Security and Privacy of AI Agents for Blockchain](https://mlsanity.com/paper/2509.07131/) (2025)
- [SmartLLM: Smart Contract Auditing using Custom Generative AI](https://mlsanity.com/paper/2502.13167/) (2025)
- [Usenix'23 Extended Version: Smart Learning to Find Dumb Contracts](https://mlsanity.com/paper/2304.10726/) (2023)
- [HyMo: Vulnerability Detection in Smart Contracts using a Novel Multi-Modal Hybrid Model](https://mlsanity.com/paper/2304.13103/) (2023)
- [Machine Learning for Blockchain Data Analysis: Progress and Opportunities](https://mlsanity.com/paper/2404.18251/) (2024)
- [Vulnerability Detection in Ethereum Smart Contracts via Machine Learning: A Qualitative Analysis](https://mlsanity.com/paper/2407.18639/) (2024)
- [CodeBC: A More Secure Large Language Model for Smart Contract Code Generation in Blockchain](https://mlsanity.com/paper/2504.21043/) (2025)
- [Adaptive Plan-Execute Framework for Smart Contract Security Auditing](https://mlsanity.com/paper/2505.15242/) (2025)
- [Detecting Rug Pulls in Decentralized Exchanges: Machine Learning Evidence from the TON Blockchain](https://mlsanity.com/paper/2509.01168/) (2025)
- [SourceP: Detecting Ponzi Schemes on Ethereum with Source Code](https://mlsanity.com/paper/2306.01665/) (2023)
- [DISL: Fueling Research with A Large Dataset of Solidity Smart Contracts](https://mlsanity.com/paper/2403.16861/) (2024)
- [FTSmartAudit: A Knowledge Distillation-Enhanced Framework for Automated Smart Contract Auditing Using Fine-Tuned LLMs](https://mlsanity.com/paper/2410.13918/) (2024)
- [UEChecker: Detecting Unchecked External Call Vulnerabilities in DApps via Graph Analysis](https://mlsanity.com/paper/2508.01343/) (2025)
- [Large Language Model based Smart Contract Auditing with LLMBugScanner](https://mlsanity.com/paper/2512.02069/) (2025)
- [Vulnerability and Transaction behavior based detection of Malicious Smart Contracts](https://mlsanity.com/paper/2106.13422/) (2021)
- [Vulnerability Detection in Smart Contracts: A Comprehensive Survey](https://mlsanity.com/paper/2407.07922/) (2024)
- [Knowledge Migration Framework for Smart Contract Vulnerability Detection](https://mlsanity.com/paper/2412.11175/) (2024)
- [Multi-Agent Collaborative Fuzzing with Continuous Reflection for Smart Contracts Vulnerability Detection](https://mlsanity.com/paper/2511.12164/) (2025)
- [Benchmarking Zero-Shot Reasoning Approaches for Error Detection in Solidity Smart Contracts](https://mlsanity.com/paper/2603.13239/) (2026)
- [Robust Smart Contract Vulnerability Detection via Contrastive Learning-Enhanced Granular-ball Training](https://mlsanity.com/paper/2603.27734/) (2026)

## Abstract (arXiv)

Over the last few years, there has been substantial research on automated analysis, testing, and debugging of Ethereum smart contracts. However, it is not trivial to compare and reproduce that research. To address this, we present an empirical evaluation of 9 state-of-the-art automated analysis tools using two new datasets: i) a dataset of 69 annotated vulnerable smart contracts that can be used to evaluate the precision of analysis tools; and ii) a dataset with all the smart contracts in the Ethereum Blockchain that have Solidity source code available on Etherscan (a total of 47,518 contracts). The datasets are part of SmartBugs, a new extendable execution framework that we created to facilitate the integration and comparison between multiple analysis tools and the analysis of Ethereum smart contracts. We used SmartBugs to execute the 9 automated analysis tools on the two datasets. In total, we ran 428,337 analyses that took approximately 564 days and 3 hours, being the largest experimental setup to date both in the number of tools and in execution time. We found that only 42% of the vulnerabilities from our annotated dataset are detected by all the tools, with the tool Mythril having the higher accuracy (27%). When considering the largest dataset, we observed that 97% of contracts are tagged as vulnerable, thus suggesting a considerable number of false positives. Indeed, only a small number of vulnerabilities (and of only two categories) were detected simultaneously by four or more tools.