# How Effective are Smart Contract Analysis Tools? Evaluating Smart Contract Static Analysis Tools Using Bug Injection

arXiv: https://arxiv.org/abs/2005.11613 · HTML: https://mlsanity.com/paper/2005.11613/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 May 2020.

## Built on by (most cited first, 20 of 20)

- [LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights](https://mlsanity.com/paper/2502.07049/) (2025)
- [Efficient Avoidance of Vulnerabilities in Auto-completed Smart Contract Code Using Vulnerability-constrained Decoding](https://mlsanity.com/paper/2309.09826/) (2023)
- [Exploring the Security Threats of Knowledge Base Poisoning in Retrieval-Augmented Code Generation](https://mlsanity.com/paper/2502.03233/) (2025)
- [Large Language Models for Security Operations Centers: A Comprehensive Survey](https://mlsanity.com/paper/2509.10858/) (2025)
- [MANDO: Multi-Level Heterogeneous Graph Embeddings for Fine-Grained Detection of Smart Contract Vulnerabilities](https://mlsanity.com/paper/2208.13252/) (2022)
- [VulnSense: Efficient Vulnerability Detection in Ethereum Smart Contracts by Multimodal Learning with Graph Neural Network and Language Model](https://mlsanity.com/paper/2309.08474/) (2023)
- [An Automated Vulnerability Detection Framework for Smart Contracts](https://mlsanity.com/paper/2301.08824/) (2023)
- [Usenix'23 Extended Version: Smart Learning to Find Dumb Contracts](https://mlsanity.com/paper/2304.10726/) (2023)
- [Evaluation of ChatGPT's Smart Contract Auditing Capabilities Based on Chain of Thought](https://mlsanity.com/paper/2402.12023/) (2024)
- [Vulnerability Detection in Ethereum Smart Contracts via Machine Learning: A Qualitative Analysis](https://mlsanity.com/paper/2407.18639/) (2024)
- [Detection Made Easy: Potentials of Large Language Models for Solidity Vulnerabilities](https://mlsanity.com/paper/2409.10574/) (2024)
- [DISL: Fueling Research with A Large Dataset of Solidity Smart Contracts](https://mlsanity.com/paper/2403.16861/) (2024)
- [AI-Based Vulnerability Analysis of NFT Smart Contracts](https://mlsanity.com/paper/2504.16113/) (2025)
- [UEChecker: Detecting Unchecked External Call Vulnerabilities in DApps via Graph Analysis](https://mlsanity.com/paper/2508.01343/) (2025)
- [Schooling to Exploit Foolish Contracts](https://mlsanity.com/paper/2304.10737/) (2023)
- [Towards Understanding Deep Learning Model in Image Recognition via Coverage Test](https://mlsanity.com/paper/2505.08814/) (2025)
- [LLMs as verification oracles for Solidity](https://mlsanity.com/paper/2509.19153/) (2025)
- [ORACAL: A Robust and Explainable Multimodal Framework for Smart Contract Vulnerability Detection with Causal Graph Enrichment](https://mlsanity.com/paper/2603.28128/) (2026)
- [CHAINTRIX: A multi-pipeline LLM-augmented framework for automated smart-contract security auditing](https://mlsanity.com/paper/2605.09350/) (2026)
- [Automated Vulnerability Injection in Smart Contracts Using Large Language Models](https://mlsanity.com/paper/2609.02624/) (2026)

## Abstract (arXiv)

Security attacks targeting smart contracts have been on the rise, which have led to financial loss and erosion of trust. Therefore, it is important to enable developers to discover security vulnerabilities in smart contracts before deployment. A number of static analysis tools have been developed for finding security bugs in smart contracts. However, despite the numerous bug-finding tools, there is no systematic approach to evaluate the proposed tools and gauge their effectiveness. This paper proposes SolidiFI, an automated and systematic approach for evaluating smart contract static analysis tools. SolidiFI is based on injecting bugs (i.e., code defects) into all potential locations in a smart contract to introduce targeted security vulnerabilities. SolidiFI then checks the generated buggy contract using the static analysis tools, and identifies the bugs that the tools are unable to detect (false-negatives) along with identifying the bugs reported as false-positives. SolidiFI is used to evaluate six widely-used static analysis tools, namely, Oyente, Securify, Mythril, SmartCheck, Manticore and Slither, using a set of 50 contracts injected by 9369 distinct bugs. It finds several instances of bugs that are not detected by the evaluated tools despite their claims of being able to detect such bugs, and all the tools report many false positives