# An Empirical Study of AI-based Smart Contract Creation

arXiv: https://arxiv.org/abs/2308.02955 · HTML: https://mlsanity.com/paper/2308.02955/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 44 references that are in the map.
- First posted on arXiv 5 Aug 2023.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [Exploring the Benefits of Training Expert Language Models over Instruction Tuning](https://mlsanity.com/paper/2302.03202/) (2023)
- [Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT](https://mlsanity.com/paper/2304.10778/) (2023)
- [On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilot](https://mlsanity.com/paper/2302.00438/) (2023)
- [CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Models](https://mlsanity.com/paper/2302.04012/) (2023)
- [Learning Deep Semantics for Test Completion](https://mlsanity.com/paper/2302.10166/) (2023)
- [Targeted Phishing Campaigns using Large Scale Language Models](https://mlsanity.com/paper/2301.00665/) (2022)
- [Harvey: A Greybox Fuzzer for Smart Contracts](https://mlsanity.com/paper/1905.06944/) (2019)
- [Lessons Learned from Blockchain Applications of Trusted Execution Environments and Implications for Future Research](https://mlsanity.com/paper/2203.12724/) (2022)

## Built on by (most cited first, 4 of 4)

- [Efficacy of Various Large Language Models in Generating Smart Contracts](https://mlsanity.com/paper/2407.11019/) (2024)
- [Teaching Machines to Code: Smart Contract Translation with LLMs](https://mlsanity.com/paper/2403.09740/) (2024)
- [LogBabylon: A Unified Framework for Cross-Log File Integration and Analysis](https://mlsanity.com/paper/2412.12364/) (2024)
- [On LLM-Assisted Generation of Smart Contracts from Business Processes](https://mlsanity.com/paper/2507.23087/) (2025)

## Abstract (arXiv)

The introduction of large language models (LLMs) like ChatGPT and Google Palm2 for smart contract generation seems to be the first well-established instance of an AI pair programmer. LLMs have access to a large number of open-source smart contracts, enabling them to utilize more extensive code in Solidity than other code generation tools. Although the initial and informal assessments of LLMs for smart contract generation are promising, a systematic evaluation is needed to explore the limits and benefits of these models. The main objective of this study is to assess the quality of generated code provided by LLMs for smart contracts. We also aim to evaluate the impact of the quality and variety of input parameters fed to LLMs. To achieve this aim, we created an experimental setup for evaluating the generated code in terms of validity, correctness, and efficiency. Our study finds crucial evidence of security bugs getting introduced in the generated smart contracts as well as the overall quality and correctness of the code getting impacted. However, we also identified the areas where it can be improved. The paper also proposes several potential research directions to improve the process, quality and safety of generated smart contract codes.