# SmartEval: A Benchmark for Evaluating LLM-Generated Smart Contracts from Natural Language Specifications

arXiv: https://arxiv.org/abs/2605.09610 · HTML: https://mlsanity.com/paper/2605.09610/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 36 references that are in the map.
- First posted on arXiv 10 May 2026.

## Builds on (most cited first, 10 of 10)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Slither: A Static Analysis Framework For Smart Contracts](https://mlsanity.com/paper/1908.09878/) (2019)
- [A Survey on Ethereum Systems Security: Vulnerabilities, Attacks and Defenses](https://mlsanity.com/paper/1908.04507/) (2019)
- [An Overview on Smart Contracts: Challenges, Advances and Platforms](https://mlsanity.com/paper/1912.10370/) (2019)
- [Designing Secure Ethereum Smart Contracts: A Finite State Machine Based Approach](https://mlsanity.com/paper/1711.09327/) (2017)
- [Detecting Buggy Contracts via Smart Testing](https://mlsanity.com/paper/2409.04597/) (2024)
- [Decentralized Finance: Protocols, Risks, and Governance](https://mlsanity.com/paper/2312.01018/) (2023)

## Abstract (arXiv)

We introduce SmartEval, a benchmark for systematically evaluating the quality of Solidity smart contracts generated by large language models (LLMs) from natural language specifications. SmartEval provides a corpus of 9,000 generated contracts paired with expert-written ground-truth implementations drawn from the FSMSCG dataset, a five-dimensional evaluation rubric covering functional completeness, variable fidelity, state-machine correctness, business-logic fidelity, and code quality, and a reproducible generation-and-evaluation pipeline. To validate the benchmark's reliability, we conduct three independent empirical studies: a five-condition ablation study (N=300 per condition) isolating the contribution of each pipeline component, a human expert evaluation by three Columbia University PhD researchers confirming automated scores align with expert judgment to within 0.34 points, and external security analysis via the Slither static analyzer confirming 79.4% agreement between the LLM auditor and a non-LLM rule-based tool. Systematic analysis of 9,000 generated contracts reveals characteristic failure modes (logic omissions at 35.3%, state transition errors at 23.4%, and complexity-driven degradation) and quantifies a +8.29 composite-score advantage of generated contracts over ground-truth implementations, attributable to LLMs' literal specification-following behavior. SmartEval establishes a reproducible, validated foundation for empirical research on LLM smart contract synthesis quality, with all data, evaluation code, and generated contracts publicly released.