# Efficient Avoidance of Vulnerabilities in Auto-completed Smart Contract Code Using Vulnerability-constrained Decoding

arXiv: https://arxiv.org/abs/2309.09826 · HTML: https://mlsanity.com/paper/2309.09826/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 44 references that are in the map.
- First posted on arXiv 18 Sep 2023.

## Builds on (most cited first, 26 of 26)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [code2vec: Learning Distributed Representations of Code](https://mlsanity.com/paper/1803.09473/) (2018)
- [Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search](https://mlsanity.com/paper/1704.07138/) (2017)
- [Beam Search Strategies for Neural Machine Translation](https://mlsanity.com/paper/1702.01806/) (2017)
- [Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation](https://mlsanity.com/paper/1804.06609/) (2018)
- [The Adverse Effects of Code Duplication in Machine Learning Models of Code](https://mlsanity.com/paper/1812.06469/) (2018)
- [Execution-based Code Generation using Deep Reinforcement Learning](https://mlsanity.com/paper/2301.13816/) (2023)
- [Slither: A Static Analysis Framework For Smart Contracts](https://mlsanity.com/paper/1908.09878/) (2019)
- [How Secure is Code Generated by ChatGPT?](https://mlsanity.com/paper/2304.09655/) (2023)
- [PanGu-Coder: Program Synthesis with Function-Level Language Modeling](https://mlsanity.com/paper/2207.11280/) (2022)
- [Empirical Review of Automated Analysis Tools on 47,587 Ethereum Smart Contracts](https://mlsanity.com/paper/1910.10601/) (2019)
- [Beyond the Safeguards: Exploring the Security Risks of ChatGPT](https://mlsanity.com/paper/2305.08005/) (2023)
- [ContractFuzzer: Fuzzing Smart Contracts for Vulnerability Detection](https://mlsanity.com/paper/1807.03932/) (2018)
- [How Effective are Smart Contract Analysis Tools? Evaluating Smart Contract Static Analysis Tools Using Bug Injection](https://mlsanity.com/paper/2005.11613/) (2020)
- [sFuzz: An Efficient Adaptive Fuzzer for Solidity Smart Contracts](https://mlsanity.com/paper/2004.08563/) (2020)
- [Detecting Gender Bias in Transformer-based Models: A Case Study on BERT](https://mlsanity.com/paper/2110.15733/) (2021)
- [Secure-by-Construction Synthesis of Cyber-Physical Systems](https://mlsanity.com/paper/2202.06677/) (2022)

## Built on by (most cited first, 14 of 14)

- [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024)
- [Constrained Decoding for Secure Code Generation](https://mlsanity.com/paper/2405.00218/) (2024)
- [From Vulnerabilities to Remediation: A Systematic Literature Review of LLMs in Code Security](https://mlsanity.com/paper/2412.15004/) (2024)
- [Smart-LLaMA: Two-Stage Post-Training of Large Language Models for Smart Contract Vulnerability Detection and Explanation](https://mlsanity.com/paper/2411.06221/) (2024)
- [SCALM: Detecting Bad Practices in Smart Contracts Through LLMs](https://mlsanity.com/paper/2502.04347/) (2025)
- [Smart-LLaMA-DPO: Reinforced Large Language Model for Explainable Smart Contract Vulnerability Detection](https://mlsanity.com/paper/2506.18245/) (2025)
- [Blockchain Meets LLMs: A Living Survey on Bidirectional Integration](https://mlsanity.com/paper/2411.16809/) (2024)
- [CodeBC: A More Secure Large Language Model for Smart Contract Code Generation in Blockchain](https://mlsanity.com/paper/2504.21043/) (2025)
- [DISL: Fueling Research with A Large Dataset of Solidity Smart Contracts](https://mlsanity.com/paper/2403.16861/) (2024)
- [Towards Secure and Explainable Smart Contract Generation with Security-Aware Group Relative Policy Optimization](https://mlsanity.com/paper/2509.09942/) (2025)
- [Towards Automated Smart Contract Generation: Evaluation, Benchmarking, and Retrieval-Augmented Repair](https://mlsanity.com/paper/2503.01098/) (2025)
- [Learning to Guarantee Type Correctness in Code Generation through Type-Guided Program Synthesis](https://mlsanity.com/paper/2510.10216/) (2025)
- [STELP: Secure Transpilation and Execution of LLM-Generated Programs](https://mlsanity.com/paper/2601.05467/) (2026)
- [LLMs + Security = Trouble](https://mlsanity.com/paper/2602.08422/) (2026)

## Abstract (arXiv)

Auto-completing code enables developers to speed up coding significantly. Recent advances in transformer-based large language model (LLM) technologies have been applied to code synthesis. However, studies show that many of such synthesized codes contain vulnerabilities. We propose a novel vulnerability-constrained decoding approach to reduce the amount of vulnerable code generated by such models. Using a small dataset of labeled vulnerable lines of code, we fine-tune an LLM to include vulnerability labels when generating code, acting as an embedded classifier. Then, during decoding, we deny the model to generate these labels to avoid generating vulnerable code. To evaluate the method, we chose to automatically complete Ethereum Blockchain smart contracts (SCs) as the case study due to the strict requirements of SC security. We first fine-tuned the 6-billion-parameter GPT-J model using 186,397 Ethereum SCs after removing the duplication from 2,217,692 SCs. The fine-tuning took more than one week using ten GPUs. The results showed that our fine-tuned model could synthesize SCs with an average BLEU (BiLingual Evaluation Understudy) score of 0.557. However, many codes in the auto-completed SCs were vulnerable. Using the code before the vulnerable line of 176 SCs containing different types of vulnerabilities to auto-complete the code, we found that more than 70% of the auto-completed codes were insecure. Thus, we further fine-tuned the model on other 941 vulnerable SCs containing the same types of vulnerabilities and applied vulnerability-constrained decoding. The fine-tuning took only one hour with four GPUs. We then auto-completed the 176 SCs again and found that our approach could identify 62% of the code to be generated as vulnerable and avoid generating 67% of them, indicating the approach could efficiently and effectively avoid vulnerabilities in the auto-completed code.