# EaTVul: ChatGPT-based Evasion Attack Against Software Vulnerability Detection

arXiv: https://arxiv.org/abs/2407.19216 · HTML: https://mlsanity.com/paper/2407.19216/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 58 references that are in the map.
- First posted on arXiv 27 Jul 2024.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [BERT-ATTACK: Adversarial Attack Against BERT Using BERT](https://mlsanity.com/paper/2004.09984/) (2020)
- [Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods](https://mlsanity.com/paper/1911.02508/) (2019)
- [BAE: BERT-based Adversarial Examples for Text Classification](https://mlsanity.com/paper/2004.01970/) (2020)
- [You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion](https://mlsanity.com/paper/2007.02220/) (2020)
- [Examining Zero-Shot Vulnerability Repair with Large Language Models](https://mlsanity.com/paper/2112.02125/) (2021)
- [OpenAttack: An Open-source Textual Adversarial Attack Toolkit](https://mlsanity.com/paper/2009.09191/) (2020)
- [Generating Adversarial Malware Examples for Black-Box Attacks Based on GAN](https://mlsanity.com/paper/1702.05983/) (2017)
- [Natural Attack for Pre-trained Models of Code](https://mlsanity.com/paper/2201.08698/) (2022)
- [Adversarial Examples for Models of Code](https://mlsanity.com/paper/1910.07517/) (2019)
- [Adversarial Malware Binaries: Evading Deep Learning for Malware Detection in Executables](https://mlsanity.com/paper/1803.04173/) (2018)
- [Adversarial Machine Learning in Image Classification: A Survey Towards the Defender's Perspective](https://mlsanity.com/paper/2009.03728/) (2020)
- [Semantic Robustness of Models of Source Code](https://mlsanity.com/paper/2002.03043/) (2020)
- [Backdoors in Neural Models of Source Code](https://mlsanity.com/paper/2006.06841/) (2020)
- [Greedy Attack and Gumbel Attack: Generating Adversarial Examples for Discrete Data](https://mlsanity.com/paper/1805.12316/) (2018)
- [LineVD: Statement-level Vulnerability Detection using Graph Neural Networks](https://mlsanity.com/paper/2203.05181/) (2022)
- [Generating Adversarial Computer Programs using Optimized Obfuscations](https://mlsanity.com/paper/2103.11882/) (2021)
- [Adversarial Robustness of Deep Code Comment Generation](https://mlsanity.com/paper/2108.00213/) (2021)
- [Query-Efficient and Scalable Black-Box Adversarial Attacks on Discrete Sequential Data via Bayesian Optimization](https://mlsanity.com/paper/2206.08575/) (2022)
- [A Survey on Data-driven Software Vulnerability Assessment and Prioritization](https://mlsanity.com/paper/2107.08364/) (2021)
- [Arms Race in Adversarial Malware Detection: A Survey](https://mlsanity.com/paper/2005.11671/) (2020)
- [ConAML: Constrained Adversarial Machine Learning for Cyber-Physical Systems](https://mlsanity.com/paper/2003.05631/) (2020)
- [Transformer-based Vulnerability Detection in Code at EditTime: Zero-shot, Few-shot, or Fine-tuning?](https://mlsanity.com/paper/2306.01754/) (2023)
- [Asteria: Deep Learning-based AST-Encoding for Cross-platform Binary Code Similarity Detection](https://mlsanity.com/paper/2108.06082/) (2021)

## Built on by (most cited first, 2 of 2)

- [False Alarms, Real Damage: Adversarial Attacks Using LLM-based Models on Text-based Cyber Threat Intelligence Systems](https://mlsanity.com/paper/2507.06252/) (2025)
- [Effective Code Membership Inference for Code Completion Models via Adversarial Prompts](https://mlsanity.com/paper/2511.15107/) (2025)

## Abstract (arXiv)

Recently, deep learning has demonstrated promising results in enhancing the accuracy of vulnerability detection and identifying vulnerabilities in software. However, these techniques are still vulnerable to attacks. Adversarial examples can exploit vulnerabilities within deep neural networks, posing a significant threat to system security. This study showcases the susceptibility of deep learning models to adversarial attacks, which can achieve 100% attack success rate (refer to Table 5). The proposed method, EaTVul, encompasses six stages: identification of important samples using support vector machines, identification of important features using the attention mechanism, generation of adversarial data based on these features using ChatGPT, preparation of an adversarial attack pool, selection of seed data using a fuzzy genetic algorithm, and the execution of an evasion attack. Extensive experiments demonstrate the effectiveness of EaTVul, achieving an attack success rate of more than 83% when the snippet size is greater than 2. Furthermore, in most cases with a snippet size of 4, EaTVul achieves a 100% attack success rate. The findings of this research emphasize the necessity of robust defenses against adversarial attacks in software vulnerability detection.