# An Extensive Study on Adversarial Attack against Pre-trained Models of Code arXiv: https://arxiv.org/abs/2311.07553 ยท HTML: https://mlsanity.com/paper/2311.07553/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 23 of its 47 references that are in the map. - First posted on arXiv 13 Nov 2023. ## Builds on (most cited first, 23 of 23) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020) - [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021) - [CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation](https://mlsanity.com/paper/2102.04664/) (2021) - [CodeSearchNet Challenge: Evaluating the State of Semantic Code Search](https://mlsanity.com/paper/1909.09436/) (2019) - [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020) - [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020) - [UniXcoder: Unified Cross-Modal Pre-training for Code Representation](https://mlsanity.com/paper/2203.03850/) (2022) - [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021) - [A Unified Survey on Anomaly, Novelty, Open-Set, and Out-of-Distribution Detection: Solutions and Future Challenges](https://mlsanity.com/paper/2110.14051/) (2021) - [OpenAttack: An Open-source Textual Adversarial Attack Toolkit](https://mlsanity.com/paper/2009.09191/) (2020) - [Natural Attack for Pre-trained Models of Code](https://mlsanity.com/paper/2201.08698/) (2022) - [Detecting Code Clones with Graph Neural Networkand Flow-Augmented Abstract Syntax Tree](https://mlsanity.com/paper/2002.08653/) (2020) - [Adversarial Examples for Models of Code](https://mlsanity.com/paper/1910.07517/) (2019) - [On the Evaluation of Neural Code Summarization](https://mlsanity.com/paper/2107.07112/) (2021) - [Semantic Robustness of Models of Source Code](https://mlsanity.com/paper/2002.03043/) (2020) - [On the Generalizability of Neural Program Models with respect to Semantic-Preserving Program Transformations](https://mlsanity.com/paper/2008.01566/) (2020) - [No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligence](https://mlsanity.com/paper/2207.11680/) (2022) - [Generating Adversarial Computer Programs using Optimized Obfuscations](https://mlsanity.com/paper/2103.11882/) (2021) - [Adversarial Robustness of Deep Code Comment Generation](https://mlsanity.com/paper/2108.00213/) (2021) - [RoPGen: Towards Robust Code Authorship Attribution via Automatic Coding Style Transformation](https://mlsanity.com/paper/2202.06043/) (2022) - [A Search-Based Testing Framework for Deep Neural Networks of Source Code Embedding](https://mlsanity.com/paper/2101.07910/) (2021) - [An In-depth Study of Java Deserialization Remote-Code Execution Exploits and Vulnerabilities](https://mlsanity.com/paper/2208.08173/) (2022) ## Built on by (most cited first, 8 of 8) - [Generalization-Enhanced Code Vulnerability Detection via Multi-Task Instruction Fine-Tuning](https://mlsanity.com/paper/2406.03718/) (2024) - [Towards Reliable Evaluation of Neural Program Repair with Natural Robustness Testing](https://mlsanity.com/paper/2402.11892/) (2024) - [A Systematic Study of Code Obfuscation Against LLM-based Vulnerability Detection](https://mlsanity.com/paper/2512.16538/) (2025) - [XOXO: Stealthy Cross-Origin Context Poisoning Attacks against AI Coding Assistants](https://mlsanity.com/paper/2503.14281/) (2025) - [Evaluate-and-Purify: Fortifying Code Language Models Against Adversarial Attacks Using LLM-as-a-Judge](https://mlsanity.com/paper/2504.19730/) (2025) - [A Metamorphic Testing Perspective on Knowledge Distillation for Language Models of Code: Does the Student Deeply Mimic the Teacher?](https://mlsanity.com/paper/2511.05476/) (2025) - [Enhancing Large Language Models with Faster Code Preprocessing for Vulnerability Detection](https://mlsanity.com/paper/2505.05600/) (2025) - [On the Lexical Superstition of Large Language Models for Code Comprehension: Re-evaluation on Code of Low Lexical Quality](https://mlsanity.com/paper/2609.26388/) (2026) ## Abstract (arXiv) Transformer-based pre-trained models of code (PTMC) have been widely utilized and have achieved state-of-the-art performance in many mission-critical applications. However, they can be vulnerable to adversarial attacks through identifier substitution or coding style transformation, which can significantly degrade accuracy and may further incur security concerns. Although several approaches have been proposed to generate adversarial examples for PTMC, the effectiveness and efficiency of such approaches, especially on different code intelligence tasks, has not been well understood. To bridge this gap, this study systematically analyzes five state-of-the-art adversarial attack approaches from three perspectives: effectiveness, efficiency, and the quality of generated examples. The results show that none of the five approaches balances all these perspectives. Particularly, approaches with a high attack success rate tend to be time-consuming; the adversarial code they generate often lack naturalness, and vice versa. To address this limitation, we explore the impact of perturbing identifiers under different contexts and find that identifier substitution within for and if statements is the most effective. Based on these findings, we propose a new approach that prioritizes different types of statements for various tasks and further utilizes beam search to generate adversarial examples. Evaluation results show that it outperforms the state-of-the-art ALERT in terms of both effectiveness and efficiency while preserving the naturalness of the generated adversarial examples.