# Syntax- and Compilation-Preserving Evasion of LLM Vulnerability Detectors

arXiv: https://arxiv.org/abs/2602.00305 · HTML: https://mlsanity.com/paper/2602.00305/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 29 references that are in the map.
- First posted on arXiv 30 Jan 2026.

## Builds on (most cited first, 19 of 19)

- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Universal Adversarial Triggers for Attacking and Analyzing NLP](https://mlsanity.com/paper/1908.07125/) (2019)
- [Tree of Attacks: Jailbreaking Black-Box LLMs Automatically](https://mlsanity.com/paper/2312.02119/) (2023)
- [SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks](https://mlsanity.com/paper/2310.03684/) (2023)
- [Vulnerability Detection with Code Language Models: How Far Are We?](https://mlsanity.com/paper/2403.18624/) (2024)
- [AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models](https://mlsanity.com/paper/2310.15140/) (2023)
- [DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detection](https://mlsanity.com/paper/2304.00409/) (2023)
- [Natural Attack for Pre-trained Models of Code](https://mlsanity.com/paper/2201.08698/) (2022)
- [Adversarial Examples for Models of Code](https://mlsanity.com/paper/1910.07517/) (2019)
- [Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilities](https://mlsanity.com/paper/2311.16169/) (2023)
- [garak: A Framework for Security Probing Large Language Models](https://mlsanity.com/paper/2406.11036/) (2024)
- [The Jailbreak Tax: How Useful are Your Jailbreak Outputs?](https://mlsanity.com/paper/2504.10694/) (2025)
- [Benchmarking Misuse Mitigation Against Covert Adversaries](https://mlsanity.com/paper/2506.06414/) (2025)
- [Are Large Language Models Robust in Understanding Code Against Semantics-Preserving Mutations?](https://mlsanity.com/paper/2505.10443/) (2025)
- [Evaluate-and-Purify: Fortifying Code Language Models Against Adversarial Attacks Using LLM-as-a-Judge](https://mlsanity.com/paper/2504.19730/) (2025)
- [HogVul: Black-box Adversarial Code Generation Framework Against LM-based Vulnerability Detectors](https://mlsanity.com/paper/2601.05587/) (2026)
- [NatGVD: Natural Adversarial Example Attack towards Graph-based Vulnerability Detection](https://mlsanity.com/paper/2510.04987/) (2025)

## Built on by (most cited first, 1 of 1)

- [ALIBI: Adaptive Agentic Attacks on LLM-Based Vulnerability Detectors via Adversarial Code Comments](https://mlsanity.com/paper/2607.24964/) (2026)

## Abstract (arXiv)

LLM-based vulnerability detectors are increasingly deployed in CI/CD security gating, yet their resilience to evasion under syntax- and compilation-preserving edits remains poorly understood. We evaluate five attack variants spanning four carrier families of behavior-preserving code transformations on a unified C/C++ benchmark (N=5000) and introduce Complete Resistance (CR), measuring the fraction of correctly detected vulnerabilities that withstand all attack variants. Our findings reveal a significant robustness gap: models achieving 70%+ clean recall exhibit CR as low as 0.12%, meaning over 87% of detected vulnerabilities can be evaded by at least one syntax-preserving edit. Universal adversarial strings optimized on a 14B surrogate transfer effectively to black-box APIs including GPT-4o, while on-target optimization further amplifies evasion (up to 92.5% ASR). These results indicate that clean benchmark accuracy alone is insufficient as a security guarantee for deployed vulnerability detectors.