Syntax- and Compilation-Preserving Evasion of LLM Vulnerability Detectors
Not on the map. Why?
Abstract (arXiv)
LLM-based vulnerability detectors are increasingly deployed in CI/CD security gating, yet their resilience to evasion under syntax- and compilation-preserving edits remains poorly understood. We evaluate five attack variants spanning four carrier families of behavior-preserving code transformations on a unified C/C++ benchmark (N=5000) and introduce Complete Resistance (CR), measuring the fraction of correctly detected vulnerabilities that withstand all attack variants. Our findings reveal a significant robustness gap: models achieving 70%+ clean recall exhibit CR as low as 0.12%, meaning over 87% of detected vulnerabilities can be evaded by at least one syntax-preserving edit. Universal adversarial strings optimized on a 14B surrogate transfer effectively to black-box APIs including GPT-4o, while on-target optimization further amplifies evasion (up to 92.5% ASR). These results indicate that clean benchmark accuracy alone is insufficient as a security guarantee for deployed vulnerability detectors.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 19 of its 29 references that are in the map. First posted on arXiv 30 Jan 2026.
Builds on
19 of 29 references.
References · 19
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
- 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
- 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709
- 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
- 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attackscited by 434
- 2024Vulnerability Detection with Code Language Models: How Far Are We?cited by 133
- 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 102
- 2023DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning Based Vulnerability Detectioncited by 98
- 2022Natural Attack for Pre-trained Models of Codecited by 72
- 2019Adversarial Examples for Models of Codecited by 61
- 2023Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilitiescited by 58
7 more
- 2024garak: A Framework for Security Probing Large Language Modelscited by 48
- 2025The Jailbreak Tax: How Useful are Your Jailbreak Outputs?cited by 26
- 2025Benchmarking Misuse Mitigation Against Covert Adversariescited by 8
- 2025Are Large Language Models Robust in Understanding Code Against Semantics-Preserving Mutations?cited by 7
- 2025Evaluate-and-Purify: Fortifying Code Language Models Against Adversarial Attacks Using LLM-as-a-Judgecited by 4
- 2026HogVul: Black-box Adversarial Code Generation Framework Against LM-based Vulnerability Detectorscited by 3
- 2025NatGVD: Natural Adversarial Example Attack towards Graph-based Vulnerability Detectioncited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.