# ALIBI: Adaptive Agentic Attacks on LLM-Based Vulnerability Detectors via Adversarial Code Comments

arXiv: https://arxiv.org/abs/2607.24964 · HTML: https://mlsanity.com/paper/2607.24964/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 35 of its 99 references that are in the map.
- First posted on arXiv 27 Jul 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 30 of 35)

- [The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions](https://mlsanity.com/paper/2404.13208/) (2024)
- [Defending Against Indirect Prompt Injection Attacks With Spotlighting](https://mlsanity.com/paper/2403.14720/) (2024)
- [You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion](https://mlsanity.com/paper/2007.02220/) (2020)
- [Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents](https://mlsanity.com/paper/2503.00061/) (2025)
- [IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities](https://mlsanity.com/paper/2405.17238/) (2024)
- [Natural Attack for Pre-trained Models of Code](https://mlsanity.com/paper/2201.08698/) (2022)
- [Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAG](https://mlsanity.com/paper/2406.11147/) (2024)
- ["Do Not Mention This to the User": Detecting and Understanding Malicious Agent Skills in the Wild](https://mlsanity.com/paper/2602.06547/) (2026)
- [RepoAudit: An Autonomous LLM-Agent for Repository-Level Code Auditing](https://mlsanity.com/paper/2501.18160/) (2025)
- [LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights](https://mlsanity.com/paper/2502.07049/) (2025)
- [Supply-Chain Poisoning Attacks Against LLM Coding Agent Skill Ecosystems](https://mlsanity.com/paper/2604.03081/) (2026)
- [Backdoors in Neural Models of Source Code](https://mlsanity.com/paper/2006.06841/) (2020)
- [Defending Against Prompt Injection with DataFilter](https://mlsanity.com/paper/2510.19207/) (2025)
- [Agentic Much? Adoption of Coding Agents on GitHub](https://mlsanity.com/paper/2601.18341/) (2026)
- [Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystems](https://mlsanity.com/paper/2601.17548/) (2026)
- [MegaVul: A C/C++ Vulnerability Dataset with Comprehensive Code Representation](https://mlsanity.com/paper/2406.12415/) (2024)
- [LLM Can be a Dangerous Persuader: Empirical Study of Persuasion Safety in Large Language Models](https://mlsanity.com/paper/2504.10430/) (2025)
- [LLMDFA: Analyzing Dataflow in Code with Large Language Models](https://mlsanity.com/paper/2402.10754/) (2024)
- [Let the Trial Begin: A Mock-Court Approach to Vulnerability Detection using LLM-Based Agents](https://mlsanity.com/paper/2505.10961/) (2025)
- [SCALE: Constructing Structured Natural Language Comment Trees for Software Vulnerability Detection](https://mlsanity.com/paper/2403.19096/) (2024)
- [CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoning](https://mlsanity.com/paper/2504.14119/) (2025)
- [PISanitizer: Preventing Prompt Injection to Long-Context LLMs via Prompt Sanitization](https://mlsanity.com/paper/2511.10720/) (2025)
- [Reducing False Positives in Static Bug Detection with LLMs: An Empirical Study in Industry](https://mlsanity.com/paper/2601.18844/) (2026)
- [Sifting the Noise: A Comparative Study of LLM Agents in Vulnerability False Positive Filtering](https://mlsanity.com/paper/2601.22952/) (2026)
- [Minimizing False Positives in Static Bug Detection via LLM-Enhanced Path Feasibility Analysis](https://mlsanity.com/paper/2506.10322/) (2025)
- [When Names Disappear: Revealing What LLMs Actually Understand About Code](https://mlsanity.com/paper/2510.03178/) (2025)
- [From SFT to RL: Demystifying the Post-Training Pipeline for LLM-based Vulnerability Detection](https://mlsanity.com/paper/2602.14012/) (2026)
- [Secret Breach Detection in Source Code with Large Language Models](https://mlsanity.com/paper/2504.18784/) (2025)
- [CoTDeceptor:Adversarial Code Obfuscation Against CoT-Enhanced LLM Code Agents](https://mlsanity.com/paper/2512.21250/) (2025)
- [LLM-based Vulnerability Detection at Project Scale: An Empirical Study](https://mlsanity.com/paper/2601.19239/) (2026)

## Abstract (arXiv)

Large language models are increasingly deployed for security-sensitive tasks such as vulnerability detection and code review. Their reliance on natural-language context embedded in source code exposes a previously underexplored attack surface: adversarial comments that can influence a detector's reasoning without changing program behavior. We study LLM-based vulnerability detectors against a new adversary: a coding agent that implements new functionality, deliberately introduces vulnerabilities, and strategically inserts adversarial source-code comments to evade detection. We present ALIBI, an automated adaptive black-box attack framework that generates and iteratively refines adversarial comments using detector reasoning and feedback. We transform real-world vulnerability-fixing commits into coding tasks and evaluate four representative LLM-based vulnerability detectors, ranging from specialized open-weight reasoning models to frontier multi-agent systems. All evaluated detectors are highly vulnerable: attack success rates exceed 90% across 125 real-world null-pointer dereference vulnerabilities, reaching 100% on one system. The framework also generalizes beyond this vulnerability class. Adversarial comments steering detector reasoning or fabricating external tool results prove most effective, while iterative refinement based on detector feedback further increases attack success. Finally, prompt-level defenses provide limited robustness against adaptive attacks, whereas architectural isolation and pre-detector comment sanitization substantially improve resilience. Our findings expose a fundamental attack surface in current LLM-based vulnerability detectors and motivate security-aware designs that carefully calibrate trust between natural-language context and program evidence.