IssueTrojanBench: Benchmarking AI Coding Agents Against Malicious Issue Requests
On the map: LLM agents and tool use → Agent threats, guardrails and governance
also in 1 more
Abstract (arXiv)
AI coding agents powered by LLMs are increasingly integrated into real-world software development, where they generate, edit, and execute code with autonomous access to local files and tools. Coding agents inherit security risks from both the LLM backbone, where adversarial prompts, poisoned training data, and backdoor triggers can cause models to emit insecure or attacker-chosen code, and their agentic architecture, where tool-using autonomy enables induced misuse of external APIs, data exfiltration, and persistent compromise of development environments. This paper presents a systematic evaluation of malicious issue requests against state-of-the-art coding agents (Cursor, Claude Code, and Codex Desktop), powered by two major model families (OpenAI GPT-5.3 Codex/GPT-5.4 and Anthropic Sonnet 4.6). Our novel benchmark IssueTrojanBench contains malicious issues that are constructed based on four novel attack categories (i.e., embedded as malicious instructions in issues), six delivery vectors (e.g., PDF, or issue comment), and further augmented by perturbations. Our results reveal critical vulnerabilities in the as-deployed modern coding agents, i.e., 66.5% of the malicious issues from IssueTrojanBench penetrate all the guardrails (agent- and LLM-level) of coding agents. Our further analysis shows that rejection is almost entirely from LLMs rather than the agent frameworks, with GPT models broadly vulnerable and Sonnet 4.6 exhibiting more selective, risk-aware blocking of high-impact actions. Our evaluation also highlights that the current agent-level defense strategy offers limited additional protection for coding agents. Our findings highlight the urgent need for stronger agent- and model-level safety mechanisms to protect AI coding agents.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 23 of its 32 references that are in the map. First posted on arXiv 22 Jul 2026.
Builds on
23 of 32 references.
Closest ancestors · 13
Same direction, newest first
- 2026Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystems24 Jan
- 2026Defense Against Indirect Prompt Injection via Tool Result Parsing8 Jan
- 2025"Your AI, My Shell": Demystifying Prompt Injection Attacks on Agentic AI Coding Editors26 Sep
- 2025IPIGuard: A Novel Tool Dependency Graph-Based Defense Against Indirect Prompt Injection in LLM Agents21 Aug
- 2025When Developer Aid Becomes Security Debt: A Systematic Analysis of Insecure Behaviors in LLM Coding Agents12 Jul
- 2025From Prompt Injections to Protocol Exploits: Threats in LLM-Powered AI Agents Workflows29 Jun
- 2025Design Patterns for Securing LLM Agents against Prompt Injections10 Jun
- 2025LlamaFirewall: An open source guardrail system for building secure AI agents6 May
- 2025Defeating Prompt Injections by Design24 Mar
- 2024The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents21 Dec
- 2024Imprompter: Tricking LLM Agents into Improper Tool Use19 Oct
- 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents3 Oct
1 more
Other references · 10
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
- 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
- 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 375
- 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 267
- 2024SG-Bench: Evaluating LLM Safety Generalization Across Diverse Tasks and Prompt Typescited by 60
- 2023DeceptPrompt: Exploiting LLM-driven Code Generation via Adversarial Natural Language Instructionscited by 33
- 2026Agentic Much? Adoption of Coding Agents on GitHubcited by 32
- 2025A Survey on Backdoor Threats in Large Language Models (LLMs): Attacks, Defenses, and Evaluationscited by 27
- 2024MaPPing Your Model: Assessing the Impact of Adversarial Attacks on LLM-based Programming Assistantscited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.
- 2026LongPIBench: A Long-Context Benchmark for Prompt Injection28 Aug 2026
Similar, not cited
Close in wording, no citation either way
- 2026Security in the Age of AI Teammates: An Empirical Study of Agentic Pull Requests on GitHub
- 2026Towards a Risk Assessment of Malicious Skill Files in Coding Agents
- 2025Takedown: How It's Done in Modern Coding Agent Exploits
- 2025Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks
- 2026MalSkillBench: A Runtime-Verified Benchmark of Malicious Agent Skills
- 2025SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios
- 2025Analyzing Code Injection Attacks on LLM-based Multi-Agent Systems in Software Development
- 2026CIBER: A Comprehensive Benchmark for Security Evaluation of Code Interpreter Agents