# ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agents

arXiv: https://arxiv.org/abs/2509.22830 · HTML: https://mlsanity.com/paper/2509.22830/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 25 of its 37 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)
- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 25 of 25)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs](https://mlsanity.com/paper/2401.06373/) (2024)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [On Adaptive Attacks to Adversarial Example Defenses](https://mlsanity.com/paper/2002.08347/) (2020)
- [InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents](https://mlsanity.com/paper/2403.02691/) (2024)
- [The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions](https://mlsanity.com/paper/2404.13208/) (2024)
- [GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models](https://mlsanity.com/paper/2508.06471/) (2025)
- [Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents](https://mlsanity.com/paper/2410.02644/) (2024)
- [Defending Against Indirect Prompt Injection Attacks With Spotlighting](https://mlsanity.com/paper/2403.14720/) (2024)
- [SecAlign: Defending Against Prompt Injection with Preference Optimization](https://mlsanity.com/paper/2410.05451/) (2024)
- [LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet](https://mlsanity.com/paper/2408.15221/) (2024)
- [Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents](https://mlsanity.com/paper/2503.00061/) (2025)
- [X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents](https://mlsanity.com/paper/2504.13203/) (2025)
- [PromptArmor: Simple yet Effective Prompt Injection Defenses](https://mlsanity.com/paper/2507.15219/) (2025)
- [Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?](https://mlsanity.com/paper/2403.06833/) (2024)
- [Foot-In-The-Door: A Multi-turn Jailbreak for LLMs](https://mlsanity.com/paper/2502.19820/) (2025)
- [ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templates](https://mlsanity.com/paper/2406.12935/) (2024)
- [On the Loss of Context-awareness in General Instruction Fine-tuning](https://mlsanity.com/paper/2411.02688/) (2024)

## Built on by (most cited first, 30 of 31)

- [Taming Various Privilege Escalation in LLM-Based Agent Systems: A Mandatory Access Control Framework](https://mlsanity.com/paper/2601.11893/) (2026)
- [AgentLAB: Benchmarking LLM Agents against Long-Horizon Attacks](https://mlsanity.com/paper/2602.16901/) (2026)
- [ClawGuard: A Runtime Security Framework for Tool-Augmented LLM Agents Against Indirect Prompt Injection](https://mlsanity.com/paper/2604.11790/) (2026)
- [Prompt Injection as Role Confusion](https://mlsanity.com/paper/2603.12277/) (2026)
- [SoK: Trust-Authorization Mismatch in LLM Agent Interactions](https://mlsanity.com/paper/2512.06914/) (2025)
- [Taxonomy, Evaluation and Exploitation of IPI-Centric LLM Agent Defense Frameworks](https://mlsanity.com/paper/2511.15203/) (2025)
- [What Matters For Safety Alignment?](https://mlsanity.com/paper/2601.03868/) (2026)
- [Safeguarding LLM Agents against Long-Horizon Threats via Shadow Memory](https://mlsanity.com/paper/2605.03228/) (2026)
- [IterInject: Indirect Prompt Injection Against LLM Agents via Feedback-Guided Iterative Optimization](https://mlsanity.com/paper/2605.24659/) (2026)
- [Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation](https://mlsanity.com/paper/2606.10749/) (2026)
- [Automating Agent Hijacking via Structural Template Injection](https://mlsanity.com/paper/2602.16958/) (2026)
- [Evaluating Performance Drift from Model Switching in Multi-Turn LLM Systems](https://mlsanity.com/paper/2603.03111/) (2026)
- [Inference-Time Backdoors via Chat Templates: From LLM Supply Chains to Agentic System Compromise](https://mlsanity.com/paper/2602.04653/) (2026)
- [ToolHazard: Scaling Adversarial Environments for Security Evaluation and Alignment of LLM-based Agents](https://mlsanity.com/paper/2608.11878/) (2026)
- [CoER: Defending against Adaptive Indirect Prompt Injection via Adversarial Co-Evolution and Refinement](https://mlsanity.com/paper/2609.07529/) (2026)
- [Security Attack and Defense Strategies for Autonomous Agent Frameworks: A Layered Review with OpenClaw as a Case Study](https://mlsanity.com/paper/2604.27464/) (2026)
- [On the Geometric Limits of Transformer Defenses against Obfuscation Attacks: Latent Embedding Collapse & Performance Robustness Gap](https://mlsanity.com/paper/2605.19159/) (2026)
- [Evaluating using Mock Tool Calls to Quarantine Untrusted Prompt Inputs](https://mlsanity.com/paper/2605.30521/) (2026)
- [Same Payload, Different Channel: Measuring Trust Asymmetry in Tool-Using Language Models](https://mlsanity.com/paper/2606.00566/) (2026)
- [Local LLM Agents as Vulnerable Runtimes:A Source-Code Audit of the Agent Runtime Layer](https://mlsanity.com/paper/2606.21071/) (2026)
- [Pigeonholing: how bad prompts hurt models, causing collapse and mistakes](https://mlsanity.com/paper/2606.24267/) (2026)
- [IHDec: Divergence-Steered Contrastive Decoding for Securing Multi-Turn Instruction Hierarchies](https://mlsanity.com/paper/2606.29960/) (2026)
- [Know Your Agent: Reconnaissance-Driven Pentesting of AI Agents](https://mlsanity.com/paper/2607.19837/) (2026)
- [Workspace Topology as an Attack Vector in Agentic Coding Assistants](https://mlsanity.com/paper/2608.14876/) (2026)
- [WebMCP-Phalanx: Enforcing and Characterizing Trust Boundaries for Browser-Integrated LLM Agents](https://mlsanity.com/paper/2608.24017/) (2026)
- [Will the User Ever Know? Covert Indirect Prompt Injection Attacks on Tool-Using LLM Agents](https://mlsanity.com/paper/2608.30362/) (2026)
- [Control-Token Injection Suppresses Chain-of-Thought and Defeats Reasoning-Based Oversight in Tool-Using Agents](https://mlsanity.com/paper/2609.27542/) (2026)
- [Prefilling the Reasoning Channel: Output-Prefix Attacks on Reasoning LLMs](https://mlsanity.com/paper/2609.29775/) (2026)
- [Epistemic Policy Divergence in Multi-Turn LLM Contamination: A Protocol-Gradient Investigation](https://mlsanity.com/paper/2609.35308/) (2026)
- [Same Bytes, Different Authority: Reserved-Token Representations in Chat-Template Prompt Injection](https://mlsanity.com/paper/2609.35932/) (2026)

## Abstract (arXiv)

The growing deployment of large language model (LLM) based agents that interact with external environments has created new attack surfaces for adversarial manipulation. One major threat is indirect prompt injection, where attackers embed malicious instructions in external environment output, causing agents to interpret and execute them as if they were legitimate prompts. While previous research has focused primarily on plain-text injection attacks, we find a significant yet underexplored vulnerability: LLMs' dependence on structured chat templates and their susceptibility to contextual manipulation through persuasive multi-turn dialogues. To this end, we introduce ChatInject, an attack that formats malicious payloads to mimic native chat templates, thereby exploiting the model's inherent instruction-following tendencies. Building on this foundation, we develop a persuasion-driven Multi-turn variant that primes the agent across conversational turns to accept and execute otherwise suspicious actions. Through comprehensive experiments across frontier LLMs, we demonstrate three critical findings: (1) ChatInject achieves significantly higher average attack success rates than traditional prompt injection methods, improving from 5.18% to 32.05% on AgentDojo and from 15.13% to 45.90% on InjecAgent, with multi-turn dialogues showing particularly strong performance at average 52.33% success rate on InjecAgent, (2) chat-template-based payloads demonstrate strong transferability across models and remain effective even against closed-source LLMs, despite their unknown template structures, and (3) existing prompt-based defenses are largely ineffective against this attack approach, especially against Multi-turn variants. These findings highlight vulnerabilities in current agent systems.