# To Protect the LLM Agent Against the Prompt Injection Attack with Polymorphic Prompt

arXiv: https://arxiv.org/abs/2506.05739 · HTML: https://mlsanity.com/paper/2506.05739/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 18 of its 34 references that are in the map.
- First posted on arXiv 6 Jun 2025.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)
- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 18 of 18)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.12773/) (2023)
- [Baseline Defenses for Adversarial Attacks Against Aligned Language Models](https://mlsanity.com/paper/2309.00614/) (2023)
- [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023)
- [JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models](https://mlsanity.com/paper/2404.01318/) (2024)
- [Formalizing and Benchmarking Prompt Injection Attacks and Defenses](https://mlsanity.com/paper/2310.12815/) (2023)
- [Jailbreak Attacks and Defenses Against Large Language Models: A Survey](https://mlsanity.com/paper/2407.04295/) (2024)
- [Better Zero-Shot Reasoning with Role-Play Prompting](https://mlsanity.com/paper/2308.07702/) (2023)
- [Security and Privacy Challenges of Large Language Models: A Survey](https://mlsanity.com/paper/2402.00888/) (2024)
- [Repair Is Nearly Generation: Multilingual Program Repair with LLMs](https://mlsanity.com/paper/2208.11640/) (2022)
- [PromptShield: Deployable Detection for Prompt Injection Attacks](https://mlsanity.com/paper/2501.15145/) (2025)
- [Defense Against Prompt Injection Attack by Leveraging Attack Techniques](https://mlsanity.com/paper/2411.00459/) (2024)
- [An Early Categorization of Prompt Injection Attacks on Large Language Models](https://mlsanity.com/paper/2402.00898/) (2024)
- [Adversarial Tuning: Defending Against Jailbreak Attacks for LLMs](https://mlsanity.com/paper/2406.06622/) (2024)
- [Self-Evaluation as a Defense Against Adversarial Attacks on LLMs](https://mlsanity.com/paper/2407.03234/) (2024)
- [GenTel-Safe: A Unified Benchmark and Shielding Framework for Defending Against Prompt Injection Attacks](https://mlsanity.com/paper/2409.19521/) (2024)
- [SPIN: Self-Supervised Prompt INjection](https://mlsanity.com/paper/2410.13236/) (2024)

## Built on by (most cited first, 15 of 15)

- [AgentArmor: Enforcing Program Analysis on Agent Runtime Trace to Defend Against Prompt Injection](https://mlsanity.com/paper/2508.01249/) (2025)
- [The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysis](https://mlsanity.com/paper/2602.10453/) (2026)
- [MAGPIE: A benchmark for Multi-AGent contextual PrIvacy Evaluation](https://mlsanity.com/paper/2510.15186/) (2025)
- [A Survey on Agentic Security: Applications, Threats and Defenses](https://mlsanity.com/paper/2510.06445/) (2025)
- [Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systems](https://mlsanity.com/paper/2510.17276/) (2025)
- [AgentGuardian: Learning Access Control Policies to Govern AI Agent Behavior](https://mlsanity.com/paper/2601.10440/) (2026)
- [Taxonomy, Evaluation and Exploitation of IPI-Centric LLM Agent Defense Frameworks](https://mlsanity.com/paper/2511.15203/) (2025)
- [Defense Against Indirect Prompt Injection via Tool Result Parsing](https://mlsanity.com/paper/2601.04795/) (2026)
- [Concept-Based Dictionary Learning for Inference-Time Safety in Vision Language Action Models](https://mlsanity.com/paper/2602.01834/) (2026)
- [Misleading the Planner through Deceptive Resumes: Registration-Time Injection in Centralized Multi-Agent Systems](https://mlsanity.com/paper/2609.15516/) (2026)
- [Decoding Latent Attack Surfaces in LLMs: Prompt Injection via HTML in Web Summarization](https://mlsanity.com/paper/2509.05831/) (2025)
- [Where Do LLM-based Systems Break? A System-Level Security Framework for Risk Assessment and Treatment](https://mlsanity.com/paper/2603.07460/) (2026)
- [Evaluation and Hardening of LLM System Instructions Against Extraction via Encoding Attacks](https://mlsanity.com/paper/2604.01039/) (2026)
- [Strengthening Polymorphic Prompt Assembling: Dynamic Separator Generation Against Emerging Prompt Injection Attacks](https://mlsanity.com/paper/2605.30534/) (2026)
- [Words Speak Louder Than Code: Investigating Cognitive Heuristics in LLM-Based Code Vulnerability Detection](https://mlsanity.com/paper/2606.30587/) (2026)

## Abstract (arXiv)

LLM agents are widely used as agents for customer support, content generation, and code assistance. However, they are vulnerable to prompt injection attacks, where adversarial inputs manipulate the model's behavior. Traditional defenses like input sanitization, guard models, and guardrails are either cumbersome or ineffective. In this paper, we propose a novel, lightweight defense mechanism called Polymorphic Prompt Assembling (PPA), which protects against prompt injection with near-zero overhead. The approach is based on the insight that prompt injection requires guessing and breaking the structure of the system prompt. By dynamically varying the structure of system prompts, PPA prevents attackers from predicting the prompt structure, thereby enhancing security without compromising performance. We conducted experiments to evaluate the effectiveness of PPA against existing attacks and compared it with other defense methods.