mlsanityData to 7 Oct 2026

To Protect the LLM Agent Against the Prompt Injection Attack with Polymorphic Prompt

arXiv 2506.05739 ↗First posted 6 Jun 2025cs.CR cs.AIcited by 14

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

LLM agents are widely used as agents for customer support, content generation, and code assistance. However, they are vulnerable to prompt injection attacks, where adversarial inputs manipulate the model's behavior. Traditional defenses like input sanitization, guard models, and guardrails are either cumbersome or ineffective. In this paper, we propose a novel, lightweight defense mechanism called Polymorphic Prompt Assembling (PPA), which protects against prompt injection with near-zero overhead. The approach is based on the insight that prompt injection requires guessing and breaking the structure of the system prompt. By dynamically varying the structure of system prompts, PPA prevents attackers from predicting the prompt structure, thereby enhancing security without compromising performance. We conducted experiments to evaluate the effectiveness of PPA against existing attacks and compared it with other defense methods.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 18 of its 34 references that are in the map. First posted on arXiv 6 Jun 2025.

Builds on

18 of 34 references.

References · 18

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  3. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  4. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  5. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  6. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  7. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 328
  8. 2023Better Zero-Shot Reasoning with Role-Play Promptingcited by 220
  9. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  10. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  11. 2022Repair Is Nearly Generation: Multilingual Program Repair with LLMscited by 68
  12. 2025PromptShield: Deployable Detection for Prompt Injection Attackscited by 43
6 more
  1. 2024Defense Against Prompt Injection Attack by Leveraging Attack Techniquescited by 31
  2. 2024An Early Categorization of Prompt Injection Attacks on Large Language Modelscited by 28
  3. 2024Adversarial Tuning: Defending Against Jailbreak Attacks for LLMscited by 23
  4. 2024Self-Evaluation as a Defense Against Adversarial Attacks on LLMscited by 13
  5. 2024GenTel-Safe: A Unified Benchmark and Shielding Framework for Defending Against Prompt Injection Attackscited by 7
  6. 2024SPIN: Self-Supervised Prompt INjectioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 8.

14 papers.

Sort:

By area:

  1. 2025AgentArmor: Enforcing Program Analysis on Agent Runtime Trace to Defend Against Prompt Injectioncited by 39
  2. 2025MAGPIE: A benchmark for Multi-AGent contextual PrIvacy Evaluationcited by 27
  3. 2026The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysiscited by 25
  4. 2025A Survey on Agentic Security: Applications, Threats and Defensescited by 17
  5. 2025Breaking and Fixing Defenses Against Control-Flow Hijacking in Multi-Agent Systemscited by 12
  6. 2026AgentGuardian: Learning Access Control Policies to Govern AI Agent Behaviorcited by 9
  7. 2026Defense Against Indirect Prompt Injection via Tool Result Parsingcited by 8
  8. 2025Taxonomy, Evaluation and Exploitation of IPI-Centric LLM Agent Defense Frameworkscited by 7
  9. 2026Concept-Based Dictionary Learning for Inference-Time Safety in Vision Language Action Modelscited by 3
  10. 2026Misleading the Planner through Deceptive Resumes: Registration-Time Injection in Centralized Multi-Agent Systemscited by 1
  11. 2025Decoding Latent Attack Surfaces in LLMs: Prompt Injection via HTML in Web Summarizationcited by 0
  12. 2026Where Do LLM-based Systems Break? A System-Level Security Framework for Risk Assessment and Treatmentcited by 0
2 more
  1. 2026Evaluation and Hardening of LLM System Instructions Against Extraction via Encoding Attackscited by 0
  2. 2026Words Speak Louder Than Code: Investigating Cognitive Heuristics in LLM-Based Code Vulnerability Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Strengthening Polymorphic Prompt Assembling: Dynamic Separator Generation Against Emerging Prompt Injection Attacks
  2. 2025PromptArmor: Simple yet Effective Prompt Injection Defenses
  3. 2026Assessing Automated Prompt Injection Attacks in Agentic Environments
  4. 2025Meta SecAlign: A Secure Foundation LLM Against Prompt Injection Attacks
  5. 2026Defending against Adaptive Prompt Injection Attacks via Reasoning-enabled Task Alignment
  6. 2026Conjunctive Prompt Attacks in Multi-Agent LLM Systems
  7. 2025A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attacks
  8. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systems