mlsanityData to 14 Oct 2026

CoDeL: Co-Evolutionary Defense against Indirect Prompt Injection in LLM-based Agents

arXiv 2609.34463 ↗First posted 28 Sep 2026cs.CR cs.AIcited by 0

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

Large language model (LLM)-based agents increasingly rely on external tools and content, exposing them to indirect prompt injection (IPI). This threat has motivated a wide range of defenses, among which training-based defenses are often regarded as most reliable. However, existing training-based defenses are typically optimized on a static distribution of explicit injections. They learn surface-form cues rather than the boundary between serving the user and obeying an injected objective, and therefore fail when malicious intent is folded into a plausible workflow and deferred for several turns. We present CoDeL, a defense that hardens agent against an attack distribution it reshapes as it trains. The defender is updated each round via LoRA-based GDPO under a decoupled reward over safety, task progress, and format compliance, so refusing injections and completing the user's task jointly define fitness. To keep supplying it with the failures worth learning from, a co-evolving prober searches over injection rounds, attack methods, and payloads for injections that still penetrate the current defender, guided jointly by attack success and attack latency so that it preferentially mines breaches the defender notices too late. Each defender update invalidates part of the attack population and forces the next round onto a new frontier, turning the defender's own failures into a moving curriculum. Extensive experiments on three IPI benchmarks, nine baselines, and two base models show that CoDeL reduces attack success rate (ASR) by 88.5% and outperforms other baselines largely (+38.0%). Codes are available.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 48 of its 67 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

48 of 67 references.

Closest ancestors · 15

Same direction, newest first

  1. 2026CoER: Defending against Adaptive Indirect Prompt Injection via Adversarial Co-Evolution and Refinement7 Sep
  2. 2026Defending against Adaptive Prompt Injection Attacks via Reasoning-enabled Task Alignment13 Jun
  3. 2026ClawGuard: A Runtime Security Framework for Tool-Augmented LLM Agents Against Indirect Prompt Injection13 Apr
  4. 2026ReasAlign: Reasoning Enhanced Safety Alignment against Prompt Injection Attack15 Jan
  5. 2026ToolSafe: Enhancing Tool Invocation Safety of LLM-based agents via Proactive Step-level Guardrail and Feedback15 Jan
  6. 2025PromptArmor: Simple yet Effective Prompt Injection Defenses21 Jul
  7. 2025VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents3 Jun
  8. 2025AgentAlign: Navigating Safety Alignment in the Shift from Informative to Agentic Large Language Models29 May
  9. 2025RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environments28 May
  10. 2025WebInject: Prompt Injection Attack to Web Agents16 May
  11. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks22 Apr
  12. 2025ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning26 Mar
3 more
  1. 2025MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents7 Feb
  2. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents3 Oct
  3. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar

Other references · 33

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  5. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  6. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  7. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  8. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  9. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  10. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 375
  11. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 264
  12. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 210
21 more
  1. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 205
  2. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMscited by 199
  3. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 193
  4. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 129
  5. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 106
  6. 2025MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teamingcited by 32
  7. 2026MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safetycited by 21
  8. 2025DynaGuard: A Dynamic Guardian Model With User-Defined Policiescited by 19
  9. 2025Tool-R1: Sample-Efficient Reinforcement Learning for Agentic Tool Usecited by 16
  10. 2026Overcoming the Retrieval Barrier: Indirect Prompt Injection in the Wild for LLM Systemscited by 16
  11. 2025RainbowPlus: Enhancing Adversarial Prompt Generation via Evolutionary Quality-Diversity Searchcited by 11
  12. 2026PIArena: A Platform for Prompt Injection Evaluationcited by 11
  13. 2026GPT-Red: Automated Red Teaming via Self-Play at Scalecited by 8
  14. 2026Be Your Own Red Teamer: Safety Alignment via Self-Play and Reflective Experience Replaycited by 6
  15. 2026TriPlay-RL: Tri-Role Self-Play Reinforcement Learning for LLM Safety Alignmentcited by 6
  16. 2026SecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillationcited by 3
  17. 2026Metis: Learning to Jailbreak LLMs via Self-Evolving Metacognitive Policy Optimizationcited by 2
  18. 2026COPA: Continual Preference Optimization for Adaptive Prompt Injection Defensecited by 2
  19. 2026When Prompt Optimization Becomes Jailbreaking: Adaptive Red-Teaming of Large Language Modelscited by 1
  20. 2026LocalAlign: Enabling Generalizable Prompt Injection Defense via Generation of Near-Target Adversarial Examples for Alignment Trainingcited by 1
  21. 2026DiscourseFlip: An Oblique Discourse-Level Opinion Manipulation Attack against Black-box Retrieval-Augmented Generationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CAITLYN: Can LLM Agents Autonomously Synthesize Defenses against Emerging Injection Attacks?
  2. 2026AgentSafeEvo: Adversarial Co-Evolution for LLM Agent Safety
  3. 2026CoopGuard: Stateful Cooperative Agents Safeguarding LLMs Against Evolving Multi-Round Attacks
  4. 2026The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysis
  5. 2025A Multi-Agent LLM Defense Pipeline Against Prompt Injection Attacks
  6. 2026Universal Defenses for Tool-Integrated LLM Agents Against Adversarial Attacks
  7. 2025Evolve the Method, Not the Prompts: Evolutionary Synthesis of Jailbreak Attacks on LLMs
  8. 2026RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents