mlsanityData to 7 Oct 2026

Prompt injection and agent security

959 papers11 this weekMedium confidence in this grouping: mostly one topic, with blurry edgesMedium confidence in this grouping: mostly one topic, with blurry edges · About

Keywords: prompt, agents, injection, prompt injection, agent

Key facts. Prompt injection and agent security: 959 papers on arXiv since 2021 (data through 7 Oct 2026). 347 of its 959 mapped papers (36%) cite “Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection” (2023, arXiv 2302.12173). 172 of its 959 mapped papers (18%) cite “AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents” (2024, arXiv 2406.13352). 42 new papers 24 Sep–7 Oct.

What’s new

24 Sep – 7 Oct 2026

42 new papers, 24 Sep – 7 Oct; 11 this week.

  1. This week · 11
  2. BRANCH: Bypassing Multi-Scanner AI Guardrails also fits: Backdoor and data poisoning attacks
  3. Constrained-Action AI Remediation for SIEM/XDR via a NeMo-Guardrails Proxy
  4. Defensive Sufficiency in a Stackelberg Model of AI Security
  5. Package Hallucination Attacks on Coding Agents through Prompt Injection in Rule Files
  6. Adversarial Images Hijack Web Agents from Visual Grounding to Browser Execution
  7. AdvSim2Real : Training Web Agents Against Adaptive Prompt Injection in a Web World Model
  8. POLAR: Ontology-Guided Risk Prevention for Tool-Calling LLM Agents
  9. Towards a Unified Misuse Monitoring Benchmark
  10. RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents
  11. Benchmarking Jailbreak Guardrails for Embodied Agents
All 42
  1. Topology-Conditioned Backdoors: Language Models That Insert Vulnerabilities When They Infer They Are in a Multi-Agent System
  2. Earlier · 31
  3. Reflections and Fragments: Securing LLMs Against Sequential Mosaic Attacks
  4. Readable Before Actionable: Causal Tracing of Indirect Prompt Injection
  5. Who Is Your Agent Serving? Provider-Side Indirect Prompt Injection in Proactive Agents
  6. Blocking at the Boundary: Auditing Long-Horizon Agents against Staged Prompt Injection
  7. Hidden Risks of Jev: An Empirical Study of Security, Privacy, and Dual Use
  8. APEX: Active Protection at Execution Boundaries for LLM Agents
  9. Self-Reflection Fine-Tuning: Enhancing Agent Security against Prompt Injection Attacks from Failure Experience
  10. Beware EviLLM: Enabling Vulnerability Injection via Large Language Models also fits: Jailbreaking
  11. Securing Computer-Use Agents Against Branch Steering Attacks
  12. Prompted to Discriminate: Generalizing Malicious-Input Probes in the Wild
  13. The Innocent Courier: Covert Exfiltration Through Legitimate LLM Web Fetching
  14. Preemptive LLM Unlearning against Forbidden Capability Acquisition via Gradient Sealing also fits: Machine unlearning
  15. Refusals That Bend: Measuring and Predicting Task Malleability in Embodied VLM Planners also fits: Safety alignment, refusal and guardrails
  16. ToolFence: Fine-Grained Authorization for Secure Tool-Using LLM Agents
  17. Constitutional adapters: Inference-time interventions for misalignment and misuse also fits: Safety alignment, refusal and guardrails
  18. Divide and Inject: Can Agents Reconstruct an Indirect Prompt Injection from Fragments?
  19. CounterSteer: Suppressing Indirect Prompt Injection with Activation Steering
  20. FinRT: Distilling Adaptive Red-Teaming Strategies into Reusable Adversarial Generators in Consumer Finance
  21. Same Bytes, Different Authority: Reserved-Token Representations in Chat-Template Prompt Injection
  22. RISE: Red-teaming via Iterative Strategy Evolution for Modern Text-to-Image Models also fits: Vision-language and text-to-image safety
  23. CoDeL: Co-Evolutionary Defense against Indirect Prompt Injection in LLM-based Agents
  24. Climbing the Hill: Prompt Injection Red-Teaming Against Frontier Models with Curriculum Reinforcement Learning
  25. PlanGuard: A Guardrail for Multi-Step Plan Safety in Embodied Agents also fits: Safety alignment, refusal and guardrails
  26. Silent Failures in Agentic Security Evaluation: A Validated Harness for Tool-Call Mediation Under Indirect Prompt Injection
  27. AnchorRep: Defending LLMs Against Cross-Model Adversarial Transfer via Representation Repulsion also fits: Backdoor and data poisoning attacks
  28. SkillDRE: Dual-Stage Red-Team Evolution of Agent Skills via Pre-Execution and Runtime Feedback
  29. SilentCall: Hidden Tool-Call Backdoors in Open-Weight Agents, and How to Catch Them
  30. AgentXploit: Autonomous Repository-to-Runtime Red-Teaming for AI Agents
  31. Prompt Injection Detection for Email Agents Through Attack Chain Modeling also fits: Backdoor and data poisoning attacks
  32. Instrumental Monitor Evasion Emerges Under Ordinary Task Pressure
  33. ENDOPROMPT: Victim-Side Pseudo-References for Utility Degradation also fits: Backdoor and data poisoning attacks

Foundations

Most cited by papers here

  1. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection2023cited by 347 here
  2. AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents2024cited by 172 here
  3. StruQ: Defending Against Prompt Injection with Structured Queries2024cited by 170 here
  4. InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents2024cited by 168 here
  5. Prompt Injection attack against LLM-integrated Applications2023cited by 148 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2022Red Teaming Language Models with Language Modelscited by 129 here
  2. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 75 here
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 165 here
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 317 here
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 177 here
  6. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 116 here
  7. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 170 here
  8. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 168 here
  9. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 172 here
  10. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 104 here
  11. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attackscited by 78 here
13 more it builds on, not in this chain
  1. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 347 here
  2. 2023Prompt Injection attack against LLM-integrated Applicationscited by 148 here
  3. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 76 here
  4. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 102 here
  5. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 143 here
  6. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 106 here
  7. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 133 here
  8. 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 118 here
  9. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 131 here
  10. 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 75 here
  11. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 97 here
  12. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agentscited by 90 here
  13. 2025Defeating Prompt Injections by Designcited by 93 here

The 24 most cited of 1,391.

Browse

30 of 959, ranked 6–35

  1. 6Formalizing and Benchmarking Prompt Injection Attacks and Defenses2023143
  2. 7HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal2024133
  3. 8The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions2024131
  4. 9Red Teaming Language Models with Language Models2022129
  5. 10Defending Against Indirect Prompt Injection Attacks With Spotlighting2024118
  6. 11Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models2023106
  7. 12SecAlign: Defending Against Prompt Injection with Preference Optimization2024104
  8. 13Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents202497
  9. 14Defeating Prompt Injections by Design202593
  10. 15AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents202490
All 30
  1. 16DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks202578
  2. 17AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases202475
  3. 18Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned202275
  4. 19WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks202557
  5. 20Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents202554
  6. 21Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game202354
  7. 22Jatmo: Prompt Injection Defense by Task-Specific Finetuning202354
  8. 23Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts202452
  9. 24GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning202452
  10. 25PromptArmor: Simple yet Effective Prompt Injection Defenses202551
  11. 26Automatic and Universal Prompt Injection Attacks against Large Language Models202451
  12. 27Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attacks202448
  13. 28MART: Improving LLM Safety with Multi-round Automatic Red-Teaming202345
  14. 29MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents202544
  15. 30Optimization-based Prompt Injection Attack to LLM-as-a-Judge202444
  16. 31System-Level Defense against Indirect Prompt Injection Attacks: An Information Flow Control Perspective202442
  17. 32Attention Tracker: Detecting Prompt Injection Attacks in LLMs202441
  18. 33Prompt Injection Attack to Tool Selection in LLM Agents202540
  19. 34Watch Out for Your Agents! Investigating Backdoor Threats to LLM-Based Agents202439
  20. 35The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents202435