mlsanityData to 7 Oct 2026

The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections

arXiv 2510.09023 ↗First posted 10 Oct 2025cs.LG cs.CRcited by 108

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

How should we evaluate the robustness of language model defenses? Current defenses against jailbreaks and prompt injections (which aim to prevent an attacker from eliciting harmful knowledge or remotely triggering malicious actions, respectively) are typically evaluated either against a static set of harmful attack strings, or against computationally weak optimization methods that were not designed with the defense in mind. We argue that this evaluation process is flawed. Instead, we should evaluate defenses against adaptive attackers who explicitly modify their attack strategy to counter a defense's design while spending considerable resources to optimize their objective. By systematically tuning and scaling general optimization techniques-gradient descent, reinforcement learning, random search, and human-guided exploration-we bypass 12 recent defenses (based on a diverse set of techniques) with attack success rate above 90% for most; importantly, the majority of defenses originally reported near-zero attack success rates. We believe that future defense work must consider stronger attacks, such as the ones we describe, in order to make reliable and convincing claims of robustness.

Key facts. Cited by 108 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 73 of its 97 references that are in the map. First posted on arXiv 10 Oct 2025.

Builds on

73 of 97 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objective24 Feb
  2. 2024LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet27 Aug
  3. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models26 Jun
  4. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs21 Apr
  5. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  6. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space14 Feb
  7. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability13 Feb
  8. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks30 Jan
  9. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  11. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  12. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
1 more
  1. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 60

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  8. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,442
  9. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  10. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  11. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  12. 2022Red Teaming Language Models with Language Modelscited by 983
38 more
  1. 2017Ensemble Adversarial Training: Attacks and Defensescited by 833
  2. 2025AlphaEvolve: A coding agent for scientific and algorithmic discoverycited by 763
  3. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  4. 2020Fast is better than free: Revisiting adversarial trainingcited by 723
  5. 2017Evasion Attacks against Machine Learning at Test Timecited by 686
  6. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 659
  7. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  8. 2020On Adaptive Attacks to Adversarial Example Defensescited by 547
  9. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 542
  10. 2019Local Model Poisoning Attacks to Byzantine-Robust Federated Learningcited by 479
  11. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 449
  12. 2015Illuminating search spaces by mapping elitescited by 406
  13. 2020Label-Only Membership Inference Attackscited by 371
  14. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 328
  15. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 325
  16. 2023Are aligned neural networks adversarially aligned?cited by 305
  17. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  18. 2025Defeating Prompt Injections by Designcited by 256
  19. 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 233
  20. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  21. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Gamecited by 135
  22. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 129
  23. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attackscited by 124
  24. 2025LlamaFirewall: An open source guardrail system for building secure AI agentscited by 114
  25. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 113
  26. 2025Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agentscited by 112
  27. 2025Securing AI Agents with Information-Flow Controlcited by 108
  28. 2025PromptArmor: Simple yet Effective Prompt Injection Defensescited by 98
  29. 2024System-Level Defense against Indirect Prompt Injection Attacks: An Information Flow Control Perspectivecited by 91
  30. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
  31. 2025AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detectioncited by 82
  32. 2024Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearningcited by 81
  33. 2024Catastrophic Failure of LLM Unlearning via Quantizationcited by 76
  34. 2025MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agentscited by 67
  35. 2024Do Unlearning Methods Remove Information from Language Model Weights?cited by 66
  36. 2025DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agentscited by 59
  37. 2025IPIGuard: A Novel Tool Dependency Graph-Based Defense Against Indirect Prompt Injection in LLM Agentscited by 46
  38. 2023PromptBench: A Unified Library for Evaluation of Large Language Modelscited by 42

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 100.

108 papers.

Sort:

By area:

  1. 2026Building Production-Ready Probes For Geminicited by 38
  2. 2026AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocationscited by 33
  3. 2026Formal Policy Enforcement for Real-World Agentic Systemscited by 31
  4. 2026AgentLAB: Benchmarking LLM Agents against Long-Horizon Attackscited by 31
  5. 2026Prompt Injection Attacks on Agentic Coding Assistants: A Systematic Analysis of Vulnerabilities in Skills, Tools, and Protocol Ecosystemscited by 30
  6. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Surveycited by 26
  7. 2025BrowseSafe: Understanding and Preventing Prompt Injection Within AI Browser Agentscited by 23
  8. 2026A Survey on Long-Term Memory Security in LLM Agents: Attacks, Defenses, and Governance Across the Memory Lifecyclecited by 21
  9. 2026MalTool: Malicious Tool Attacks on LLM Agentscited by 17
  10. 2026How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competitioncited by 16
  11. 2026CaMeLs Can Use Computers Too: System-level Security for Computer Use Agentscited by 15
  12. 2025ceLLMate: Sandboxing Browser AI Agentscited by 14
38 more
  1. 2026Learning to Inject: Automated Prompt Injection via Reinforcement Learningcited by 14
  2. 2026IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMscited by 14
  3. 2026CausalArmor: Efficient Indirect Prompt Injection Guardrails via Causal Attributioncited by 13
  4. 2026PISmith: Reinforcement Learning-based Red Teaming for Prompt Injection Defensescited by 13
  5. 2026ARGUS: Defending LLM Agents Against Context-Aware Prompt Injectioncited by 13
  6. 2026Prompt Injection as Role Confusioncited by 11
  7. 2026AgentSys: Secure and Dynamic LLM Agents Through Explicit Hierarchical Memory Managementcited by 10
  8. 2026Optimizing Agent Planning for Security and Autonomycited by 10
  9. 2026PIArena: A Platform for Prompt Injection Evaluationcited by 10
  10. 2025Mitigating Indirect Prompt Injection via Instruction-Following Intent Analysiscited by 9
  11. 2025Systems Security Foundations for Agentic Computingcited by 9
  12. 2026Architecting Secure AI Agents: Perspectives on System-Level Defenses Against Indirect Prompt Injection Attackscited by 9
  13. 2026When Benchmarks Lie: Evaluating Malicious Prompt Classifiers Under True Distribution Shiftcited by 8
  14. 2025Diffusion LLMs are Natural Adversaries for any LLMcited by 7
  15. 2025Taxonomy, Evaluation and Exploitation of IPI-Centric LLM Agent Defense Frameworkscited by 7
  16. 2026T-MAP: Red-Teaming LLM Agents with Trajectory-aware Evolutionary Searchcited by 6
  17. 2026Safeguarding LLM Agents against Long-Horizon Threats via Shadow Memorycited by 6
  18. 2026Cloak and Detonate: Scanner Evasion and Dynamic Detection of Agent Skill Malwarecited by 6
  19. 2026GPT-Red: Automated Red Teaming via Self-Play at Scalecited by 5
  20. 2026Tracking Capabilities for Safer Agentscited by 4
  21. 2026Learning to Attack and Defend: Adaptive Red Teaming of Language Models via GRPOcited by 4
  22. 2026AutoDojo: Adaptive Black-Box Attacks Reveal the Limits of IPI Defenses and Task-Specification Effects in LLM Agentscited by 4
  23. 2026REDAgentBench: Executable Red Teaming and Faithful Measurement of LLM Agent Systemscited by 4
  24. 2025Soft Instruction De-escalation Defensecited by 3
  25. 2026An AI Agent Execution Environment to Safeguard User Datacited by 3
  26. 2026Web Agents Should Adopt the Plan-Then-Execute Paradigmcited by 3
  27. 2026WARD: Adversarially Robust Defense of Web Agents Against Prompt Injectionscited by 3
  28. 2026Untrusted Content Masking for Web Agents with Security Guaranteescited by 3
  29. 2026Your Agentic LLMs Secretly Encode Indirect Prompt-Injection Exposure in Hidden Statescited by 3
  30. 2026KAIJU: An Executive Kernel for Intent-Gated Execution of LLM Agentscited by 2
  31. 2026SUDP: Secret-Use Delegation Protocol for Agentic Systemscited by 2
  32. 2026Agent Security is a Systems Problemcited by 2
  33. 2026When Claws Remember but Do Not Tell: Stealthy Memory Injection in Persistent Personal Agentscited by 2
  34. 2026Agent Hacks Agent: Autoresearch for Production-Agent Red-Teamingcited by 2
  35. 2026How Agents Ask for Permission: User Permissions for AI Agents, from Interfaces to Enforcementcited by 2
  36. 2026SecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillationcited by 2
  37. 2026Intent Laundering: AI Safety Datasets Are Not What They Seemcited by 1
  38. 2026A Systematic Review of Algorithmic Red Teaming Methodologies for Assurance and Security of AI Applicationscited by 1

50 most cited of 108 All 108 (JSON)