mlsanityData to 14 Oct 2026

May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attacks

arXiv 2507.07417 ↗First posted 10 Jul 2025cs.CR cs.AI cs.CLcited by 15

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

A popular class of defenses against prompt injection attacks on large language models (LLMs) relies on fine-tuning to separate instructions and data, so that the LLM does not follow instructions that might be present with data. We evaluate the robustness of this approach in the whitebox setting by constructing strong optimization-based attacks, and show that the defenses do not provide the claimed security properties. Specifically, we construct a novel attention-based attack algorithm for textual LLMs and apply it to three recent whitebox defenses SecAlign (CCS 2025), SecAlign++, and StruQ (USENIX Security 2025), showing attacks with success rates of up to 85-95% on unseen prompts with modest increase in attacker budget in terms of tokens. Our findings make fundamental progress towards understanding the robustness of prompt injection defenses in the whitebox setting. We release our code and attacks at https://github.com/nishitvp/betteroptsattacks

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 49 of its 60 references that are in the map. First posted on arXiv 10 Jul 2025.

Builds on

49 of 60 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Lessons from Defending Gemini Against Indirect Prompt Injections20 May
  2. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks15 Apr
  3. 2025Defeating Prompt Injections by Design24 Mar
  4. 2024SecAlign: Defending Against Prompt Injection with Preference Optimization7 Oct
  5. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions19 Apr
  6. 2024CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Models19 Apr
  7. 2024Defending Against Indirect Prompt Injection Attacks With Spotlighting20 Mar
  8. 2024Automatic and Universal Prompt Injection Attacks against Large Language Models7 Mar
  9. 2024Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attacks6 Mar
  10. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Prompts26 Feb
  11. 2024StruQ: Defending Against Prompt Injection with Structured Queries9 Feb
  12. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defenses19 Oct
1 more
  1. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb

Other references · 36

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  5. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,998
  6. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  7. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,268
  8. 2022In-context Learning and Induction Headscited by 964
  9. 2023Large Language Modelscited by 780
  10. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  11. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  12. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 651
24 more
  1. 2020On Adaptive Attacks to Adversarial Example Defensescited by 555
  2. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  3. 2019On Evaluating Adversarial Robustnesscited by 447
  4. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
  5. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
  6. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 175
  7. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 172
  8. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 156
  9. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Modelscited by 134
  10. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Modelscited by 111
  11. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 110
  12. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 88
  13. 2024Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?cited by 81
  14. 2024Imprompter: Tricking LLM Agents into Improper Tool Usecited by 68
  15. 2024Query-Based Adversarial Prompt Generationcited by 62
  16. 2025Which Attention Heads Matter for In-Context Learning?cited by 61
  17. 2023Misusing Tools in Large Language Models With Visual Adversarial Examplescited by 42
  18. 2024Boosting Jailbreak Attack with Momentumcited by 40
  19. 2024Accelerating Greedy Coordinate Gradient and General Prompt Optimization via Probe Samplingcited by 37
  20. 2024AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulationcited by 33
  21. 2023Attention-Enhancing Backdoor Attacks Against BERT-based Modelscited by 32
  22. 2025Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interfacecited by 17
  23. 2024AICAttack: Adversarial Image Captioning Attack with Attention-Based Optimizationcited by 5
  24. 2024Backdoor Attack Against Vision Transformers via Attention Gradient-Based Image Erosioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 11.

15 papers.

Sort:

By area:

  1. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 129
  2. 2025RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injectioncited by 36
  3. 2026The Landscape of Prompt Injection Threats in LLM Agents: From Taxonomy to Analysiscited by 30
  4. 2025ceLLMate: Sandboxing Browser AI Agentscited by 16
  5. 2025Systems Security Foundations for Agentic Computingcited by 13
  6. 2026PIArena: A Platform for Prompt Injection Evaluationcited by 11
  7. 2026AgentSys: Secure and Dynamic LLM Agents Through Explicit Hierarchical Memory Managementcited by 10
  8. 2026Assessing Automated Prompt Injection Attacks in Agentic Environmentscited by 7
  9. 2026Agent Security is a Systems Problemcited by 5
  10. 2026Measuring Real-World Prompt Injection Attacks in LLM-based Resume Screeningcited by 5
  11. 2026AttenMIA: LLM Membership Inference Attack through Attention Signalscited by 4
  12. 2026Adaptive Evaluation of Out-of-Band Defenses Against Prompt Injection in LLM Agentscited by 3
3 more
  1. 2026MATRA: Modeling the Attack Surface of Agentic AI Systems -- OpenClaw Case Studycited by 0
  2. 2026When the Agent Becomes the Kernel: A Systematization of Security on the Path to AI-Native Operating Systemscited by 0
  3. 2026What the Model Cannot See: Covert and Provable Text Invisibility in LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Critical Evaluation of Defenses against Prompt Injection Attacks
  2. 2024Defense Against Prompt Injection Attack by Leveraging Attack Techniques
  3. 2026BASIS: Breach-Aware Selective Prompt Injection Shielding with Prefill Attention Probes
  4. 2023Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection
  5. 2025Attention is All You Need to Defend Against Indirect Prompt Injection Attacks in LLMs
  6. 2026AttackEval: A Systematic Empirical Study of Prompt Injection Attack Effectiveness Against Large Language Models
  7. 2025LLMail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challenge
  8. 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMs