mlsanityData to 7 Oct 2026

StruQ: Defending Against Prompt Injection with Structured Queries

arXiv 2402.06363 ↗First posted 9 Feb 2024cs.CRcited by 325

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Recent advances in Large Language Models (LLMs) enable exciting LLM-integrated applications, which perform text-based tasks by utilizing their advanced language understanding capabilities. However, as LLMs have improved, so have the attacks against them. Prompt injection attacks are an important threat: they trick the model into deviating from the original application's instructions and instead follow user directives. These attacks rely on the LLM's ability to follow instructions and inability to separate prompts and user data. We introduce structured queries, a general approach to tackle this problem. Structured queries separate prompts and data into two channels. We implement a system that supports structured queries. This system is made of (1) a secure front-end that formats a prompt and user data into a special format, and (2) a specially trained LLM that can produce high-quality outputs from these inputs. The LLM is trained using a novel fine-tuning strategy: we convert a base (non-instruction-tuned) LLM to a structured instruction-tuned model that will only follow instructions in the prompt portion of a query. To do so, we augment standard instruction tuning datasets with examples that also include instructions in the data portion of the query, and fine-tune the model to ignore these. Our system significantly improves resistance to prompt injection attacks, with little or no impact on utility. Our code is released at https://github.com/Sizhe-Chen/StruQ.

Key facts. Cited by 325 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. First posted on arXiv 9 Feb 2024.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 38; 2025: 87; 2026: 200.

325 papers.

Sort:

By area:

  1. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agentscited by 476
  2. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  3. 2025Defeating Prompt Injections by Designcited by 256
  4. 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 233
  5. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 179
  6. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakagecited by 146
  7. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 145
  8. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 140
  9. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 129
  10. 2024Prompt Infection: LLM-to-LLM Prompt Injection within Multi-Agent Systemscited by 126
  11. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  12. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attackscited by 124
38 more
  1. 2024Optimization-based Prompt Injection Attack to LLM-as-a-Judgecited by 123
  2. 2025Progent: Securing AI Agents with Privilege Controlcited by 121
  3. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 118
  4. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  5. 2025Securing AI Agents with Information-Flow Controlcited by 108
  6. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 108
  7. 2025PromptArmor: Simple yet Effective Prompt Injection Defensescited by 98
  8. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
  9. 2024Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documentscited by 93
  10. 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 93
  11. 2024The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agentscited by 89
  12. 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMscited by 82
  13. 2025Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agentscited by 81
  14. 2024Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?cited by 70
  15. 2025MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agentscited by 67
  16. 2025Design Patterns for Securing LLM Agents against Prompt Injectionscited by 67
  17. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
  18. 2025Effectively Controlling Reasoning Models through Thinking Interventioncited by 63
  19. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 62
  20. 2024Get my drift? Catching LLM Task Drift with Activation Deltascited by 61
  21. 2025DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agentscited by 59
  22. 2025Multi-Agent Systems Execute Arbitrary Malicious Codecited by 52
  23. 2025ACE: A Security Architecture for LLM-Integrated App Systemscited by 52
  24. 2026Taming Various Privilege Escalation in LLM-Based Agent Systems: A Mandatory Access Control Frameworkcited by 50
  25. 2025AgentVigil: Generic Black-Box Red-teaming for Indirect Prompt Injection against LLM Agentscited by 42
  26. 2025RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environmentscited by 42
  27. 2024Voice Jailbreak Attacks Against GPT-4ocited by 39
  28. 2025AgentArmor: Enforcing Program Analysis on Agent Runtime Trace to Defend Against Prompt Injectioncited by 39
  29. 2024Boosting Jailbreak Attack with Momentumcited by 38
  30. 2024Adversarial Search Engine Optimization for Large Language Modelscited by 37
  31. 2024InjecGuard: Benchmarking and Mitigating Over-defense in Prompt Injection Guardrail Modelscited by 36
  32. 2024Feedback-Guided Extraction of Knowledge Base from Retrieval-Augmented LLM Applicationscited by 36
  33. 2026AttriGuard: Defeating Indirect Prompt Injection in LLM Agents via Causal Attribution of Tool Invocationscited by 33
  34. 2025RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injectioncited by 32
  35. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoningcited by 31
  36. 2025A Closer Look at System Prompt Robustnesscited by 31
  37. 2025Control Illusion: The Failure of Instruction Hierarchies in Large Language Modelscited by 31
  38. 2025A Critical Evaluation of Defenses against Prompt Injection Attackscited by 31

50 most cited of 325 All 325 (JSON)