mlsanityData to 14 Oct 2026

The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions

arXiv 2404.13208 ↗First posted 19 Apr 2024cs.CR cs.CL cs.LGcited by 493

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Today's LLMs are susceptible to prompt injections, jailbreaks, and other attacks that allow adversaries to overwrite a model's original instructions with their own malicious prompts. In this work, we argue that one of the primary vulnerabilities underlying these attacks is that LLMs often consider system prompts (e.g., text from an application developer) to be the same priority as text from untrusted users and third parties. To address this, we propose an instruction hierarchy that explicitly defines how models should behave when instructions of different priorities conflict. We then propose a data generation method to demonstrate this hierarchical instruction following behavior, which teaches LLMs to selectively ignore lower-privileged instructions. We apply this method to GPT-3.5, showing that it drastically increases robustness -- even for attack types not seen during training -- while imposing minimal degradations on standard capabilities.

Key facts. Cited by 493 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 21 of its 31 references that are in the map. First posted on arXiv 19 Apr 2024.

Builds on

21 of 31 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024StruQ: Defending Against Prompt Injection with Structured Queries9 Feb
  2. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models21 Dec
  3. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game2 Nov
  4. 2023Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competition24 Oct
  5. 2023Prompt Injection attack against LLM-integrated Applications8 Jun
  6. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb
  7. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug
  8. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 13

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  6. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,190
  7. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  8. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  9. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  10. 2022TALM: Tool Augmented Language Modelscited by 197
  11. 2022Learning by Distilling Contextcited by 138
  12. 2024Coercing LLMs to do and reveal (almost) anythingcited by 89
1 more
  1. 2024Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?cited by 81

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 66; 2025: 128; 2026: 299.

493 papers.

Sort:

By area:

  1. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  2. 2024OpenAI o1 System Cardcited by 1,244
  3. 2025Defeating Prompt Injections by Designcited by 297
  4. 2024Frontier Models are Capable of In-context Schemingcited by 287
  5. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 210
  6. 2025Generalizing Verifiable Instruction Followingcited by 194
  7. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 157
  8. 2024EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakagecited by 150
  9. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 145
  10. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 144
  11. 2025DataSentinel: A Game-Theoretic Detection of Prompt Injection Attackscited by 143
  12. 2025Progent: Securing AI Agents with Privilege Controlcited by 141
38 more
  1. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 127
  2. 2025LlamaFirewall: An open source guardrail system for building secure AI agentscited by 127
  3. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 121
  4. 2025Securing AI Agents with Information-Flow Controlcited by 121
  5. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 120
  6. 2024On Memorization of Large Language Models in Logical Reasoningcited by 117
  7. 2024Attacking Vision-Language Computer Agents via Pop-upscited by 116
  8. 2025PromptArmor: Simple yet Effective Prompt Injection Defensescited by 108
  9. 2024Machine Against the RAG: Jamming Retrieval-Augmented Generation with Blocker Documentscited by 102
  10. 2024The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agentscited by 99
  11. 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMscited by 95
  12. 2024AirGapAgent: Protecting Privacy-Conscious Conversational Agentscited by 94
  13. 2024The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Modelscited by 93
  14. 2024Improving Steering Vectors by Targeting Sparse Autoencoder Featurescited by 92
  15. 2026Skill-Inject: Measuring Agent Vulnerability to Skill File Attackscited by 90
  16. 2024LLMs know their vulnerabilities: Uncover Safety Gaps through Natural Distribution Shiftscited by 86
  17. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  18. 2025MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agentscited by 77
  19. 2024Get my drift? Catching LLM Task Drift with Activation Deltascited by 72
  20. 2024Refuse Whenever You Feel Unsafe: Improving Safety in LLMs via Decoupled Refusal Trainingcited by 69
  21. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 67
  22. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 67
  23. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 65
  24. 2025Effectively Controlling Reasoning Models through Thinking Interventioncited by 65
  25. 2025Contextual Agent Security: A Policy for Every Purposecited by 59
  26. 2024LLMmap: Fingerprinting For Large Language Modelscited by 58
  27. 2025Reinforcement Learning from Human Feedbackcited by 58
  28. 2024Caution for the Environment: Multimodal LLM Agents are Susceptible to Environmental Distractionscited by 56
  29. 2025ACE: A Security Architecture for LLM-Integrated App Systemscited by 55
  30. 2026Taming Various Privilege Escalation in LLM-Based Agent Systems: A Mandatory Access Control Frameworkcited by 54
  31. 2025IHEval: Evaluating Language Models on Following the Instruction Hierarchycited by 53
  32. 2025Multi-Agent Systems Execute Arbitrary Malicious Codecited by 53
  33. 2025Can Indirect Prompt Injection Attacks Be Detected and Removed?cited by 52
  34. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 51
  35. 2024LatentQA: Teaching LLMs to Decode Activations Into Natural Languagecited by 51
  36. 2025PromptShield: Deployable Detection for Prompt Injection Attackscited by 50
  37. 2024garak: A Framework for Security Probing Large Language Modelscited by 49
  38. 2025AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarioscited by 49

50 most cited of 493 All 493 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection
  2. 2026Evaluation and Hardening of LLM System Instructions Against Extraction via Encoding Attacks
  3. 2026Prompt Control-Flow Integrity: A Priority-Aware Runtime Defense Against Prompt Injection in LLM Systems
  4. 2026Know Thy Enemy: Securing LLMs Against Prompt Injection via Diverse Data Synthesis and Instruction-Level Chain-of-Thought Learning
  5. 2026Beyond Over-Refusal: Defending Indirect Prompt Injection via Latent Instruction Manifolds
  6. 2024MaPPing Your Model: Assessing the Impact of Adversarial Attacks on LLM-based Programming Assistants
  7. 2024Instruction Backdoor Attacks Against Customized LLMs
  8. 2025PISanitizer: Preventing Prompt Injection to Long-Context LLMs via Prompt Sanitization