mlsanityData to 14 Oct 2026

Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy

arXiv 2410.09102 ↗First posted 9 Oct 2024cs.LG cs.AI cs.CLcited by 67

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Large Language Models (LLMs) are susceptible to security and safety threats, such as prompt injection, prompt extraction, and harmful requests. One major cause of these vulnerabilities is the lack of an instruction hierarchy. Modern LLM architectures treat all inputs equally, failing to distinguish between and prioritize various types of instructions, such as system messages, user prompts, and data. As a result, lower-priority user prompts may override more critical system instructions, including safety protocols. Existing approaches to achieving instruction hierarchy, such as delimiters and instruction-based training, do not address this issue at the architectural level. We introduce the Instructional Segment Embedding (ISE) technique, inspired by BERT, to modern large language models, which embeds instruction priority information directly into the model. This approach enables models to explicitly differentiate and prioritize various instruction types, significantly improving safety against malicious prompts that attempt to override priority rules. Our experiments on the Structured Query and Instruction Hierarchy benchmarks demonstrate an average robust accuracy increase of up to 15.75% and 18.68%, respectively. Furthermore, we observe an improvement in instruction-following capability of up to 4.1% evaluated on AlpacaEval. Overall, our approach offers a promising direction for enhancing the safety and effectiveness of LLM architectures.

Key facts. Cited by 67 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 40 of its 54 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

40 of 54 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions19 Apr
  2. 2024Defending Against Indirect Prompt Injection Attacks With Spotlighting20 Mar
  3. 2024InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents5 Mar
  4. 2024StruQ: Defending Against Prompt Injection with Structured Queries9 Feb
  5. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuning29 Dec
  6. 2023Assessing Prompt Injection Risks in 200+ Custom GPTs20 Nov
  7. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game2 Nov
  8. 2023Prompt Injection attack against LLM-integrated Applications8 Jun
  9. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb
  10. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug
  11. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  8. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  9. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  10. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  11. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
  12. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
17 more
  1. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
  2. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  3. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
  4. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 397
  5. 2024Improving Alignment and Robustness with Circuit Breakerscited by 322
  6. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
  7. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 211
  8. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 173
  9. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 172
  10. 2023Conversational Health Agents: A Personalized LLM-Powered Agent Frameworkcited by 92
  11. 2023Effective Prompt Extraction from Language Modelscited by 85
  12. 2024Long-Context Language Modeling with Parallel Context Encodingcited by 84
  13. 2024Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?cited by 81
  14. 2024Transformers Can Do Arithmetic with the Right Embeddingscited by 78
  15. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defensescited by 70
  16. 2023Large Language Models in Education: Vision and Opportunitiescited by 54
  17. 2023Can LLMs Follow Simple Rules?cited by 46

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 24; 2026: 42.

67 papers.

Sort:

By area:

  1. 2025Defeating Prompt Injections by Designcited by 297
  2. 2025WASP: Benchmarking Web Agent Security Against Prompt Injection Attackscited by 157
  3. 2025Securing AI Agents with Information-Flow Controlcited by 121
  4. 2024The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agentscited by 99
  5. 2025Effectively Controlling Reasoning Models through Thinking Interventioncited by 65
  6. 2025A Critical Evaluation of Defenses against Prompt Injection Attackscited by 36
  7. 2025RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injectioncited by 36
  8. 2025Defending Against Prompt Injection with DataFiltercited by 35
  9. 2025Control Illusion: The Failure of Instruction Hierarchies in Large Language Modelscited by 33
  10. 2025Defending Against Prompt Injection With a Few DefensiveTokenscited by 33
  11. 2025A Closer Look at System Prompt Robustnesscited by 32
  12. 2025PromptLocate: Localizing Prompt Injection Attackscited by 28
38 more
  1. 2026The Attack and Defense Landscape of Agentic AI: A Comprehensive Surveycited by 28
  2. 2025Simple Prompt Injection Attacks Can Leak Personal Data Observed by LLM Agents During Task Executioncited by 26
  3. 2025ASIDE: Architectural Separation of Instructions and Data in Language Modelscited by 24
  4. 2026MUZZLE: Adaptive Agentic Red-Teaming of Web Agents Against Indirect Prompt Injection Attackscited by 21
  5. 2025PIShield: Detecting Prompt Injection Attacks via Intrinsic LLM Featurescited by 20
  6. 2025ObliInjection: Order-Oblivious Prompt Injection Attack to LLM Agents with Multi-source Datacited by 18
  7. 2026VIGIL: Defending LLM Agents Against Tool Stream Injection via Verify-Before-Commitcited by 17
  8. 2026Prompt Injection as Role Confusioncited by 17
  9. 2026A Framework for Formalizing LLM Agent Securitycited by 16
  10. 2026IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMscited by 15
  11. 2025Systems Security Foundations for Agentic Computingcited by 13
  12. 2026Defenses Against Prompt Attacks Learn Surface Heuristicscited by 11
  13. 2026The Granularity Mismatch in Agent Security: Argument-Level Provenance Solves Enforcement and Isolates the LLM Reasoning Bottleneckcited by 11
  14. 2025Meta SecAlign: A Secure Foundation LLM Against Prompt Injection Attackscited by 10
  15. 2026Optimizing Agent Planning for Security and Autonomycited by 10
  16. 2025Stronger Enforcement of Instruction Hierarchy via Augmented Intermediate Representationscited by 9
  17. 2026The Promptware Kill Chain: How Prompt Injections Gradually Evolved Into a Multistep Malware Delivery Mechanismcited by 9
  18. 2026Inherited Goal Drift: Contextual Pressure Can Undermine Agentic Goalscited by 9
  19. 2026The Autonomy Tax: Defense Training Breaks LLM Agentscited by 9
  20. 2026Security Considerations for Artificial Intelligence Agentscited by 8
  21. 2025AttnTrace: Contextual Attribution of Prompt Injection and Knowledge Corruptioncited by 7
  22. 2025The Illusion of Role Separation: Hidden Shortcuts in LLM Role Learning (and How to Fix Them)cited by 6
  23. 2026Many-Tier Instruction Hierarchy in LLM Agentscited by 6
  24. 2025Reasoning Up the Instruction Ladder for Controllable Language Modelscited by 5
  25. 2026Toward Securing AI Agents Like Operating Systemscited by 5
  26. 2026Agent Security is a Systems Problemcited by 5
  27. 2026Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputscited by 3
  28. 2026Untrusted Content Masking for Web Agents with Security Guaranteescited by 3
  29. 2026SecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillationcited by 3
  30. 2025Checkpoint-GCG: Auditing and Attacking Fine-Tuning-Based Prompt Injection Defensescited by 2
  31. 2025EAGER: Edge-Aligned LLM Defense for Robust, Efficient, and Accurate Cybersecurity Question Answeringcited by 2
  32. 2026Steering Instruction Hierarchies at Inference Timecited by 2
  33. 2026CoER: Defending against Adaptive Indirect Prompt Injection via Adversarial Co-Evolution and Refinementcited by 2
  34. 2026The Paradox of Robustness: Decoupling Rule-Based Logic from Affective Noise in High-Stakes Decision-Makingcited by 1
  35. 2026Extended to Reality: Prompt Injection in 3D Environmentscited by 1
  36. 2026Are Dilemmas and Conflicts in LLM Alignment Solvable? A View from Priority Graphcited by 1
  37. 2026Hierarchical Alignment: Enforcing Hierarchical Instruction-Following in LLMs through Logical Consistencycited by 1
  38. 2026LongPIBench: A Long-Context Benchmark for Prompt Injectioncited by 1

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Evaluating the Instruction-Following Robustness of Large Language Models to Prompt Injection
  2. 2025Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instruction
  3. 2025Sysformer: Safeguarding Frozen Large Language Models with Adaptive System Prompts
  4. 2025Prompt Injection as an Emerging Threat: Evaluating the Resilience of Large Language Models
  5. 2026Know Thy Enemy: Securing LLMs Against Prompt Injection via Diverse Data Synthesis and Instruction-Level Chain-of-Thought Learning
  6. 2026Reflex-Guard: A Low-Latency Guardrail for LLM Prompt Safety Using Dense Semantic Embeddings
  7. 2025Defending against Indirect Prompt Injection by Instruction Detection
  8. 2024On Prompt-Driven Safeguarding for Large Language Models