mlsanityData to 14 Oct 2026

Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?

arXiv 2403.06833 ↗First posted 11 Mar 2024cs.LG cs.CLcited by 81

Not on the map. Why?

Cited by 48 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Instruction-tuned Large Language Models (LLMs) show impressive results in numerous practical applications, but they lack essential safety features that are common in other areas of computer science, particularly an explicit separation of instructions and data. This makes them vulnerable to manipulations such as indirect prompt injections and generally unsuitable for safety-critical tasks. Surprisingly, there is currently no established definition or benchmark to quantify this phenomenon. In this work, we close this gap by introducing a formal measure for instruction-data separation and an empirical variant that is calculable from a model's outputs. We also present a new dataset, SEP, that allows estimating the measure for real-world models. Our results on various LLMs show that the problem of instruction-data separation is real: all models fail to achieve high separation, and canonical mitigation techniques, such as prompt engineering and fine-tuning, either fail to substantially improve separation or reduce model utility. The source code and SEP dataset are openly accessible at https://github.com/egozverev/Shold-It-Be-Executed-Or-Processed.

Key facts. Cited by 81 papers in the map (data through 14 Oct 2026). Builds on 25 of its 54 references that are in the map. First posted on arXiv 11 Mar 2024.

Builds on

25 of 54 references.

References · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  8. 2016Deep Learning with Differential Privacycited by 2,728
  9. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  10. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  11. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  12. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
13 more
  1. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  2. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 653
  3. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  4. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  5. 2019On Evaluating Adversarial Robustnesscited by 447
  6. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 375
  7. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 366
  8. 2023Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Modelscited by 264
  9. 2024Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attackscited by 156
  10. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 106
  11. 2024Neural Exec: Learning (and Learning from) Execution Triggers for Prompt Injection Attackscited by 85
  12. 2023Can LLMs Follow Simple Rules?cited by 46
  13. 2021Research Communities in cyber security: A Comprehensive Literature Reviewcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 24; 2026: 46.

81 papers.

Sort:

By area:

  1. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  2. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 240
  3. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 210
  4. 2024AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agentscited by 145
  5. 2024Attention Tracker: Detecting Prompt Injection Attacks in LLMscited by 95
  6. 2024AirGapAgent: Protecting Privacy-Conscious Conversational Agentscited by 94
  7. 2026Skill-Inject: Measuring Agent Vulnerability to Skill File Attackscited by 90
  8. 2025Design Patterns for Securing LLM Agents against Prompt Injectionscited by 73
  9. 2024Get my drift? Catching LLM Task Drift with Activation Deltascited by 72
  10. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 67
  11. 2025Effectively Controlling Reasoning Models through Thinking Interventioncited by 65
  12. 2025IHEval: Evaluating Language Models on Following the Instruction Hierarchycited by 53
38 more
  1. 2025Can Indirect Prompt Injection Attacks Be Detected and Removed?cited by 52
  2. 2025RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environmentscited by 46
  3. 2025Defending Against Prompt Injection with DataFiltercited by 35
  4. 2025Control Illusion: The Failure of Instruction Hierarchies in Large Language Modelscited by 33
  5. 2025Defending Against Prompt Injection With a Few DefensiveTokenscited by 33
  6. 2025ChatInject: Abusing Chat Templates for Prompt Injection in LLM Agentscited by 31
  7. 2025PromptLocate: Localizing Prompt Injection Attackscited by 28
  8. 2025Simple Prompt Injection Attacks Can Leak Personal Data Observed by LLM Agents During Task Executioncited by 26
  9. 2025ASIDE: Architectural Separation of Instructions and Data in Language Modelscited by 24
  10. 2024FATH: Authentication-based Test-time Defense against Indirect Prompt Injection Attackscited by 22
  11. 2025LLMail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challengecited by 21
  12. 2026Prompt Injection as Role Confusioncited by 17
  13. 2025May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attackscited by 15
  14. 2025Systems Security Foundations for Agentic Computingcited by 13
  15. 2025CommandSans: Securing AI Agents with Surgical Precision Prompt Sanitizationcited by 12
  16. 2026PIArena: A Platform for Prompt Injection Evaluationcited by 11
  17. 2025Meta SecAlign: A Secure Foundation LLM Against Prompt Injection Attackscited by 10
  18. 2025Stronger Enforcement of Instruction Hierarchy via Augmented Intermediate Representationscited by 9
  19. 2025Mitigating Indirect Prompt Injection via Instruction-Following Intent Analysiscited by 9
  20. 2024Casper: Prompt Sanitization for Protecting User Privacy in Web-Based Large Language Modelscited by 8
  21. 2026Security Considerations for Artificial Intelligence Agentscited by 8
  22. 2026Assessing Automated Prompt Injection Attacks in Agentic Environmentscited by 7
  23. 2025The Illusion of Role Separation: Hidden Shortcuts in LLM Role Learning (and How to Fix Them)cited by 6
  24. 2026AlignSentinel: Alignment-Aware Detection of Prompt Injection Attackscited by 5
  25. 2026Agent Security is a Systems Problemcited by 5
  26. 2025Can Large Language Models Really Recognize Your Name?cited by 4
  27. 2026SD-RAG: A Prompt-Injection-Resilient Framework for Selective Disclosure in Retrieval-Augmented Generationcited by 4
  28. 2024Soft Begging: Modular and Efficient Shielding of LLMs against Prompt Injection and Jailbreaking based on Prompt Tuningcited by 3
  29. 2025Exploring How LLMs Capture and Represent Domain-Specific Knowledgecited by 3
  30. 2026Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputscited by 3
  31. 2026Adaptive Evaluation of Out-of-Band Defenses Against Prompt Injection in LLM Agentscited by 3
  32. 2026SecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillationcited by 3
  33. 2025Checkpoint-GCG: Auditing and Attacking Fine-Tuning-Based Prompt Injection Defensescited by 2
  34. 2025Your Mailbox Is Mine: Prompt Injection Attacks Against Real-World LLM Email Agentscited by 2
  35. 2026RedVisor: Reasoning-Aware Prompt Injection Defense via Zero-Copy KV Cache Reusecited by 2
  36. 2026No More, No Less: Task Alignment in Terminal Agentscited by 2
  37. 2026Security--Fidelity Tradeoffs: The Hidden Cost of Prompt Injection Defensecited by 2
  38. 2026HarnessRisk: A Lifecycle-Oriented Benchmark for Agent Harness Safetycited by 2

50 most cited of 81 All 81 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Nevermind: Instruction Override and Moderation in Large Language Models
  2. 2025How Many Instructions Can LLMs Follow at Once?
  3. 2024Enhancing and Assessing Instruction-Following with Fine-Grained Instruction Variants
  4. 2023Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning
  5. 2023Rethinking the Instruction Quality: LIFT is What You Need
  6. 2025ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in Instructions
  7. 2024SelectLLM: Can LLMs Select Important Instructions to Annotate?
  8. 2024InverseCoder: Self-improving Instruction-Tuned Code LLMs with Inverse-Instruct