mlsanityData to 14 Oct 2026

Control Illusion: The Failure of Instruction Hierarchies in Large Language Models

arXiv 2502.15851 ↗First posted 21 Feb 2025cs.CL cs.AIcited by 33

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are increasingly deployed with hierarchical instruction schemes, where certain instructions (e.g., system-level directives) are expected to take precedence over others (e.g., user messages). Yet, we lack a systematic understanding of how effectively these hierarchical control mechanisms work. We introduce a systematic evaluation framework based on constraint prioritization to assess how well LLMs enforce instruction hierarchies. Our experiments across six state-of-the-art LLMs reveal that models struggle with consistent instruction prioritization, even for simple formatting conflicts. We find that the widely-adopted system/user prompt separation fails to establish a reliable instruction hierarchy, and models exhibit strong inherent biases toward certain constraint types regardless of their priority designation. Interestingly, we also find that societal hierarchy framings (e.g., authority, expertise, consensus) show stronger influence on model behavior than system/user roles, suggesting that pretraining-derived social structures function as latent behavioral priors with potentially greater impact than post-training guardrails.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Builds on 21 of its 24 references that are in the map. First posted on arXiv 21 Feb 2025.

Builds on

21 of 24 references.

References · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  6. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  7. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  8. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  9. 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
  10. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  11. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  12. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
9 more
  1. 2024StruQ: Defending Against Prompt Injection with Structured Queriescited by 375
  2. 2024Defending Against Indirect Prompt Injection Attacks With Spotlightingcited by 267
  3. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Gamecited by 145
  4. 2024Aligning to Thousands of Preferences via System Message Generalizationcited by 84
  5. 2024Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?cited by 81
  6. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchycited by 67
  7. 2023From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuningcited by 56
  8. 2023Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuningcited by 49
  9. 2024Libra-Leaderboard: Towards Responsible AI through a Balanced Leaderboard of Safety and Capabilitycited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 26.

33 papers.

Sort:

By area:

  1. 2025Effectively Controlling Reasoning Models through Thinking Interventioncited by 65
  2. 2026Prompt Injection as Role Confusioncited by 17
  3. 2025BuildArena: A Physics-Aligned Interactive Benchmark of LLMs for Engineering Constructioncited by 9
  4. 2026Inherited Goal Drift: Contextual Pressure Can Undermine Agentic Goalscited by 9
  5. 2025Extracting alignment data in open modelscited by 8
  6. 2026Security Considerations for Artificial Intelligence Agentscited by 8
  7. 2025ToolDreamer: Instilling LLM Reasoning Into Tool Retrieverscited by 7
  8. 2026ContextCov: Deriving and Enforcing Executable Constraints from Agent Instruction Filescited by 6
  9. 2025A2AS: Agentic AI Runtime Security and Self-Defensecited by 5
  10. 2025Reasoning Up the Instruction Ladder for Controllable Language Modelscited by 5
  11. 2026Measuring Pragmatic Influence in Large Language Model Instructionscited by 4
  12. 2025Who is In Charge? Dissecting Role Conflicts in Instruction Followingcited by 3
21 more
  1. 2026HIPO: Instruction Hierarchy via Constrained Reinforcement Learningcited by 3
  2. 2026Instruction Adherence in Coding Agent Configuration Files: A Factorial Study of Four File-Structure Variablescited by 3
  3. 2026Neural Procedural Memory: Empowering LLM Agents with Implicit Activation Steeringcited by 3
  4. 2026Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Taskscited by 3
  5. 2026Steering Instruction Hierarchies at Inference Timecited by 2
  6. 2026The Curse of Helpfulness: Inverse Scaling Law in Robustness to Distractor Instructions via DistractionIFcited by 1
  7. 2026Cognitive Firewall: A Proactive, Zero-Trust, Multi-Gate Framework for LLM Safetycited by 1
  8. 2026What Keeps Agent Skills from Being Reusable? Evidence from 138K SKILL.md Filescited by 1
  9. 2026Prompt Complexity Dilutes Structured Reasoning: A Follow-Up Study on the Car Wash Problemcited by 0
  10. 2026Ambient Persuasion in a Deployed AI Agent: Unauthorized Escalation Following Routine Non-Adversarial Content Exposurecited by 0
  11. 2026StructBreak: Structural Cognitive Overload-Induced Safety Failures in MLLMscited by 0
  12. 2026Adversarial Feeds Steer LLM Agent Decisions Against Their Defaultscited by 0
  13. 2026VATS: Exploiting Implicit Authority in Error-Path Injection via Systematic Mutationcited by 0
  14. 2026Training LLMs to Enforce Multi-Level Instruction Hierarchies via Gravity-Weighted Direct Preference Optimizationcited by 0
  15. 2026Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Modelscited by 0
  16. 2026Measuring Stability and Failure Behavior in Language Models Under Structured Perturbationscited by 0
  17. 2026Recognition Without Enforcement: Configuration-Dependent Failures in LLM Agent Instruction Arbitration and External Controlcited by 0
  18. 2026How Language Models Choose Sides: Internal Representations of Instruction Hierarchycited by 0
  19. 2026MetroLLM-Bench: Evaluating Language Models as Transit Kiosk Runtimescited by 0
  20. 2026The Persona Is Still There, but Who Is Speaking? Latent Identity Reversion in Persistent AI Agentscited by 0
  21. 2026Pincer: Resource Authorization for Agents using a Digital Twincited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Many-Tier Instruction Hierarchy in LLM Agents
  2. 2026Hierarchical Alignment: Enforcing Hierarchical Instruction-Following in LLMs through Logical Consistency
  3. 2026Do as I Say, Not as I Do: Instruction-Induction Conflict in LLMs
  4. 2026Imperative Interference: Social Register Shapes Instruction Topology in Large Language Models
  5. 2024Nevermind: Instruction Override and Moderation in Large Language Models
  6. 2026The System Prompt Illusion: How Instruction Preambles Modify Computation in Language Models
  7. 2024Benchmarking Large Language Models on Controllable Generation under Diversified Instructions
  8. 2025Instruction Tuning with and without Context: Behavioral Shifts and Downstream Impact