# Control Illusion: The Failure of Instruction Hierarchies in Large Language Models

arXiv: https://arxiv.org/abs/2502.15851 · HTML: https://mlsanity.com/paper/2502.15851/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 24 references that are in the map.
- First posted on arXiv 21 Feb 2025.

## Builds on (most cited first, 21 of 21)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023)
- [How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs](https://mlsanity.com/paper/2401.06373/) (2024)
- [The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions](https://mlsanity.com/paper/2404.13208/) (2024)
- [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://mlsanity.com/paper/2309.07875/) (2023)
- [StruQ: Defending Against Prompt Injection with Structured Queries](https://mlsanity.com/paper/2402.06363/) (2024)
- [Defending Against Indirect Prompt Injection Attacks With Spotlighting](https://mlsanity.com/paper/2403.14720/) (2024)
- [Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game](https://mlsanity.com/paper/2311.01011/) (2023)
- [Aligning to Thousands of Preferences via System Message Generalization](https://mlsanity.com/paper/2405.17977/) (2024)
- [Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?](https://mlsanity.com/paper/2403.06833/) (2024)
- [Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy](https://mlsanity.com/paper/2410.09102/) (2024)
- [From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuning](https://mlsanity.com/paper/2310.00492/) (2023)
- [Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuning](https://mlsanity.com/paper/2305.11383/) (2023)
- [Libra-Leaderboard: Towards Responsible AI through a Balanced Leaderboard of Safety and Capability](https://mlsanity.com/paper/2412.18551/) (2024)

## Built on by (most cited first, 30 of 33)

- [Effectively Controlling Reasoning Models through Thinking Intervention](https://mlsanity.com/paper/2503.24370/) (2025)
- [Prompt Injection as Role Confusion](https://mlsanity.com/paper/2603.12277/) (2026)
- [BuildArena: A Physics-Aligned Interactive Benchmark of LLMs for Engineering Construction](https://mlsanity.com/paper/2510.16559/) (2025)
- [Inherited Goal Drift: Contextual Pressure Can Undermine Agentic Goals](https://mlsanity.com/paper/2603.03258/) (2026)
- [Extracting alignment data in open models](https://mlsanity.com/paper/2510.18554/) (2025)
- [Security Considerations for Artificial Intelligence Agents](https://mlsanity.com/paper/2603.12230/) (2026)
- [ToolDreamer: Instilling LLM Reasoning Into Tool Retrievers](https://mlsanity.com/paper/2510.19791/) (2025)
- [ContextCov: Deriving and Enforcing Executable Constraints from Agent Instruction Files](https://mlsanity.com/paper/2603.00822/) (2026)
- [A2AS: Agentic AI Runtime Security and Self-Defense](https://mlsanity.com/paper/2510.13825/) (2025)
- [Reasoning Up the Instruction Ladder for Controllable Language Models](https://mlsanity.com/paper/2511.04694/) (2025)
- [Measuring Pragmatic Influence in Large Language Model Instructions](https://mlsanity.com/paper/2602.21223/) (2026)
- [Who is In Charge? Dissecting Role Conflicts in Instruction Following](https://mlsanity.com/paper/2510.01228/) (2025)
- [HIPO: Instruction Hierarchy via Constrained Reinforcement Learning](https://mlsanity.com/paper/2603.16152/) (2026)
- [Instruction Adherence in Coding Agent Configuration Files: A Factorial Study of Four File-Structure Variables](https://mlsanity.com/paper/2605.10039/) (2026)
- [Neural Procedural Memory: Empowering LLM Agents with Implicit Activation Steering](https://mlsanity.com/paper/2606.29824/) (2026)
- [Every Model Cheats: Prompt-Level Mitigation of Cheating on Offensive Cyber Tasks](https://mlsanity.com/paper/2607.21763/) (2026)
- [Steering Instruction Hierarchies at Inference Time](https://mlsanity.com/paper/2607.26228/) (2026)
- [The Curse of Helpfulness: Inverse Scaling Law in Robustness to Distractor Instructions via DistractionIF](https://mlsanity.com/paper/2605.29491/) (2026)
- [Cognitive Firewall: A Proactive, Zero-Trust, Multi-Gate Framework for LLM Safety](https://mlsanity.com/paper/2607.01277/) (2026)
- [What Keeps Agent Skills from Being Reusable? Evidence from 138K SKILL.md Files](https://mlsanity.com/paper/2608.08453/) (2026)
- [Prompt Complexity Dilutes Structured Reasoning: A Follow-Up Study on the Car Wash Problem](https://mlsanity.com/paper/2603.13351/) (2026)
- [Ambient Persuasion in a Deployed AI Agent: Unauthorized Escalation Following Routine Non-Adversarial Content Exposure](https://mlsanity.com/paper/2605.00055/) (2026)
- [StructBreak: Structural Cognitive Overload-Induced Safety Failures in MLLMs](https://mlsanity.com/paper/2605.25534/) (2026)
- [Adversarial Feeds Steer LLM Agent Decisions Against Their Defaults](https://mlsanity.com/paper/2606.00914/) (2026)
- [VATS: Exploiting Implicit Authority in Error-Path Injection via Systematic Mutation](https://mlsanity.com/paper/2606.07992/) (2026)
- [Training LLMs to Enforce Multi-Level Instruction Hierarchies via Gravity-Weighted Direct Preference Optimization](https://mlsanity.com/paper/2606.10860/) (2026)
- [Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Models](https://mlsanity.com/paper/2607.02914/) (2026)
- [Measuring Stability and Failure Behavior in Language Models Under Structured Perturbations](https://mlsanity.com/paper/2608.22138/) (2026)
- [Recognition Without Enforcement: Configuration-Dependent Failures in LLM Agent Instruction Arbitration and External Control](https://mlsanity.com/paper/2608.28502/) (2026)
- [How Language Models Choose Sides: Internal Representations of Instruction Hierarchy](https://mlsanity.com/paper/2608.28648/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly deployed with hierarchical instruction schemes, where certain instructions (e.g., system-level directives) are expected to take precedence over others (e.g., user messages). Yet, we lack a systematic understanding of how effectively these hierarchical control mechanisms work. We introduce a systematic evaluation framework based on constraint prioritization to assess how well LLMs enforce instruction hierarchies. Our experiments across six state-of-the-art LLMs reveal that models struggle with consistent instruction prioritization, even for simple formatting conflicts. We find that the widely-adopted system/user prompt separation fails to establish a reliable instruction hierarchy, and models exhibit strong inherent biases toward certain constraint types regardless of their priority designation. Interestingly, we also find that societal hierarchy framings (e.g., authority, expertise, consensus) show stronger influence on model behavior than system/user roles, suggesting that pretraining-derived social structures function as latent behavioral priors with potentially greater impact than post-training guardrails.