# Many-Tier Instruction Hierarchy in LLM Agents

arXiv: https://arxiv.org/abs/2604.09443 · HTML: https://mlsanity.com/paper/2604.09443/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 10 Apr 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 25 of 25)

- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Instruction-Following Evaluation for Large Language Models](https://mlsanity.com/paper/2311.07911/) (2023)
- [Prompt Injection attack against LLM-integrated Applications](https://mlsanity.com/paper/2306.05499/) (2023)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents](https://mlsanity.com/paper/2403.02691/) (2024)
- [The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions](https://mlsanity.com/paper/2404.13208/) (2024)
- [Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models](https://mlsanity.com/paper/2312.14197/) (2023)
- [Generalizing Verifiable Instruction Following](https://mlsanity.com/paper/2507.02833/) (2025)
- [Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game](https://mlsanity.com/paper/2311.01011/) (2023)
- [Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks](https://mlsanity.com/paper/2602.20156/) (2026)
- [LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Texts](https://mlsanity.com/paper/2501.00274/) (2024)
- [Effective Prompt Extraction from Language Models](https://mlsanity.com/paper/2307.06865/) (2023)
- [Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy](https://mlsanity.com/paper/2410.09102/) (2024)
- [IHEval: Evaluating Language Models on Following the Instruction Hierarchy](https://mlsanity.com/paper/2502.08745/) (2025)
- [Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements](https://mlsanity.com/paper/2410.08968/) (2024)
- [CodeIF: Benchmarking the Instruction-Following Capabilities of Large Language Models for Code Generation](https://mlsanity.com/paper/2502.19166/) (2025)
- [IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMs](https://mlsanity.com/paper/2603.10521/) (2026)
- [ConInstruct: Evaluating Large Language Models on Conflict Detection and Resolution in Instructions](https://mlsanity.com/paper/2511.14342/) (2025)
- [CCTU: A Benchmark for Tool Use under Complex Constraints](https://mlsanity.com/paper/2603.15309/) (2026)
- [Jailbreak Distillation: Renewable Safety Benchmarking](https://mlsanity.com/paper/2505.22037/) (2025)
- [When Instructions Multiply: Measuring and Estimating LLM Capabilities of Multiple Instructions Following](https://mlsanity.com/paper/2509.21051/) (2025)
- [Reasoning Up the Instruction Ladder for Controllable Language Models](https://mlsanity.com/paper/2511.04694/) (2025)
- [DeonticBench: A Benchmark for Reasoning over Rules](https://mlsanity.com/paper/2604.04443/) (2026)

## Built on by (most cited first, 6 of 6)

- [Prime Agent: A Self-Improving RLM Harness](https://mlsanity.com/paper/2608.23552/) (2026)
- [TSQAgent: Rating Time Series Data Quality via Dedicated Agentic Reasoning](https://mlsanity.com/paper/2606.03629/) (2026)
- [Evaluating using Mock Tool Calls to Quarantine Untrusted Prompt Inputs](https://mlsanity.com/paper/2605.30521/) (2026)
- [Where Instruction Hierarchy Breaks: Diagnosing and Repairing Failures in Reasoning Language Models](https://mlsanity.com/paper/2606.07808/) (2026)
- [Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Models](https://mlsanity.com/paper/2607.02914/) (2026)
- [Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Models](https://mlsanity.com/paper/2609.22234/) (2026)

## Abstract (arXiv)

Large language model agents receive instructions from many sources-system messages, user prompts, tool outputs, other agents, and more-each carrying different levels of trust and authority. When these instructions conflict, agents must reliably follow the highest-privilege instruction to remain safe and effective. The dominant paradigm, instruction hierarchy (IH), assumes a fixed, small set of privilege levels (typically fewer than five) defined by rigid role labels (e.g., system > user). This is inadequate for real-world agentic settings, where conflicts can arise across far more sources and contexts. In this work, we propose Many-Tier Instruction Hierarchy (ManyIH), a paradigm for resolving instruction conflicts among instructions with arbitrarily many privilege levels. We introduce ManyIH-Bench, the first benchmark for ManyIH. ManyIH-Bench requires models to navigate up to 12 levels of conflicting instructions with varying privileges, comprising 853 agentic tasks (427 coding and 426 instruction-following). ManyIH-Bench composes constraints developed by LLMs and verified by humans to create realistic and difficult test cases spanning 46 real-world agents. Our experiments show that even the current frontier models perform poorly (~40% accuracy) when instruction conflict scales. This work underscores the urgent need for methods that explicitly target fine-grained, scalable instruction conflict resolution in agentic settings.