mlsanityData to 14 Oct 2026

IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMs

arXiv 2603.10521 ↗First posted 11 Mar 2026cs.AI cs.CL cs.CRcited by 15

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Instruction hierarchy (IH) defines how LLMs prioritize system, developer, user, and tool instructions under conflict, providing a concrete, trust-ordered policy for resolving instruction conflicts. IH is key to defending against jailbreaks, system prompt extractions, and agentic prompt injections. However, robust IH behavior is difficult to train: IH failures can be confounded with instruction-following failures, conflicts can be nuanced, and models can learn shortcuts such as overrefusing. We introduce IH-Challenge, a reinforcement learning training dataset, to address these difficulties. Fine-tuning GPT-5-Mini on IH-Challenge with online adversarial example generation improves IH robustness by +10.0% on average across 16 in-distribution, out-of-distribution, and human red-teaming benchmarks (84.1% to 94.1%), reduces unsafe behavior from 6.6% to 0.7% while improving helpfulness on general safety evaluations, and saturates an internal static agentic prompt injection evaluation, with minimal capability regression. We release the IH-Challenge dataset (https://huggingface.co/datasets/openai/ih-challenge) to support future research on robust instruction hierarchy.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 24 of its 30 references that are in the map. First posted on arXiv 11 Mar 2026.

Builds on

24 of 30 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Reasoning Up the Instruction Ladder for Controllable Language Models30 Oct
  2. 2025RL Is a Hammer and LLMs Are Nails: A Simple Reinforcement Learning Recipe for Strong Prompt Injection6 Oct
  3. 2025A Closer Look at System Prompt Robustness15 Feb
  4. 2024Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy9 Oct
  5. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions19 Apr
  6. 2024CyberSecEval 2: A Wide-Ranging Cybersecurity Evaluation Suite for Large Language Models19 Apr
  7. 2024Defending Against Indirect Prompt Injection Attacks With Spotlighting20 Mar
  8. 2023Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game2 Nov
  9. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection23 Feb

Other references · 15

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  6. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  7. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  8. 2016Concrete Problems in AI Safetycited by 1,642
  9. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  10. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  11. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  12. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
3 more
  1. 2025Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teamingcited by 181
  2. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 129
  3. 2025From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Trainingcited by 52

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 15.

15 papers.

Sort:

By area:

  1. 2026The Granularity Mismatch in Agent Security: Argument-Level Provenance Solves Enforcement and Isolates the LLM Reasoning Bottleneckcited by 11
  2. 2026GPT-Red: Automated Red Teaming via Self-Play at Scalecited by 8
  3. 2026Many-Tier Instruction Hierarchy in LLM Agentscited by 6
  4. 2026Reinforcement Learning Towards Broadly and Persistently Beneficial Modelscited by 4
  5. 2026SecOPD: Mitigating Adaptive Prompt Injections by On-Policy Distillationcited by 3
  6. 2026Steering Instruction Hierarchies at Inference Timecited by 2
  7. 2026CLaaS: Continual learning as a service for sample efficient online learningcited by 1
  8. 2026What Keeps Agent Skills from Being Reusable? Evidence from 138K SKILL.md Filescited by 1
  9. 2026System-Prompt Anchoring with Cross-Attention Layerscited by 0
  10. 2026Evaluating using Mock Tool Calls to Quarantine Untrusted Prompt Inputscited by 0
  11. 2026Where Instruction Hierarchy Breaks: Diagnosing and Repairing Failures in Reasoning Language Modelscited by 0
  12. 2026IHDec: Divergence-Steered Contrastive Decoding for Securing Multi-Turn Instruction Hierarchiescited by 0
3 more
  1. 2026Language Shapes Instruction Hierarchy Compliance in Multilingual LLMscited by 0
  2. 2026Seeing Through Conflicts: Improving Instruction Hierarchy Alignment in Vision-Language Modelscited by 0
  3. 2026What Pretraining and Midtraining Make Learnable from Rewards?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Training LLMs to Enforce Multi-Level Instruction Hierarchies via Gravity-Weighted Direct Preference Optimization
  2. 2026Hierarchical Alignment: Enforcing Hierarchical Instruction-Following in LLMs through Logical Consistency
  3. 2026Evaluating Prompt Injection Defenses for Educational LLM Tutors: Security-Usability-Latency Trade-offs
  4. 2026RAISED: Self-Distillation for Robustness to Prompt Injection in LLM Agents
  5. 2026The Autonomy Tax: Defense Training Breaks LLM Agents
  6. 2025Stronger Enforcement of Instruction Hierarchy via Augmented Intermediate Representations
  7. 2025Jailbreak-R1: Exploring the Jailbreak Capabilities of LLMs via Reinforcement Learning
  8. 2026Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks