mlsanityData to 7 Oct 2026

HIPO: Instruction Hierarchy via Constrained Reinforcement Learning

arXiv 2603.16152 ↗First posted 17 Mar 2026cs.LG cs.AI cs.CLcited by 2

Not on the map. Why?

Abstract (arXiv)

Hierarchical Instruction Following (HIF) refers to the problem of prompting large language models with a priority-ordered stack of instructions. Standard methods like RLHF and DPO typically fail in this problem since they mainly optimize for a single objective, failing to explicitly enforce system prompt compliance. Meanwhile, supervised fine-tuning relies on mimicking filtered, compliant data, which fails to establish the priority asymmetry at the algorithmic level. In this paper, we introduce textscHIPO, a novel alignment framework that formulates HIF as a Constrained Markov Decision Process. textscHIPO elevates system prompts from mere input context to strict algorithmic boundaries. Using a primal-dual safe reinforcement learning approach, the algorithm dynamically enforces system prompt compliance as an explicit constraint, maximizing user utility strictly within this feasible region. Extensive evaluations across diverse model architectures (e.g., Qwen, Phi, Llama) demonstrate that textscHIPO significantly improves both system compliance and user utility. Furthermore, mechanistic analysis reveals that this constrained optimization autonomously drives the model to shift its attention toward long-range system tokens, providing a principled foundation for reliable LLM deployment in complex workflows.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 34 of its 49 references that are in the map. First posted on arXiv 17 Mar 2026.

Builds on

34 of 49 references.

References · 34

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  10. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  11. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  12. 2023Large Language Models are not Fair Evaluatorscited by 1,050
22 more
  1. 2017Constrained Policy Optimizationcited by 928
  2. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  3. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  4. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  5. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  6. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
  7. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  8. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 140
  9. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
  10. 2024Self-Evolving Multi-Agent Collaboration Networks for Software Developmentcited by 73
  11. 2024Measuring and Controlling Instruction (In)Stability in Language Model Dialogscited by 51
  12. 2025A Closer Look at System Prompt Robustnesscited by 31
  13. 2025Control Illusion: The Failure of Instruction Hierarchies in Large Language Modelscited by 31
  14. 2024CoMAL: Collaborative Multi-Agent Large Language Models for Mixed-Autonomy Trafficcited by 22
  15. 2024Multi-Objective Alignment of Large Language Models Through Hypervolume Maximizationcited by 20
  16. 2025TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedbackcited by 11
  17. 2025Alignment of large language models with constrained learningcited by 8
  18. 2025Bounded Rationality for LLMs: Satisficing Alignment at Inference-Timecited by 8
  19. 2026From Rubrics to Reliable Scores: Evidence-Grounded Text Evaluation with LLM Judgescited by 6
  20. 2024Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimizationcited by 5
  21. 2025Reasoning Up the Instruction Ladder for Controllable Language Modelscited by 5
  22. 2024Towards Fast Safe Online Reinforcement Learning via Policy Finetuningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Solver-Aware Decompositions for Programming-by-Example: When Dividing Requires Knowing how to Conquercited by 0
  2. 2026MeshHeal: Two-Timescale Self-Healing for Gray Failures in Decentralized LLM Agent Networkscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Hierarchical Alignment: Enforcing Hierarchical Instruction-Following in LLMs through Logical Consistency
  2. 2026STAIF: A Stage-wise Optimization for Complex Instruction Following
  3. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followers
  4. 2026Training LLMs to Enforce Multi-Level Instruction Hierarchies via Gravity-Weighted Direct Preference Optimization
  5. 2025Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Following
  6. 2025Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models
  7. 2026TinyJudge: Unverifiable Constraint Alignment via Lightweight Specialist Ensembles
  8. 2025Dynamic Policy Induction for Adaptive Prompt Optimization: Bridging the Efficiency-Accuracy Gap via Lightweight Reinforcement Learning