mlsanityData to 7 Oct 2026

Hierarchical Reward Design from Language: Enhancing Alignment of Agent Behavior with Human Specifications

arXiv 2602.18582 ↗First posted 20 Feb 2026cs.AI cs.CL cs.HCcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

When training artificial intelligence (AI) to perform tasks, humans often care not only about whether a task is completed but also how it is performed. As AI agents tackle increasingly complex tasks, aligning their behavior with human-provided specifications becomes critical for responsible AI deployment. Reward design provides a direct channel for such alignment by translating human expectations into reward functions that guide reinforcement learning (RL). However, existing methods are often too limited to capture nuanced human preferences that arise in long-horizon tasks. Hence, we introduce Hierarchical Reward Design from Language (HRDL): a problem formulation that extends classical reward design to encode richer behavioral specifications for hierarchical RL agents. We further propose Language to Hierarchical Rewards (L2HR) as a solution to HRDL. Experiments show that AI agents trained with rewards designed via L2HR not only complete tasks effectively but also better adhere to human specifications. Together, HRDL and L2HR advance the research on human-aligned AI agents.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 81 references that are in the map. First posted on arXiv 20 Feb 2026.

Builds on

24 of 81 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learning20 Sep
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 22

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  5. 2016Generative Adversarial Imitation Learningcited by 1,576
  6. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  7. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  8. 2016The Option-Critic Architecturecited by 433
  9. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  10. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  11. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 282
  12. 2017Inverse Reward Designcited by 246
10 more
  1. 2020Reward Machines: Exploiting Reward Function Structure in Reinforcement Learningcited by 183
  2. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  3. 2023LLM-Powered Hierarchical Language Agent for Real-time Human-AI Coordinationcited by 48
  4. 2019DAC: The Double Actor-Critic Architecture for Learning Optionscited by 43
  5. 2018Learning from Richer Human Guidance: Augmenting Comparison-Based Learning with Feature Queriescited by 16
  6. 2024Extracting Heuristics from Large Language Models for Reward Shaping in Reinforcement Learningcited by 7
  7. 2024IDIL: Imitation Learning of Intent-Driven Expert Behaviorcited by 5
  8. 2025Hierarchical Imitation Learning of Team Behavior from Heterogeneous Demonstrationscited by 2
  9. 2025Socratic: Enhancing Human Teamwork via AI-enabled Coachingcited by 2
  10. 2024Task-priority Intermediated Hierarchical Distributed Policies: Reinforcement Learning of Adaptive Multi-robot Cooperative Transportcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The End of Reward Engineering: How LLMs Are Redefining Multi-Agent Coordination
  2. 2023Reward Design with Language Models
  3. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Design
  4. 2025Reflective Verbal Reward Design for Pluralistic Alignment
  5. 2018Scalable agent alignment via reward modeling: a research direction
  6. 2024REvolve: Reward Evolution with Large Language Models using Human Feedback
  7. 2025Ground-Compose-Reinforce: Grounding Language in Agentic Behaviours using Limited Data
  8. 2026Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling