mlsanityData to 14 Oct 2026

Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach

arXiv 2411.00361 ↗First posted 1 Nov 2024cs.LGcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Hierarchical reinforcement learning (HRL) enables agents to solve complex, long-horizon tasks by decomposing them into manageable sub-tasks. However, HRL methods face two fundamental challenges: (i) non-stationarity caused by the evolving lower-level policy during training, which destabilizes higher-level learning, and (ii) the generation of infeasible subgoals that lower-level policies cannot achieve. To address these challenges, we introduce DIPPER, a novel HRL framework that formulates goal-conditioned HRL as a bi-level optimization problem and leverages direct preference optimization (DPO) to train the higher-level policy. By learning from stationary preference comparisons over subgoal sequences rather than rewards that depend on the evolving lower-level policy, DIPPER mitigates the impact of non-stationarity on hierarchical learning. To address infeasible subgoals, DIPPER incorporates lower-level value function regularization that encourages the higher-level policy to propose achievable subgoals. We also introduce two novel metrics to quantitatively verify that DIPPER mitigates non-stationarity and infeasible subgoal generation issues in HRL. We perform empirical evaluations on challenging robotic navigation and manipulation benchmarks and show that DIPPER achieves upto 40% improvements over state-of-the-art baselines, demonstrating that preference-based methods can effectively alleviate persistent challenges in hierarchical

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 28 of its 42 references that are in the map. First posted on arXiv 1 Nov 2024.

Builds on

28 of 42 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024PIPER: Primitive-Informed Preference-based Hierarchical Reinforcement Learning via Hindsight Relabeling20 Apr
  2. 2023Contrastive Preference Learning: Learning from Human Feedback without RL20 Oct
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training9 Jun
  5. 2020Weak Human Preference Supervision For Deep Reinforcement Learning25 Jul
  6. 2018Reward learning from human preferences and demonstrations in Atari15 Nov
  7. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 21

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2017Hindsight Experience Replaycited by 1,136
  3. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  4. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  5. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  6. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  7. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  8. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 204
  9. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 193
  10. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 134
  11. 2022BOME! Bilevel Optimization Made Easy: A Simple First-Order Approachcited by 105
  12. 2021Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Taskscited by 78
9 more
  1. 2019Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?cited by 68
  2. 2021Accelerating Robotic Reinforcement Learning via Parameterized Action Primitivescited by 65
  3. 2017When Waiting is not an Option : Learning Options with a Deliberation Costcited by 58
  4. 2021Hierarchical Reinforcement Learning By Discovering Intrinsic Optionscited by 58
  5. 2019Sub-policy Adaptation for Hierarchical Reinforcement Learningcited by 47
  6. 2018Learning Curriculum Policies for Reinforcement Learningcited by 35
  7. 2022State-Conditioned Adversarial Subgoal Generationcited by 8
  8. 2023PEAR: Primitive Enabled Adaptive Relabeling for Boosting Hierarchical Reinforcement Learningcited by 3
  9. 2023CRISP: Curriculum Inducing Primitive Informed Subgoal Prediction for Hierarchical Reinforcement Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026HiPO: Hierarchical Preference Optimization for Adaptive Reasoning in LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024DIPPER: Direct Preference Optimization to Accelerate Primitive-Enabled Hierarchical Reinforcement Learning
  2. 2026Hierarchical Decision Making with Structured Policies: A Principled Design via Inverse Optimization
  3. 2024Exploring the limits of Hierarchical World Models in Reinforcement Learning
  4. 2024A Provably Efficient Option-Based Algorithm for both High-Level and Low-Level Learning
  5. 2025Realizable Abstractions: Near-Optimal Hierarchical Reinforcement Learning
  6. 2026Sample Efficient Hierarchical Reinforcement Learning via Best Policy Identification
  7. 2024Bidirectional-Reachable Hierarchical Reinforcement Learning with Mutually Responsive Policies
  8. 2024Reinforcement Learning with Options and State Representation