# PEAR: Primitive Enabled Adaptive Relabeling for Boosting Hierarchical Reinforcement Learning

arXiv: https://arxiv.org/abs/2306.06394 · HTML: https://mlsanity.com/paper/2306.06394/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 55 references that are in the map.
- First posted on arXiv 10 Jun 2023.

## Builds on (most cited first, 30 of 35)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning](https://mlsanity.com/paper/1910.11956/) (2019)
- [Overcoming Exploration in Reinforcement Learning with Demonstrations](https://mlsanity.com/paper/1709.10089/) (2017)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [Accelerating Reinforcement Learning with Learned Skill Priors](https://mlsanity.com/paper/2010.11944/) (2020)
- [OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning](https://mlsanity.com/paper/2010.13611/) (2020)
- [Goal-conditioned Imitation Learning](https://mlsanity.com/paper/1906.05838/) (2019)
- [Parrot: Data-Driven Behavioral Priors for Reinforcement Learning](https://mlsanity.com/paper/2011.10024/) (2020)
- [Goal-Conditioned Reinforcement Learning with Imagined Subgoals](https://mlsanity.com/paper/2107.00541/) (2021)
- [Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks](https://mlsanity.com/paper/2110.03655/) (2021)
- [Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?](https://mlsanity.com/paper/1909.10618/) (2019)
- [Latent Space Policies for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1804.02808/) (2018)
- [Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives](https://mlsanity.com/paper/2110.15360/) (2021)
- [TACO: Learning Task Decomposition via Temporal Alignment for Control](https://mlsanity.com/paper/1803.01840/) (2018)
- [When Waiting is not an Option : Learning Options with a Deliberation Cost](https://mlsanity.com/paper/1709.04571/) (2017)
- [Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2006.11485/) (2020)
- [Multi-Level Discovery of Deep Options](https://mlsanity.com/paper/1703.08294/) (2017)
- [DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrations](https://mlsanity.com/paper/1710.05421/) (2017)
- [Data-efficient Hindsight Off-policy Option Learning](https://mlsanity.com/paper/2007.15588/) (2020)
- [Deep Q-learning from Demonstrations](https://mlsanity.com/paper/1704.03732/) (2017)
- [Learnings Options End-to-End for Continuous Action Tasks](https://mlsanity.com/paper/1712.00004/) (2017)

## Built on by (most cited first, 3 of 3)

- [Training Fast Robot Policies with Slow Foundation Models](https://mlsanity.com/paper/2406.05881/) (2024)
- [Direct Preference Optimization for Primitive-Enabled Hierarchical RL: A Bilevel Approach](https://mlsanity.com/paper/2411.00361/) (2024)
- [Hierarchical Time-aware Bootstrapping for Off-Policy Subgoal Value Learning](https://mlsanity.com/paper/2610.05446/) (2026)

## Abstract (arXiv)

Hierarchical reinforcement learning (HRL) has the potential to solve complex long horizon tasks using temporal abstraction and increased exploration. However, hierarchical agents are difficult to train due to inherent non-stationarity. We present primitive enabled adaptive relabeling (PEAR), a two-phase approach where we first perform adaptive relabeling on a few expert demonstrations to generate efficient subgoal supervision, and then jointly optimize HRL agents by employing reinforcement learning (RL) and imitation learning (IL). We perform theoretical analysis to bound the sub-optimality of our approach and derive a joint optimization framework using RL and IL. Since PEAR utilizes only a few expert demonstrations and considers minimal limiting assumptions on the task structure, it can be easily integrated with typical off-policy RL algorithms to produce a practical HRL approach. We perform extensive experiments on challenging environments and show that PEAR is able to outperform various hierarchical and non-hierarchical baselines and achieve upto 80% success rates in complex sparse robotic control tasks where other baselines typically fail to show significant progress. We also perform ablations to thoroughly analyse the importance of our various design choices. Finally, we perform real world robotic experiments on complex tasks and demonstrate that PEAR consistently outperforms the baselines.