Hierarchical Time-aware Bootstrapping for Off-Policy Subgoal Value Learning
On the map: Deep reinforcement learning methods → Deep RL algorithms and representations
Abstract (arXiv)
Off-policy hierarchical reinforcement learning must estimate the values of high-level decisions while the low-level policy changes. HIRO adapts replay data through subgoal relabeling, but after a label change, the value update targets the relabeled subgoal instead of the subgoal the high-level policy originally needed to update. We propose Hierarchical Time-aware Bootstrapping (HTB), which evaluates specified subgoals under the current low-level policy while retaining accumulated task rewards. Remaining execution time distinguishes subgoal continuation from a new high-level decision. Together with primitive-action conditioning, it enables off-policy Bellman updates based on the stationary environment transition law. HTB combines these one-step updates with multi-step suffix returns and truncated relabeling, reducing dependence on intermediate value estimates. A shared value component supports learning across actions, while nonnegative residuals constrain upward corrections relative to that component. At a fixed mixture weight of 0.95, HTB achieves 32.8% AntFall success versus 9.6% for matched local HIRO over five paired seeds at 10M environment steps. Ablations identify contributions from recursive continuation and mixed supervision; fixed-policy tests show more accurate predictions for actions whose returns were excluded from fitting.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 18 of its 25 references that are in the map. First posted on arXiv 4 Oct 2026.
Builds on
18 of 25 references.
References · 18
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
- 2017Hindsight Experience Replaycited by 1,136
- 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
- 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 785
- 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
- 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
- 2016The Option-Critic Architecturecited by 448
- 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
- 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
- 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
- 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 157
- 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
6 more
- 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 110
- 2022Deep Hierarchical Planning from Pixelscited by 101
- 2017Time Limits in Reinforcement Learningcited by 79
- 2020Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learningcited by 58
- 2021Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learningcited by 47
- 2023PEAR: Primitive Enabled Adaptive Relabeling for Boosting Hierarchical Reinforcement Learningcited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Flattening Hierarchies with Policy Bootstrapping
- 2022A Generalized Bootstrap Target for Value-Learning, Efficiently Combining Value and Feature Predictions
- 2026On Semi-Markov Suboptimality in Hierarchical Reinforcement Learning
- 2024Target Networks and Over-parameterization Stabilize Off-policy Bootstrapping with Function Approximation
- 2026Role-Adaptive Policy Optimization for Offline Reinforcement Learning
- 2026Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
- 2026Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
- 2026Horizon Adaptive Offline Policy Learning via Value Stitching