mlsanityData to 14 Oct 2026

A tale of two goals: leveraging sequentiality in multi-goal scenarios

arXiv 2503.21677 ↗First posted 27 Mar 2025cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Several hierarchical reinforcement learning methods leverage planning to create a graph or sequences of intermediate goals, guiding a lower-level goal-conditioned (GC) policy to reach some final goals. The low-level policy is typically conditioned on the current goal, with the aim of reaching it as quickly as possible. However, this approach can fail when an intermediate goal can be reached in multiple ways, some of which may make it impossible to continue toward subsequent goals. To address this issue, we introduce two instances of Markov Decision Process (MDP) where the optimization objective favors policies that not only reach the current goal but also subsequent ones. In the first, the agent is conditioned on both the current and final goals, while in the second, it is conditioned on the next two goals in the sequence. We conduct a series of experiments on navigation and pole-balancing tasks in which sequences of intermediate goals are given. By evaluating policies trained with TD3+HER on both the standard GC-MDP and our proposed MDPs, we show that, in most cases, conditioning on the next two goals improves stability and sample efficiency over other approaches.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 13 of its 19 references that are in the map. First posted on arXiv 27 Mar 2025.

Builds on

13 of 19 references.

References · 13

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  4. 2017Hindsight Experience Replaycited by 1,136
  5. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  6. 2016The Option-Critic Architecturecited by 448
  7. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 194
  8. 2019Planning with Goal-Conditioned Policiescited by 149
  9. 2018Temporal Difference Models: Model-Free Deep RL for Model-Based Controlcited by 89
  10. 2019Mapping State Space using Landmarks for Universal Goal Reachingcited by 44
  11. 2022DHRL: A Graph-Based Approach for Long-Horizon and Sparse Hierarchical Reinforcement Learningcited by 19
  12. 2023Imitating Graph-Based Planning with Goal-Conditioned Policiescited by 13
1 more
  1. 2021Adjacency constraint for efficient hierarchical reinforcement learningcited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026On Semi-Markov Suboptimality in Hierarchical Reinforcement Learning
  2. 2026Multi-level meta-reinforcement learning with skill-based curriculum
  3. 2025Proposing Hierarchical Goal-Conditioned Policy Planning in Multi-Goal Reinforcement Learning
  4. 2026Abstraction for Offline Goal-Conditioned Reinforcement Learning
  5. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning
  6. 2024Performative Reinforcement Learning with Linear Markov Decision Process
  7. 2025Why Goal-Conditioned Reinforcement Learning Works: Relation to Dual Control
  8. 2025Reinforcement Learning with Anticipation: A Hierarchical Approach for Long-Horizon Tasks