mlsanityData to 14 Oct 2026

Reinforcement Learning with Anticipation: A Hierarchical Approach for Long-Horizon Tasks

arXiv 2509.05545 ↗First posted 6 Sep 2025cs.LGcited by 1

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Solving long-horizon goal-conditioned tasks remains a significant challenge in reinforcement learning (RL). Hierarchical reinforcement learning (HRL) addresses this by decomposing tasks into more manageable sub-tasks, but the automatic discovery of the hierarchy and the joint training of multi-level policies often suffer from instability and can lack theoretical guarantees. In this paper, we introduce Reinforcement Learning with Anticipation (RLA), a principled and potentially scalable framework designed to address these limitations. The RLA agent learns two synergistic models: a low-level, goal-conditioned policy that learns to reach specified subgoals, and a high-level anticipation model that functions as a planner, proposing intermediate subgoals on the optimal path to a final goal. The key feature of RLA is the training of the anticipation model, which is guided by a principle of value geometric consistency, regularized to prevent degenerate solutions. We present proofs that RLA approaches the globally optimal policy under various conditions, establishing a principled and convergent method for hierarchical planning and execution in long-horizon goal-conditioned tasks.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 9 of its 28 references that are in the map. First posted on arXiv 6 Sep 2025.

Builds on

9 of 28 references.

References · 9

  1. 2015Continuous control with deep reinforcement learningcited by 3,407
  2. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  3. 2017Hindsight Experience Replaycited by 1,136
  4. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  5. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  6. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  7. 2016Value Iteration Networkscited by 160
  8. 2020Near-optimal Regret Bounds for Stochastic Shortest Pathcited by 50
  9. 2021Learning Stochastic Shortest Path with Linear Function Approximationcited by 23

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025ReLAM: Learning Anticipation Model for Rewarding Visual Robotic Manipulationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Anticipation-VLA: Solving Long-Horizon Embodied Tasks via Anticipation-based Subgoal Generation
  2. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning
  3. 2026Dynamical Priors as a Training Objective in Reinforcement Learning
  4. 2025Strict Subgoal Execution: Reliable Long-Horizon Planning in Hierarchical Reinforcement Learning
  5. 2026Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
  6. 2025DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning
  7. 2026On Semi-Markov Suboptimality in Hierarchical Reinforcement Learning
  8. 2018Hierarchical Imitation and Reinforcement Learning