mlsanityData to 7 Oct 2026

Chain-of-Goals Hierarchical Policy for Long-Horizon Offline Goal-Conditioned RL

arXiv 2602.03389 ↗First posted 3 Feb 2026cs.LG cs.AIcited by 5

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

Offline goal-conditioned reinforcement learning remains challenging for long-horizon tasks. While hierarchical approaches mitigate this issue by decomposing tasks, most existing methods rely on separate high- and low-level networks and generate only a single intermediate subgoal, leaving several structural limitations in long-horizon decision-making. To address this limitation, we draw inspiration from chain-of-thought reasoning and propose the Chain-of-Goals Hierarchical Policy (CoGHP), a novel framework that reformulates hierarchical decision-making as autoregressive sequence modeling within a unified architecture. Given a state and a final goal, CoGHP autoregressively generates a sequence of latent subgoals followed by the primitive action, where each latent subgoal acts as a reasoning step that conditions subsequent predictions. To implement this efficiently, we introduce an MLP-Mixer backbone, which supports cross-token communication and captures structural relationships among state, goal, latent subgoals, and action. Across challenging navigation and manipulation benchmarks, CoGHP consistently outperforms strong offline baselines, demonstrating improved performance on long-horizon tasks. Project page: https://wlsdn9350.github.io/projects/coghp/

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 46 of its 52 references that are in the map. First posted on arXiv 3 Feb 2026.

Builds on

46 of 52 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning19 May
  2. 2024OGBench: Benchmarking Offline Goal-Conditioned RL26 Oct
  3. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regression7 Jun
  4. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning24 May
  5. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL9 Feb

Other references · 41

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  6. 2022Are Transformers Effective for Time Series Forecasting?cited by 1,400
  7. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  8. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  9. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 684
  10. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  11. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  12. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
29 more
  1. 2024Robotic Control via Embodied Chain-of-Thought Reasoningcited by 340
  2. 2019Learning Latent Plans from Playcited by 310
  3. 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 276
  4. 2020Critic Regularized Regressioncited by 269
  5. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
  6. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  7. 2023TSMixer: An All-MLP Architecture for Time Series Forecastingcited by 232
  8. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  9. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
  10. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  11. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  12. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 118
  13. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
  14. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  15. 2024NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoningcited by 77
  16. 2024Simple Hierarchical Planning with Diffusioncited by 71
  17. 2023Reinforcement Learning from Passive Data via Latent Intentionscited by 64
  18. 2022Efficient Planning in a Compact Latent Action Spacecited by 57
  19. 2025Training Strategies for Efficient Embodied Reasoningcited by 53
  20. 2022Skill-based Model-based Reinforcement Learningcited by 50
  21. 2017DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrationscited by 43
  22. 2024Vision-Language Models Provide Promptable Representations for Reinforcement Learningcited by 41
  23. 2025Generative Trajectory Stitching through Diffusion Compositioncited by 38
  24. 2025Monte Carlo Tree Diffusion for System 2 Planningcited by 33
  25. 2022DynaMixer: A Vision MLP Architecture with Dynamic Mixingcited by 28
  26. 2022Cross-Domain Transfer via Semantic Skill Imitationcited by 17
  27. 2025Chain-of-Action: Trajectory Autoregressive Modeling for Robotic Manipulationcited by 16
  28. 2024Offline Hierarchical Reinforcement Learning via Inverse Optimizationcited by 4
  29. 2025Dynamic Contrastive Skill Learning with State-Transition Based Skill Clustering and Dynamic Length Adjustmentcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Unifying Object-Centric World Models and Diffusion Policy: A Hierarchical Framework for Multi-Stage Robotic Taskscited by 2
  2. 2026Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learningcited by 1
  3. 2026Beyond Success Rates: Trainability and Extractability for Offline GCRLcited by 0
  4. 2026Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learningcited by 0
  5. 2026Learning Multiple Timescales for Goal-Conditioned Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
  2. 2025Flattening Hierarchies with Policy Bootstrapping
  3. 2026PathBridger: Subgoal Bridges for Offline Goal-Conditioned Reinforcement Learning
  4. 2026STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order
  5. 2026Abstraction for Offline Goal-Conditioned Reinforcement Learning
  6. 2026Efficient Hierarchical Implicit Flow Q-learning for Offline Goal-conditioned Reinforcement Learning
  7. 2026Offline RL with Hierarchical Action Chunking
  8. 2025Structural Information-based Hierarchical Diffusion for Offline Reinforcement Learning