# Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning

arXiv: https://arxiv.org/abs/2605.28127 · HTML: https://mlsanity.com/paper/2605.28127/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 20 of its 28 references that are in the map.
- First posted on arXiv 27 May 2026.

## Placed in

- Offline RL and imitation learning from data / [Diffusion and flow policies for offline RL](https://mlsanity.com/offlinerl/diffusion-and-flow-policies-for-offline-rl/)

## Builds on (most cited first, 20 of 20)

- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Learning Latent Plans from Play](https://mlsanity.com/paper/1903.01973/) (2019)
- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [OGBench: Benchmarking Offline Goal-Conditioned RL](https://mlsanity.com/paper/2410.20092/) (2024)
- [Search on the Replay Buffer: Bridging Planning and Reinforcement Learning](https://mlsanity.com/paper/1906.05253/) (2019)
- [Goal-conditioned Imitation Learning](https://mlsanity.com/paper/1906.05838/) (2019)
- [Goal-Conditioned Reinforcement Learning with Imagined Subgoals](https://mlsanity.com/paper/2107.00541/) (2021)
- [Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning](https://mlsanity.com/paper/2304.01203/) (2023)
- [Learning to Navigate in Cities Without a Map](https://mlsanity.com/paper/1804.00168/) (2018)
- [Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning](https://mlsanity.com/paper/2205.11790/) (2022)
- [Sub-Goal Trees -- a Framework for Goal-Based Reinforcement Learning](https://mlsanity.com/paper/2002.12361/) (2020)
- [Offline Goal-conditioned Reinforcement Learning with Quasimetric Representations](https://mlsanity.com/paper/2509.20478/) (2025)
- [Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2505.12737/) (2025)
- [Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2509.06782/) (2025)
- [Subgoal Diffuser: Coarse-to-fine Subgoal Generation to Guide Model Predictive Control for Robot Manipulation](https://mlsanity.com/paper/2403.13085/) (2024)
- [Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2506.07744/) (2025)
- [Transitive RL: Value Learning via Divide and Conquer](https://mlsanity.com/paper/2510.22512/) (2025)
- [Structural Information-based Hierarchical Diffusion for Offline Reinforcement Learning](https://mlsanity.com/paper/2509.21942/) (2025)
- [Strict Subgoal Execution: Reliable Long-Horizon Planning in Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2506.21039/) (2025)

## Built on by (most cited first, 1 of 1)

- [Learning Multiple Timescales for Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2610.00849/) (2026)

## Abstract (arXiv)

Offline goal-conditioned reinforcement learning (GCRL) is challenging in long-horizon tasks, where distant state--goal pairs provide weak supervision and value estimates become vulnerable to accumulated bootstrapping errors. Hierarchical methods mitigate this difficulty by introducing intermediate subgoals, but fixed temporal abstractions or fixed hierarchy depths can be mismatched to state--goal pairs with different reachability horizons. We propose Coarse-to-Fine Hierarchical Goal Reinforcement Learning (CFHRL), a fully offline GCRL framework that adaptively refines distant goals before execution. Starting from the final goal, CFHRL recursively proposes intermediate targets, trained from replay-supported candidates, and stops refinement once the current target is estimated to be locally executable by a learned reachability cost. The key idea is that a subgoal need not be an exact midpoint or globally optimal waypoint; it only needs to provide reliable progress and reduce the remaining reaching difficulty, enabling subsequent refinement over shorter horizons. A stylized analysis further supports the robustness of approximate recursive contraction. Experiments on OGBench show substantial gains on several long-horizon tasks, with ablations validating the proposed refinement and stopping mechanisms