mlsanityData to 7 Oct 2026

Recursive Value Learning for Long-Horizon Offline Goal-Conditioned RL

arXiv 2609.02237 ↗First posted 2 Sep 2026cs.LG cs.ROcited by 1

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

Scaling offline goal-conditioned reinforcement learning (GCRL) to long-horizon tasks is difficult because (1) long-range value learning depends on shorter-range estimates that may still be inaccurate, and (2) max-based value backups can amplify overestimation through repeated propagation. We propose DCRL (Divide-and-Conquer RL), which recursively decomposes each trajectory segment into a balanced binary tree and trains the values from leaves to root. Each parent is therefore updated only after its children, using an exact factorization of the observed route rather than selecting among noisy alternatives. Since this objective learns values along demonstrated routes that are not necessarily optimal, DCRL jointly propagates values across trajectories to discover shorter routes. Thanks to the balanced binary tree, DCRL reduces worst-case bootstrap depth from linear to logarithmic, and this shorter dependency structure empirically corresponds to much slower error accumulation. Across diverse goal-reaching tasks, DCRL substantially outperforms prior flat offline GCRL methods, and on the five most challenging long-horizon OGBench tasks, it improves the best prior average score from 55 to 64, surpassing all flat and hierarchical baselines.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 34 of its 43 references that are in the map. First posted on arXiv 2 Sep 2026.

Builds on

34 of 43 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Offline Goal-conditioned Reinforcement Learning with Quasimetric Representations24 Sep
  2. 2024OGBench: Benchmarking Offline Goal-Conditioned RL26 Oct
  3. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  4. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling29 Sep

Other references · 30

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  5. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  6. 2017Hindsight Experience Replaycited by 1,096
  7. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  8. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  9. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  10. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  11. 2019Learning Latent Plans from Playcited by 310
  12. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
18 more
  1. 2017Automated Curriculum Learning for Neural Networkscited by 251
  2. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  3. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  4. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  5. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 118
  6. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
  7. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 99
  8. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
  9. 2025Horizon Reduction Makes RL Scalablecited by 55
  10. 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 53
  11. 2022Skill-based Model-based Reinforcement Learningcited by 50
  12. 2020Model-Based Visual Planning with Self-Supervised Functional Distancescited by 41
  13. 2023Contrastive Difference Predictive Codingcited by 35
  14. 2021Regularized Behavior Value Estimationcited by 32
  15. 2020Sub-Goal Trees -- a Framework for Goal-Based Reinforcement Learningcited by 27
  16. 2020Divide-and-Conquer Monte Carlo Tree Search For Goal-Directed Planningcited by 19
  17. 2025Transitive RL: Value Learning via Divide and Conquercited by 12
  18. 2018Floyd-Warshall Reinforcement Learning: Learning from Past Experiences to Reach New Goalscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Learning Multiple Timescales for Goal-Conditioned Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026GTRL: Grounding Divide-and-Conquer Value Learning with Temporal Differences
  2. 2026Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning
  3. 2025Flattening Hierarchies with Policy Bootstrapping
  4. 2026Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
  5. 2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning
  6. 2026Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
  7. 2023Provably Efficient Offline Goal-Conditioned Reinforcement Learning with General Function Approximation and Single-Policy Concentrability
  8. 2023Goal-conditioned Offline Reinforcement Learning through State Space Partitioning