mlsanityData to 7 Oct 2026

GTRL: Grounding Divide-and-Conquer Value Learning with Temporal Differences

arXiv 2609.33259 ↗First posted 27 Sep 2026cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL also fits: Offline RL algorithms

Abstract (arXiv)

In offline goal-conditioned reinforcement learning (GCRL), divide-and-conquer scales to long horizons by joining two shorter segments at a subgoal. However, under stochastic dynamics, the base case of this rule values the luckiest trajectories through the data. The subgoal must also lie on a shared trajectory, so a state-goal pair that no trajectory connects gets no value update at all. To address both, we present Grounded Transitive RL (GTRL), an offline GCRL value learning algorithm that grounds the divide-and-conquer update with a one-step TD target. Over a single step, TD is correct, as its target averages over the successors and needs no subgoal. GTRL adds this target to the composition rather than replacing it, so every pair receives an update, and the composition still carries the long horizon. GTRL also corrects the bias from hindsight relabeling by reweighting each goal against how reachable it was from other successors. We evaluate our algorithm on nineteen OGBench tasks spanning stochastic, deterministic, and stitching environments, where it achieves the highest average success rate. Code will be released soon.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 45 of its 56 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

45 of 56 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Offline Goal-conditioned Reinforcement Learning with Quasimetric Representations24 Sep
  2. 2025Flow Q-Learning4 Feb
  3. 2024OGBench: Benchmarking Offline Goal-Conditioned RL26 Oct
  4. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL9 Feb

Other references · 41

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2017Hindsight Experience Replaycited by 1,096
  7. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  8. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  10. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  11. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  12. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
29 more
  1. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  2. 2021Offline RL Without Off-Policy Evaluationcited by 150
  3. 2019Planning with Goal-Conditioned Policiescited by 144
  4. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  5. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
  6. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  7. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 118
  8. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
  9. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 99
  10. 2025Reinforcement Learning with Action Chunkingcited by 97
  11. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
  12. 2021Learning Successor States and Goal-Dependent Values: A Mathematical Viewpointcited by 67
  13. 2022You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environmentscited by 62
  14. 2025Horizon Reduction Makes RL Scalablecited by 55
  15. 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 53
  16. 2022Dichotomy of Control: Separating What You Can Control from What You Cannotcited by 47
  17. 2020Model-Based Visual Planning with Self-Supervised Functional Distancescited by 41
  18. 2023Contrastive Difference Predictive Codingcited by 35
  19. 2020Sub-Goal Trees -- a Framework for Goal-Based Reinforcement Learningcited by 27
  20. 2020An Inductive Bias for Distances: Neural Nets that Respect the Triangle Inequalitycited by 24
  21. 2022Improved Representation of Asymmetrical Distances with Interval Quasimetric Embeddingscited by 22
  22. 2020Divide-and-Conquer Monte Carlo Tree Search For Goal-Directed Planningcited by 19
  23. 2021Solving Compositional Reinforcement Learning Problems via Task Reductioncited by 19
  24. 2025Decoupled Q-Chunkingcited by 18
  25. 2025Transitive RL: Value Learning via Divide and Conquercited by 12
  26. 2018Floyd-Warshall Reinforcement Learning: Learning from Past Experiences to Reach New Goalscited by 8
  27. 2021Unbiased Methods for Multi-Goal Reinforcement Learningcited by 7
  28. 2022USHER: Unbiased Sampling for Hindsight Experience Replaycited by 6
  29. 2026NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RLcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Recursive Value Learning for Long-Horizon Offline Goal-Conditioned RL
  2. 2026Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learning
  3. 2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning
  4. 2023Provably Efficient Offline Goal-Conditioned Reinforcement Learning with General Function Approximation and Single-Policy Concentrability
  5. 2026Learning Multiple Timescales for Goal-Conditioned Reinforcement Learning
  6. 2025Flattening Hierarchies with Policy Bootstrapping
  7. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regression
  8. 2025Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning