mlsanityData to 14 Oct 2026

Direction-Conditioned Policies via Compositional Subgoal Scoring for Online Goal-Conditioned Reinforcement Learning

arXiv 2606.16515 ↗First posted 15 Jun 2026cs.LG cs.AI cs.ROcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Hamilton-Jacobi-Bellman theory implies that the optimal goal-conditioned action depends on the goal only through the gradient of the goal-reaching distance at the current state, yet standard online GCRL still conditions the actor on the raw goal -- a signal that is geometrically uninformative when the goal is far from the data distribution. We propose Direction-Conditioned Policies (DCP), a fully online method that decomposes goal-reaching into two components sharing one InfoNCE representation psi: a subgoal-scoring step that selects a visited state zt aligned with the final goal g in psig, and a direction-conditioned actor that consumes the unit direction dt and magnitude rt from psi(st) to psi(zt). The two components train jointly, factor cleanly at deployment (subgoal scoring is removed, while direction conditioning remains with g in place of zt), and admit independent modification at the same (dt,rt) interface. We prove three results. First, direction sufficiency under HJB: the optimal action under control-affine dynamics depends on the goal only through the value gradient. Second, a quantitative bound showing that, under mild conditions on the learned representation and assuming the scoring rule returns an on-path zt, the actor's conditioning input at training and at deployment coincide up to representation error and geodesic slack. Third, a controllable-subspace characterization of when directional conditioning fails. Across nine environments, DCP improves over Contrastive RL on most final metrics, with the largest gains on manipulation and obstacle-interaction tasks; a qualitative analysis of the learned psi-distance landscape shows the contrastive representation behaves as an online quasimetric encoding environment topology, and the single failure case (AntSoccer) localizes to a learned-gradient pathology that the theory anticipates.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 22 of its 28 references that are in the map. First posted on arXiv 15 Jun 2026.

Builds on

22 of 28 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Goal Reaching with Eikonal-Constrained Hierarchical Quasimetric Reinforcement Learning12 Dec
  2. 2025Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learning11 Nov
  3. 2025Dual Goal Representations8 Oct
  4. 2025Horizon Reduction Makes RL Scalable4 Jun
  5. 20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities19 Mar
  6. 2025Horizon Generalization in Reinforcement Learning6 Jan
  7. 2024Reduce, Reuse, Recycle: Categories for Compositional Reinforcement Learning23 Aug
  8. 2024Accelerating Goal-Conditioned RL Algorithms and Research20 Aug
  9. 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making24 Jun
  10. 2024Foundation Policies with Hilbert Representations23 Feb

Other references · 12

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  3. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  4. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  5. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 121
  6. 2026LeRobot: An Open-Source Library for End-to-End Robot Learningcited by 73
  7. 2023SkillDiffuser: Interpretable Hierarchical Planning via Skill Abstractions in Diffusion-Based Task Executioncited by 70
  8. 2020World Model as a Graph: Learning Latent Landmarks for Planningcited by 69
  9. 2025Offline Goal-conditioned Reinforcement Learning with Quasimetric Representationscited by 24
  10. 2024A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoalscited by 23
  11. 2024Compositional Automata Embeddings for Goal-Conditioned Reinforcement Learningcited by 19
  12. 2019Plan Arithmetic: Compositional Plan Vectors for Multi-Task Controlcited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Direction-Conditioned Policies for Online Goal-Conditioned Reinforcement Learning
  2. 2023Learning to Reach Goals via Diffusion
  3. 2026Locally-Guided Actor-Critic: Training a Goal-conditioned Actor with a Subgoal-aware Critic
  4. 2026Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
  5. 2026World-Model Policy Arbiter for Goal-Conditioned Reinforcement Learning
  6. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regression
  7. 2026Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning
  8. 2025Flattening Hierarchies with Policy Bootstrapping