mlsanityData to 14 Oct 2026

Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning

arXiv 2509.06782 ↗First posted 8 Sep 2025cs.LGcited by 15

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

Offline Goal-Conditioned Reinforcement Learning (GCRL) holds great promise for domains such as autonomous navigation and locomotion, where collecting interactive data is costly and unsafe. However, it remains challenging in practice due to the need to learn from datasets with limited coverage of the state-action space and to generalize across long-horizon tasks. To improve on these challenges, we propose a Physics-informed (Pi) regularized loss for value learning, derived from the Eikonal Partial Differential Equation (PDE) and which induces a geometric inductive bias in the learned value function. Unlike generic gradient penalties that are primarily used to stabilize training, our formulation is grounded in continuous-time optimal control and encourages value functions to align with cost-to-go structures. The proposed regularizer is broadly compatible with temporal-difference-based value learning and can be integrated into existing Offline GCRL algorithms. When combined with Hierarchical Implicit Q-Learning (HIQL), the resulting method, Eikonal-regularized HIQL (Eik-HIQL), yields significant improvements in both performance and generalization, with pronounced gains in stitching regimes and large-scale navigation tasks.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 52 references that are in the map. First posted on arXiv 8 Sep 2025.

Builds on

35 of 52 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024OGBench: Benchmarking Offline Goal-Conditioned RL26 Oct
  2. 2023SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learning3 Nov
  3. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?30 May
  4. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regression7 Jun
  5. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RL9 Feb

Other references · 30

  1. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2017Improved Training of Wasserstein GANscited by 2,235
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  8. 2017Hindsight Experience Replaycited by 1,136
  9. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  10. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  11. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  12. 2020Critic Regularized Regressioncited by 275
18 more
  1. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  2. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  3. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  4. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 121
  5. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 89
  6. 2023Reinforcement Learning from Passive Data via Latent Intentionscited by 68
  7. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 59
  8. 2020EikoNet: Solving the Eikonal equation with Deep Neural Networkscited by 50
  9. 2020Hamilton-Jacobi Deep Q-Learning for Deterministic Continuous-Time Systems with Lipschitz Continuous Controlscited by 31
  10. 2021Koopman Q-learning: Offline Reinforcement Learning via Symmetries of Dynamicscited by 27
  11. 2023Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shapingcited by 25
  12. 2022NTFields: Neural Time Fields for Physics-Informed Robot Motion Planningcited by 22
  13. 2023Is Model Ensemble Necessary? Model-based RL via a Single Model with Lipschitz Regularized Value Functioncited by 19
  14. 2023Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RLcited by 18
  15. 2024Physics-informed Neural Motion Planning on Constraint Manifoldscited by 6
  16. 2025Physics-informed Temporal Difference Metric Learning for Robot Motion Planningcited by 6
  17. 2021Online Baum-Welch algorithm for Hierarchical Imitation Learningcited by 3
  18. 2017Nonsmooth optimal value and policy functions in mechanical systems subject to unilateral constraintscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 13.

15 papers.

Sort:

By area:

  1. 2025Test-Time Graph Search for Goal-Conditioned Reinforcement Learningcited by 8
  2. 2025Goal Reaching with Eikonal-Constrained Hierarchical Quasimetric Reinforcement Learningcited by 6
  3. 2026Diffusion Subgoal Planning for Long-Horizon Offline Goal-Conditioned Reinforcement Learningcited by 3
  4. 2026NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RLcited by 2
  5. 2026Action-Sufficient Goal Representationscited by 1
  6. 2026Mollified Value Learningcited by 1
  7. 2026Compositional Transduction with Latent Analogies for Offline Goal-Conditioned Reinforcement Learningcited by 1
  8. 2026Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learningcited by 1
  9. 2026Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learningcited by 1
  10. 2026Beyond Success Rates: Trainability and Extractability for Offline GCRLcited by 0
  11. 2026QHyer: Q-conditioned Hybrid Attention-mamba Transformer for Offline Goal-conditioned RLcited by 0
  12. 2026Functional Graphs for Predicting and Explaining Goal Failure in Sparse Goal-Conditioned RLcited by 0
3 more
  1. 2026Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learningcited by 0
  2. 2026Physics-informed Goal-Conditioned Reinforcement Learning under Hybrid Contact Dynamicscited by 0
  3. 2026Learning Goal-Reaching Quasimetric Geometry From Finite-Time Reachabilitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Efficient Hierarchical Implicit Flow Q-learning for Offline Goal-conditioned Reinforcement Learning
  2. 2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning
  3. 2026GTRL: Grounding Divide-and-Conquer Value Learning with Temporal Differences
  4. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learning
  5. 2026Data-Efficient Hierarchical Goal-Conditioned Reinforcement Learning via Normalizing Flows
  6. 2026Recursive Value Learning for Long-Horizon Offline Goal-Conditioned RL
  7. 2023Provably Efficient Offline Goal-Conditioned Reinforcement Learning with General Function Approximation and Single-Policy Concentrability
  8. 2026Offline RL with Hierarchical Action Chunking