# Offline Goal-conditioned Reinforcement Learning with Quasimetric Representations

arXiv: https://arxiv.org/abs/2509.20478 · HTML: https://mlsanity.com/paper/2509.20478/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 33 of its 50 references that are in the map.
- First posted on arXiv 24 Sep 2025.

## Placed in

- Offline RL and imitation learning from data / [Diffusion and flow policies for offline RL](https://mlsanity.com/offlinerl/diffusion-and-flow-policies-for-offline-rl/)

## Builds on (most cited first, 30 of 33)

- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action](https://mlsanity.com/paper/2207.04429/) (2022)
- [VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training](https://mlsanity.com/paper/2210.00030/) (2022)
- [Control Barrier Functions: Theory and Applications](https://mlsanity.com/paper/1903.11199/) (2019)
- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [OGBench: Benchmarking Offline Goal-Conditioned RL](https://mlsanity.com/paper/2410.20092/) (2024)
- [A Review of Safe Reinforcement Learning: Methods, Theory and Applications](https://mlsanity.com/paper/2205.10330/) (2022)
- [Goal-conditioned Imitation Learning](https://mlsanity.com/paper/1906.05838/) (2019)
- [Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning](https://mlsanity.com/paper/2304.01203/) (2023)
- [Extreme Q-Learning: MaxEnt RL without Entropy](https://mlsanity.com/paper/2301.02328/) (2023)
- [Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency](https://mlsanity.com/paper/1809.01812/) (2018)
- [Temporal Difference Models: Model-Free Deep RL for Model-Based Control](https://mlsanity.com/paper/1802.09081/) (2018)
- [The Safety Filter: A Unified View of Safety-Critical Control in Autonomous Systems](https://mlsanity.com/paper/2309.05837/) (2023)
- [Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discovery](https://mlsanity.com/paper/1907.08225/) (2019)
- [Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making](https://mlsanity.com/paper/2406.17098/) (2024)
- [Model-Based Visual Planning with Self-Supervised Functional Distances](https://mlsanity.com/paper/2012.15373/) (2020)
- [Learning to Act by Predicting the Future](https://mlsanity.com/paper/1611.01779/) (2016)
- [Contrastive Difference Predictive Coding](https://mlsanity.com/paper/2310.20141/) (2023)
- [Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Control](https://mlsanity.com/paper/2307.00117/) (2023)
- [Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of View](https://mlsanity.com/paper/2401.11237/) (2024)
- [Improved Representation of Asymmetrical Distances with Interval Quasimetric Embeddings](https://mlsanity.com/paper/2211.15120/) (2022)
- [Accelerating Goal-Conditioned RL Algorithms and Research](https://mlsanity.com/paper/2408.11052/) (2024)

## Built on by (most cited first, 24 of 24)

- [Q-learning with Adjoint Matching](https://mlsanity.com/paper/2601.14234/) (2026)
- [Trust Region Q Adjoint Matching](https://mlsanity.com/paper/2605.27079/) (2026)
- [NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RL](https://mlsanity.com/paper/2607.07855/) (2026)
- [Action from Adjacent Set in Physical Space Outperforms the Best Prediction in World Models](https://mlsanity.com/paper/2607.23602/) (2026)
- [Hitting Time Isomorphism for Multi-Stage Planning with Foundation Policies](https://mlsanity.com/paper/2605.06470/) (2026)
- [Compositional Transduction with Latent Analogies for Offline Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2605.20609/) (2026)
- [Adaptive Coarse-to-Fine Subgoal Refinement for Long-Horizon Offline Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2605.28127/) (2026)
- [Recursive Value Learning for Long-Horizon Offline Goal-Conditioned RL](https://mlsanity.com/paper/2609.02237/) (2026)
- [The Geometry of Empowerment](https://mlsanity.com/paper/2610.07796/) (2026)
- [Dense and Diverse Goal Coverage in Multi Goal Reinforcement Learning](https://mlsanity.com/paper/2510.25311/) (2025)
- [Beyond Success Rates: Trainability and Extractability for Offline GCRL](https://mlsanity.com/paper/2602.05459/) (2026)
- [Zero-Shot Signal Temporal Logic Planning with Disjunctive Branch Selection in Dynamic Semantic Maps](https://mlsanity.com/paper/2605.01222/) (2026)
- [Functional Graphs for Predicting and Explaining Goal Failure in Sparse Goal-Conditioned RL](https://mlsanity.com/paper/2605.09335/) (2026)
- [Dual Advantage Fields](https://mlsanity.com/paper/2606.04188/) (2026)
- [ChronoForest: Closed-Loop Multi-Tree Diffusion Planning for Efficient Bridge Search and Route Composition](https://mlsanity.com/paper/2606.06618/) (2026)
- [Direction-Conditioned Policies via Compositional Subgoal Scoring for Online Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2606.16515/) (2026)
- [GTRL: Grounding Divide-and-Conquer Value Learning with Temporal Differences](https://mlsanity.com/paper/2609.33259/) (2026)
- [ChronoSRL: Temporal Geometry for Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/2609.36238/) (2026)
- [Do Better Goal Representations Improve Goal-Conditioned Reinforcement Learning?](https://mlsanity.com/paper/2609.39901/) (2026)
- [Learning Transferable Skills using Goal-Conditioned Bisimulation](https://mlsanity.com/paper/2610.00676/) (2026)
- [Learning Goal-Reaching Quasimetric Geometry From Finite-Time Reachability](https://mlsanity.com/paper/2610.00778/) (2026)
- [Direction-Conditioned Policies for Online Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2610.05087/) (2026)
- [Counterfactual Rollouts for Offline Contrastive Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2610.14953/) (2026)
- [Outperforming 1000 Euclidean Layers With Hyperbolic Goal-Conditioned RL](https://mlsanity.com/paper/2610.20009/) (2026)

## Abstract (arXiv)

Approaches for goal-conditioned reinforcement learning (GCRL) often use learned state representations to extract goal-reaching policies. Two frameworks for representation structure have yielded particularly effective GCRL algorithms: (1) *contrastive representations*, in which methods learn "successor features" with a contrastive objective that performs inference over future outcomes, and (2) *temporal distances*, which link the (quasimetric) distance in representation space to the transit time from states to goals. We propose an approach that unifies these two frameworks, using the structure of a quasimetric representation space (triangle inequality) with the right additional constraints to learn successor representations that enable optimal goal-reaching. Unlike past work, our approach is able to exploit a **quasimetric** distance parameterization to learn **optimal** goal-reaching distances, even with **suboptimal** data and in **stochastic** environments. This gives us the best of both worlds: we retain the stability and long-horizon capabilities of Monte Carlo contrastive RL methods, while getting the free stitching capabilities of quasimetric network parameterizations. On existing offline GCRL benchmarks, our representation learning objective improves performance on stitching tasks where methods based on contrastive learning struggle, and on noisy, high-dimensional environments where methods based on quasimetric networks struggle.