# Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learning

arXiv: https://arxiv.org/abs/2511.07730 · HTML: https://mlsanity.com/paper/2511.07730/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 46 of its 53 references that are in the map.
- First posted on arXiv 11 Nov 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 30 of 46)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [BridgeData V2: A Dataset for Robot Learning at Scale](https://mlsanity.com/paper/2308.12952/) (2023)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [Efficient Online Reinforcement Learning with Offline Data](https://mlsanity.com/paper/2302.02948/) (2023)
- [Safe and Efficient Off-Policy Reinforcement Learning](https://mlsanity.com/paper/1606.02647/) (2016)
- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [OGBench: Benchmarking Offline Goal-Conditioned RL](https://mlsanity.com/paper/2410.20092/) (2024)
- [Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models](https://mlsanity.com/paper/2502.19417/) (2025)
- [Steering Your Diffusion Policy with Latent Space Reinforcement Learning](https://mlsanity.com/paper/2506.15799/) (2025)
- [Goal-conditioned Imitation Learning](https://mlsanity.com/paper/1906.05838/) (2019)
- [HIQL: Offline Goal-Conditioned RL with Latent States as Actions](https://mlsanity.com/paper/2307.11949/) (2023)
- [SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning](https://mlsanity.com/paper/2401.16013/) (2024)
- [Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning](https://mlsanity.com/paper/2304.01203/) (2023)
- [Extreme Q-Learning: MaxEnt RL without Entropy](https://mlsanity.com/paper/2301.02328/) (2023)
- [Reinforcement Learning with Action Chunking](https://mlsanity.com/paper/2507.07969/) (2025)
- [Steering Your Generalists: Improving Robotic Foundation Models via Value Guidance](https://mlsanity.com/paper/2410.13816/) (2024)
- [C-Learning: Learning to Achieve Goals via Recursive Classification](https://mlsanity.com/paper/2011.08909/) (2020)
- [Foundation Policies with Hilbert Representations](https://mlsanity.com/paper/2402.15567/) (2024)
- [Is Value Learning Really the Main Bottleneck in Offline RL?](https://mlsanity.com/paper/2406.09329/) (2024)
- [Vision-Language-Action Models for Robotics: A Review Towards Real-World Applications](https://mlsanity.com/paper/2510.07077/) (2025)
- [Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone](https://mlsanity.com/paper/2412.06685/) (2024)

## Built on by (most cited first, 8 of 8)

- [Beyond Success Rates: Trainability and Extractability for Offline GCRL](https://mlsanity.com/paper/2602.05459/) (2026)
- [Dual Advantage Fields](https://mlsanity.com/paper/2606.04188/) (2026)
- [ChronoForest: Closed-Loop Multi-Tree Diffusion Planning for Efficient Bridge Search and Route Composition](https://mlsanity.com/paper/2606.06618/) (2026)
- [Learning Object Manipulation from Scratch via Contrastive Interaction](https://mlsanity.com/paper/2606.11525/) (2026)
- [Direction-Conditioned Policies via Compositional Subgoal Scoring for Online Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2606.16515/) (2026)
- [ChronoSRL: Temporal Geometry for Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/2609.36238/) (2026)
- [Learning Commute-Time-Preserving World Models for Planning](https://mlsanity.com/paper/2610.01373/) (2026)
- [Outperforming 1000 Euclidean Layers With Hyperbolic Goal-Conditioned RL](https://mlsanity.com/paper/2610.20009/) (2026)

## Abstract (arXiv)

Learning how to reach goals in an environment is a longstanding challenge in AI, yet reasoning over long horizons remains a challenge for modern methods. The key question is how to estimate the temporal distance between pairs of observations. While temporal difference methods leverage local updates to provide optimality guarantees, they often perform worse than Monte Carlo methods that perform global updates (e.g., with multi-step returns), which lack such guarantees. We show how these approaches can be integrated into a practical offline GCRL method that fits a quasimetric distance using a multistep Monte-Carlo return. We show our method outperforms existing offline GCRL methods on long-horizon simulated tasks with up to 4000 steps, even with visual observations. We also demonstrate that our method can enable stitching in the real-world robotic manipulation domain (Bridge setup). Our approach is the first end-to-end offline GCRL method that enables multistep stitching in this real-world manipulation domain from an unlabeled offline dataset of visual observations and demonstrate robust horizon generalization.