mlsanityData to 14 Oct 2026

Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learning

arXiv 2511.07730 ↗First posted 11 Nov 2025cs.LG cs.ROcited by 8

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Learning how to reach goals in an environment is a longstanding challenge in AI, yet reasoning over long horizons remains a challenge for modern methods. The key question is how to estimate the temporal distance between pairs of observations. While temporal difference methods leverage local updates to provide optimality guarantees, they often perform worse than Monte Carlo methods that perform global updates (e.g., with multi-step returns), which lack such guarantees. We show how these approaches can be integrated into a practical offline GCRL method that fits a quasimetric distance using a multistep Monte-Carlo return. We show our method outperforms existing offline GCRL methods on long-horizon simulated tasks with up to 4000 steps, even with visual observations. We also demonstrate that our method can enable stitching in the real-world robotic manipulation domain (Bridge setup). Our approach is the first end-to-end offline GCRL method that enables multistep stitching in this real-world manipulation domain from an unlabeled offline dataset of visual observations and demonstrate robust horizon generalization.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 46 of its 53 references that are in the map. First posted on arXiv 11 Nov 2025.

Builds on

46 of 53 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Reinforcement Learning with Action Chunking10 Jul
  2. 2025Horizon Reduction Makes RL Scalable4 Jun
  3. 2025Horizon Generalization in Reinforcement Learning6 Jan
  4. 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making24 Jun
  5. 2024Foundation Policies with Hilbert Representations23 Feb
  6. 2024Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of View20 Jan

Other references · 40

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2024π0: A Vision-Language-Action Flow Model for General Robot Controlcited by 2,407
  6. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  9. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  10. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  11. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
  12. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
28 more
  1. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  2. 2016Safe and Efficient Off-Policy Reinforcement Learningcited by 264
  3. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  4. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  5. 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 221
  6. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 169
  7. 2019Goal-conditioned Imitation Learningcited by 162
  8. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  9. 2024SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learningcited by 136
  10. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 121
  11. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 116
  12. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 106
  13. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 103
  14. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 89
  15. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 84
  16. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 79
  17. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 76
  18. 2021Learning Successor States and Goal-Dependent Values: A Mathematical Viewpointcited by 70
  19. 2024RLDG: Robotic Generalist Policy Distillation via Reinforcement Learningcited by 47
  20. 2025FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Controlcited by 44
  21. 2017Multi-step Reinforcement Learning: A Unifying Algorithmcited by 42
  22. 2020An Inductive Bias for Distances: Neural Nets that Respect the Triangle Inequalitycited by 24
  23. 2023Grow Your Limits: Continuous Improvement with Real-World RL for Robotic Locomotioncited by 24
  24. 2023Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Datacited by 23
  25. 2024Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inferencecited by 17
  26. 2025Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returnscited by 16
  27. 2025Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Followingcited by 14
  28. 2022Metric Residual Networks for Sample Efficient Goal-Conditioned Reinforcement Learningcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026Beyond Success Rates: Trainability and Extractability for Offline GCRLcited by 0
  2. 2026Dual Advantage Fieldscited by 0
  3. 2026ChronoForest: Closed-Loop Multi-Tree Diffusion Planning for Efficient Bridge Search and Route Compositioncited by 0
  4. 2026Learning Object Manipulation from Scratch via Contrastive Interactioncited by 0
  5. 2026Direction-Conditioned Policies via Compositional Subgoal Scoring for Online Goal-Conditioned Reinforcement Learningcited by 0
  6. 2026ChronoSRL: Temporal Geometry for Self-Supervised Reinforcement Learningcited by 0
  7. 2026Learning Commute-Time-Preserving World Models for Planningcited by 0
  8. 2026Outperforming 1000 Euclidean Layers With Hyperbolic Goal-Conditioned RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Offline Goal-conditioned Reinforcement Learning with Quasimetric Representations
  2. 2026PathBridger: Subgoal Bridges for Offline Goal-Conditioned Reinforcement Learning
  3. 2024Stitching Sub-Trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL
  4. 2026Learning Goal-Reaching Quasimetric Geometry From Finite-Time Reachability
  5. 2024TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representations
  6. 2025Offline Goal-Conditioned Reinforcement Learning with Projective Quasimetric Planning
  7. 2025Test-Time Graph Search for Goal-Conditioned Reinforcement Learning
  8. 2022How to Spend Your Robot Time: Bridging Kickstarting and Offline Reinforcement Learning for Vision-based Robotic Manipulation