mlsanityData to 14 Oct 2026

Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning

arXiv 2304.01203 ↗First posted 3 Apr 2023cs.LGcited by 121

Not on the map. Why?

Cited by 59 in Deep reinforcement learning methods, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In goal-reaching reinforcement learning (RL), the optimal value function has a particular geometry, called quasimetric structure. This paper introduces Quasimetric Reinforcement Learning (QRL), a new RL method that utilizes quasimetric models to learn optimal value functions. Distinct from prior approaches, the QRL objective is specifically designed for quasimetrics, and provides strong theoretical recovery guarantees. Empirically, we conduct thorough analyses on a discretized MountainCar environment, identifying properties of QRL and its advantages over alternatives. On offline and online goal-reaching benchmarks, QRL also demonstrates improved sample efficiency and performance, across both state-based and image-based observations.

Key facts. Cited by 121 papers in the map (data through 14 Oct 2026). Builds on 45 of its 64 references that are in the map. First posted on arXiv 3 Apr 2023.

Builds on

45 of 64 references.

References · 45

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  6. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  9. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  10. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  11. 2017Hindsight Experience Replaycited by 1,136
  12. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
33 more
  1. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  2. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  3. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  4. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  5. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  6. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 260
  7. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  8. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 185
  9. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 136
  10. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  11. 2015Model Predictive Path Integral Control using Covariance Variable Importance Samplingcited by 124
  12. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 103
  13. 2019Reward-Conditioned Policiescited by 102
  14. 2022Why So Pessimistic? Estimating Uncertainties for Offline RL through Ensembles, and Why Their Independence Matterscited by 76
  15. 2022Denoised MDPs: Learning World Models Better Than the World Itselfcited by 68
  16. 2022You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environmentscited by 63
  17. 2020Generating Adjacency-Constrained Subgoals in Hierarchical Reinforcement Learningcited by 58
  18. 2021Learning Task Informed Abstractionscited by 58
  19. 2020Predictive Information Accelerates Learning in RLcited by 56
  20. 2019A Geometric Perspective on Optimal Representations for Reinforcement Learningcited by 54
  21. 2022Dichotomy of Control: Separating What You Can Control from What You Cannotcited by 47
  22. 2020Model-Based Visual Planning with Self-Supervised Functional Distancescited by 42
  23. 2021Adversarial Intrinsic Motivation for Reinforcement Learningcited by 40
  24. 2021Instabilities of Offline RL with Pre-Trained Neural Representationcited by 38
  25. 2022Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Errorcited by 32
  26. 2021Robust Predictable Controlcited by 29
  27. 2020An Inductive Bias for Distances: Neural Nets that Respect the Triangle Inequalitycited by 24
  28. 2022Improved Representation of Asymmetrical Distances with Interval Quasimetric Embeddingscited by 24
  29. 2020Plan2Vec: Unsupervised Representation Learning by Latent Planscited by 21
  30. 2022On the Learning and Learnability of Quasimetricscited by 15
  31. 2022Learning Dynamics and Generalization in Reinforcement Learningcited by 14
  32. 2022Metric Residual Networks for Sample Efficient Goal-Conditioned Reinforcement Learningcited by 13
  33. 2020Multi-task Reinforcement Learning with a Planning Quasi-Metriccited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 7; 2024: 19; 2025: 39; 2026: 56.

121 papers.

Sort:

By area:

  1. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  2. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 141
  3. 2024Foundation Policies with Hilbert Representationscited by 94
  4. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 89
  5. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 86
  6. 2025Horizon Reduction Makes RL Scalablecited by 57
  7. 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 56
  8. 20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilitiescited by 45
  9. 2025Generative Trajectory Stitching through Diffusion Compositioncited by 41
  10. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  11. 2026Temporal Straightening for Latent Planningcited by 38
  12. 2023Contrastive Difference Predictive Codingcited by 36
38 more
  1. 2024Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodingscited by 31
  2. 2025Offline Goal-conditioned Reinforcement Learning with Quasimetric Representationscited by 24
  3. 2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learningcited by 23
  4. 2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
  5. 2025Value-guided action planning with JEPA world modelscited by 21
  6. 2026Predictive but Not Plannable: RC-aux for Latent World Modelscited by 19
  7. 2024Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inferencecited by 17
  8. 2024TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representationscited by 17
  9. 2025Horizon Generalization in Reinforcement Learningcited by 17
  10. 2023Structure in Deep Reinforcement Learning: A Survey and Open Problemscited by 16
  11. 2025Episodic Novelty Through Temporal Distancecited by 16
  12. 2025Dual Goal Representationscited by 16
  13. 2025Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learningcited by 15
  14. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modelingcited by 15
  15. 2025Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Followingcited by 14
  16. 2025Normalizing Flows are Capable Models for Continuous Controlcited by 14
  17. 2025Flow-Based Single-Step Completion for Efficient and Expressive Policy Learningcited by 14
  18. 2025State-Covering Trajectory Stitching for Diffusion Plannerscited by 13
  19. 2025DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learningcited by 12
  20. 2025Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learningcited by 12
  21. 2025Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloningcited by 12
  22. 2025Transitive RL: Value Learning via Divide and Conquercited by 12
  23. 2024Unsupervised-to-Online Reinforcement Learningcited by 11
  24. 2025Flattening Hierarchies with Policy Bootstrappingcited by 11
  25. 2026Conditional Diffusion Guidance under Hard Constraint: A Stochastic Analysis Approachcited by 11
  26. 2024Align Your Intents: Offline Imitation Learning via Optimal Transportcited by 8
  27. 2024Do's and Don'ts: Learning Desirable Skills with Instruction Videoscited by 8
  28. 2025Test-time Offline Reinforcement Learning on Goal-related Experiencecited by 8
  29. 2025Test-Time Graph Search for Goal-Conditioned Reinforcement Learningcited by 8
  30. 2025Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learningcited by 8
  31. 2026Temporal-Distance JEPA: Plan-Aware Representation Learning for Latent World Model Predictive Controlcited by 8
  32. 2026Hierarchical Entity-centric Reinforcement Learning with Factored Subgoal Diffusioncited by 7
  33. 2026RynnValue: Scaling Robotic Value Foundation Models with Temporal Distancecited by 7
  34. 2023Value Functions are Control Barrier Functions: Verification of Safe Policies using Control Theorycited by 6
  35. 2025Zero-Shot Trajectory Planning for Signal Temporal Logic Taskscited by 6
  36. 2025Physics-informed Temporal Difference Metric Learning for Robot Motion Planningcited by 6
  37. 2025Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learningcited by 6
  38. 2025Expressive Value Learning for Scalable Offline Reinforcement Learningcited by 6

50 most cited of 121 All 121 (JSON)