# Self-supervised Learning of Distance Functions for Goal-Conditioned Reinforcement Learning

arXiv: https://arxiv.org/abs/1907.02998 · HTML: https://mlsanity.com/paper/1907.02998/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jul 2019.

## Built on by (most cited first, 18 of 18)

- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making](https://mlsanity.com/paper/2406.17098/) (2024)
- [Adversarial Intrinsic Motivation for Reinforcement Learning](https://mlsanity.com/paper/2105.13345/) (2021)
- [Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping](https://mlsanity.com/paper/2301.02099/) (2023)
- [Improved Representation of Asymmetrical Distances with Interval Quasimetric Embeddings](https://mlsanity.com/paper/2211.15120/) (2022)
- [Successor Feature Landmarks for Long-Horizon Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2111.09858/) (2021)
- [A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals](https://mlsanity.com/paper/2408.05804/) (2024)
- [Outcome-Driven Reinforcement Learning via Variational Inference](https://mlsanity.com/paper/2104.10190/) (2021)
- [Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learning](https://mlsanity.com/paper/2110.12997/) (2021)
- [Contrastive Representations for Temporal Reasoning](https://mlsanity.com/paper/2508.13113/) (2025)
- [Offline Goal-Conditioned Reinforcement Learning with Projective Quasimetric Planning](https://mlsanity.com/paper/2506.18847/) (2025)
- [Exploiting Unlabeled Data for Feedback Efficient Human Preference based Reinforcement Learning](https://mlsanity.com/paper/2302.08738/) (2023)
- [Walk the Random Walk: Learning to Discover and Reach Goals Without Supervision](https://mlsanity.com/paper/2206.11733/) (2022)
- [Goal-Conditioned Reinforcement Learning from Sub-Optimal Data on Metric Spaces](https://mlsanity.com/paper/2402.10820/) (2024)
- [Automatic Reward Design via Learning Motivation-Consistent Intrinsic Rewards](https://mlsanity.com/paper/2207.14722/) (2022)
- [Autonomous Learning From Success and Failure: Goal-Conditioned Supervised Learning with Negative Feedback](https://mlsanity.com/paper/2509.03206/) (2025)
- [Inter-Agent Relative Representations for Multi-Agent Option Discovery](https://mlsanity.com/paper/2512.24827/) (2025)
- [ChronoSRL: Temporal Geometry for Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/2609.36238/) (2026)

## Abstract (arXiv)

Goal-conditioned policies are used in order to break down complex reinforcement learning (RL) problems by using subgoals, which can be defined either in state space or in a latent feature space. This can increase the efficiency of learning by using a curriculum, and also enables simultaneous learning and generalization across goals. A crucial requirement of goal-conditioned policies is to be able to determine whether the goal has been achieved. Having a notion of distance to a goal is thus a crucial component of this approach. However, it is not straightforward to come up with an appropriate distance, and in some tasks, the goal space may not even be known a priori. In this work we learn a distance-to-goal estimate which is computed in terms of the number of actions that would need to be carried out in a self-supervised approach. Our method solves complex tasks without prior domain knowledge in the online setting in three different scenarios in the context of goal-conditioned policies a) the goal space is the same as the state space b) the goal space is given but an appropriate distance is unknown and c) the state space is accessible, but only a subset of the state space represents desired goals, and this subset is known a priori. We also propose a goal-generation mechanism as a secondary contribution.