# Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discovery

arXiv: https://arxiv.org/abs/1907.08225 · HTML: https://mlsanity.com/paper/1907.08225/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Jul 2019.

## Built on by (most cited first, 30 of 68)

- [LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action](https://mlsanity.com/paper/2207.04429/) (2022)
- [ViNT: A Foundation Model for Visual Navigation](https://mlsanity.com/paper/2306.14846/) (2023)
- [Goal-Conditioned Reinforcement Learning: Problems and Solutions](https://mlsanity.com/paper/2201.08299/) (2022)
- [Discovering and Achieving Goals via World Models](https://mlsanity.com/paper/2110.09514/) (2021)
- [Deep Hierarchical Planning from Pixels](https://mlsanity.com/paper/2206.04114/) (2022)
- [METRA: Scalable Unsupervised RL with Metric-Aware Abstraction](https://mlsanity.com/paper/2310.08887/) (2023)
- [BYOL-Explore: Exploration by Bootstrapped Prediction](https://mlsanity.com/paper/2206.08332/) (2022)
- [Winner Takes It All: Training Performant RL Populations for Combinatorial Optimization](https://mlsanity.com/paper/2210.03475/) (2022)
- [Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making](https://mlsanity.com/paper/2406.17098/) (2024)
- [Adversarial Intrinsic Motivation for Reinforcement Learning](https://mlsanity.com/paper/2105.13345/) (2021)
- [Decomposing the Enigma: Subgoal-based Demonstration Learning for Formal Theorem Proving](https://mlsanity.com/paper/2305.16366/) (2023)
- [Rank2Reward: Learning Shaped Reward Functions from Passive Video](https://mlsanity.com/paper/2404.14735/) (2024)
- [Self-Improving Embodied Foundation Models](https://mlsanity.com/paper/2509.15155/) (2025)
- [Planning Goals for Exploration](https://mlsanity.com/paper/2303.13002/) (2023)
- [Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shaping](https://mlsanity.com/paper/2301.02099/) (2023)
- [Reward Models in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2506.15421/) (2025)
- [ALAN: Autonomously Exploring Robotic Agents in the Real World](https://mlsanity.com/paper/2302.06604/) (2023)
- [Offline Goal-conditioned Reinforcement Learning with Quasimetric Representations](https://mlsanity.com/paper/2509.20478/) (2025)
- [Continuously Discovering Novel Strategies via Reward-Switching Policy Optimization](https://mlsanity.com/paper/2204.02246/) (2022)
- [A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals](https://mlsanity.com/paper/2408.05804/) (2024)
- [Direct then Diffuse: Incremental Unsupervised Skill Discovery for State Covering and Goal Reaching](https://mlsanity.com/paper/2110.14457/) (2021)
- [Distance Weighted Supervised Learning for Offline Interaction Data](https://mlsanity.com/paper/2304.13774/) (2023)
- [Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learning](https://mlsanity.com/paper/2104.05043/) (2021)
- [Learning Reward Functions for Robotic Manipulation by Observing Humans](https://mlsanity.com/paper/2211.09019/) (2022)
- [Understanding the Complexity Gains of Single-Task RL with a Curriculum](https://mlsanity.com/paper/2212.12809/) (2022)
- [TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representations](https://mlsanity.com/paper/2407.08464/) (2024)
- [Episodic Novelty Through Temporal Distance](https://mlsanity.com/paper/2501.15418/) (2025)
- [Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learning](https://mlsanity.com/paper/2110.12997/) (2021)
- [Outcome-directed Reinforcement Learning by Uncertainty & Temporal Distance-Aware Curriculum Goal Generation](https://mlsanity.com/paper/2301.11741/) (2023)
- [MapGo: Model-Assisted Policy Optimization for Goal-Oriented Tasks](https://mlsanity.com/paper/2105.06350/) (2021)

## Abstract (arXiv)

Reinforcement learning requires manual specification of a reward function to learn a task. While in principle this reward function only needs to specify the task goal, in practice reinforcement learning can be very time-consuming or even infeasible unless the reward function is shaped so as to provide a smooth gradient towards a successful outcome. This shaping is difficult to specify by hand, particularly when the task is learned from raw observations, such as images. In this paper, we study how we can automatically learn dynamical distances: a measure of the expected number of time steps to reach a given goal state from any other state. These dynamical distances can be used to provide well-shaped reward functions for reaching new goals, making it possible to learn complex tasks efficiently. We show that dynamical distances can be used in a semi-supervised regime, where unsupervised interaction with the environment is used to learn the dynamical distances, while a small amount of preference supervision is used to determine the task goal, without any manually engineered reward function or goal examples. We evaluate our method both on a real-world robot and in simulation. We show that our method can learn to turn a valve with a real-world 9-DoF hand, using raw image observations and just ten preference labels, without any other supervision. Videos of the learned skills can be found on the project website: https://sites.google.com/view/dynamical-distance-learning.