# Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making

arXiv: https://arxiv.org/abs/2406.17098 · HTML: https://mlsanity.com/paper/2406.17098/

## Key facts

- Cited by 56 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 55 of its 78 references that are in the map.
- First posted on arXiv 24 Jun 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 30 of 55)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [AWAC: Accelerating Online Reinforcement Learning with Offline Datasets](https://mlsanity.com/paper/2006.09359/) (2020)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [A Theoretical Analysis of Contrastive Unsupervised Representation Learning](https://mlsanity.com/paper/1902.09229/) (2019)
- [VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training](https://mlsanity.com/paper/2210.00030/) (2022)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [Data-Efficient Reinforcement Learning with Self-Predictive Representations](https://mlsanity.com/paper/2007.05929/) (2020)
- [Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Models](https://mlsanity.com/paper/2310.10639/) (2023)
- [Contrastive Learning as Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2206.07568/) (2022)
- [Semi-parametric Topological Memory for Navigation](https://mlsanity.com/paper/1803.00653/) (2018)
- [Language Conditioned Imitation Learning over Unstructured Data](https://mlsanity.com/paper/2005.07648/) (2020)
- [LIV: Language-Image Representations and Rewards for Robotic Control](https://mlsanity.com/paper/2306.00958/) (2023)
- [HIQL: Offline Goal-Conditioned RL with Latent States as Actions](https://mlsanity.com/paper/2307.11949/) (2023)
- [Goal-Conditioned Reinforcement Learning with Imagined Subgoals](https://mlsanity.com/paper/2107.00541/) (2021)
- [Discovering and Achieving Goals via World Models](https://mlsanity.com/paper/2110.09514/) (2021)
- [Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning](https://mlsanity.com/paper/2304.01203/) (2023)
- [Learning One Representation to Optimize All Rewards](https://mlsanity.com/paper/2103.07945/) (2021)

## Built on by (most cited first, 30 of 56)

- [OGBench: Benchmarking Offline Goal-Conditioned RL](https://mlsanity.com/paper/2410.20092/) (2024)
- [1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities](https://mlsanity.com/paper/2503.14858/) (2025)
- [Offline Goal-conditioned Reinforcement Learning with Quasimetric Representations](https://mlsanity.com/paper/2509.20478/) (2025)
- [Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learning](https://mlsanity.com/paper/2412.08021/) (2024)
- [Horizon Generalization in Reinforcement Learning](https://mlsanity.com/paper/2501.02709/) (2025)
- [Episodic Novelty Through Temporal Distance](https://mlsanity.com/paper/2501.15418/) (2025)
- [Dual Goal Representations](https://mlsanity.com/paper/2510.06714/) (2025)
- [Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Following](https://mlsanity.com/paper/2502.05454/) (2025)
- [Normalizing Flows are Capable Models for Continuous Control](https://mlsanity.com/paper/2505.23527/) (2025)
- [Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloning](https://mlsanity.com/paper/2506.10137/) (2025)
- [Transitive RL: Value Learning via Divide and Conquer](https://mlsanity.com/paper/2510.22512/) (2025)
- [Intention-Conditioned Flow Occupancy Models](https://mlsanity.com/paper/2506.08902/) (2025)
- [Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learning](https://mlsanity.com/paper/2511.07730/) (2025)
- [RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance](https://mlsanity.com/paper/2608.09853/) (2026)
- [Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning](https://mlsanity.com/paper/2505.13144/) (2025)
- [Contrastive Representations for Temporal Reasoning](https://mlsanity.com/paper/2508.13113/) (2025)
- [Demystifying the Mechanisms Behind Emergent Exploration in Goal-conditioned RL](https://mlsanity.com/paper/2510.14129/) (2025)
- [Laplacian Representations for Decision-Time Planning](https://mlsanity.com/paper/2602.05031/) (2026)
- [Is Temporal Difference Learning the Gold Standard for Stitching in RL?](https://mlsanity.com/paper/2510.21995/) (2025)
- [CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queries](https://mlsanity.com/paper/2506.00388/) (2025)
- [STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learning](https://mlsanity.com/paper/2509.23802/) (2025)
- [Can We Really Learn One Representation to Optimize All Rewards?](https://mlsanity.com/paper/2602.11399/) (2026)
- [Three Steps at a Time: Learning Representations from Action Sequences in Contrastive RL](https://mlsanity.com/paper/2608.30640/) (2026)
- [Unifying Agent Interaction and World Information for Multi-agent Coordination](https://mlsanity.com/paper/2509.25550/) (2025)
- [VLD: Visual Language Goal Distance for Reinforcement Learning Navigation](https://mlsanity.com/paper/2512.07976/) (2025)
- [Efficient Hierarchical Implicit Flow Q-learning for Offline Goal-conditioned Reinforcement Learning](https://mlsanity.com/paper/2604.08960/) (2026)
- [NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RL](https://mlsanity.com/paper/2607.07855/) (2026)
- [Learning The Minimum Action Distance](https://mlsanity.com/paper/2506.09276/) (2025)
- [Mixed-Initiative Dialog for Human-Robot Collaborative Manipulation](https://mlsanity.com/paper/2508.05535/) (2025)
- [Consistent Zero-Shot Imitation with Contrastive Goal Inference](https://mlsanity.com/paper/2510.17059/) (2025)

## Abstract (arXiv)

Temporal distances lie at the heart of many algorithms for planning, control, and reinforcement learning that involve reaching goals, allowing one to estimate the transit time between two states. However, prior attempts to define such temporal distances in stochastic settings have been stymied by an important limitation: these prior approaches do not satisfy the triangle inequality. This is not merely a definitional concern, but translates to an inability to generalize and find shortest paths. In this paper, we build on prior work in contrastive learning and quasimetrics to show how successor features learned by contrastive learning (after a change of variables) form a temporal distance that does satisfy the triangle inequality, even in stochastic settings. Importantly, this temporal distance is computationally efficient to estimate, even in high-dimensional and stochastic settings. Experiments in controlled settings and benchmark suites demonstrate that an RL algorithm based on these new temporal distances exhibits combinatorial generalization (i.e., "stitching") and can sometimes learn more quickly than prior methods, including those based on quasimetrics.