mlsanityData to 7 Oct 2026

Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making

arXiv 2406.17098 ↗First posted 24 Jun 2024cs.LG cs.AIcited by 53

Not on the map. Why?

Abstract (arXiv)

Temporal distances lie at the heart of many algorithms for planning, control, and reinforcement learning that involve reaching goals, allowing one to estimate the transit time between two states. However, prior attempts to define such temporal distances in stochastic settings have been stymied by an important limitation: these prior approaches do not satisfy the triangle inequality. This is not merely a definitional concern, but translates to an inability to generalize and find shortest paths. In this paper, we build on prior work in contrastive learning and quasimetrics to show how successor features learned by contrastive learning (after a change of variables) form a temporal distance that does satisfy the triangle inequality, even in stochastic settings. Importantly, this temporal distance is computationally efficient to estimate, even in high-dimensional and stochastic settings. Experiments in controlled settings and benchmark suites demonstrate that an RL algorithm based on these new temporal distances exhibits combinatorial generalization (i.e., "stitching") and can sometimes learn more quickly than prior methods, including those based on quasimetrics.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Builds on 55 of its 78 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

55 of 78 references.

References · 55

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  5. 2015Continuous control with deep reinforcement learningcited by 3,279
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  9. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  10. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  11. 2017Hindsight Experience Replaycited by 1,096
  12. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
38 more
  1. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  2. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  3. 2019On Variational Bounds of Mutual Informationcited by 550
  4. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  5. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  6. 2020Spatiotemporal Contrastive Video Representation Learningcited by 372
  7. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  8. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  9. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 264
  10. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  11. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  12. 2018Semi-parametric Topological Memory for Navigationcited by 203
  13. 2023LIV: Language-Image Representations and Rewards for Robotic Controlcited by 180
  14. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  15. 2021Discovering and Achieving Goals via World Modelscited by 131
  16. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
  17. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
  18. 2021Learning One Representation to Optimize All Rewardscited by 110
  19. 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 103
  20. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 99
  21. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  22. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  23. 2020Deep Reinforcement and InfoMax Learningcited by 79
  24. 2023MUTEX: Learning Unified Policies from Multimodal Task Specificationscited by 70
  25. 2019Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discoverycited by 65
  26. 2023Reinforcement Learning from Passive Data via Latent Intentionscited by 64
  27. 2018Learning Actionable Representations with Goal-Conditioned Policiescited by 53
  28. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 52
  29. 2021Adversarial Intrinsic Motivation for Reinforcement Learningcited by 40
  30. 2022Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learningcited by 38
  31. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
  32. 2023Contrastive Difference Predictive Codingcited by 35
  33. 2022Generalizing Goal-Conditioned Reinforcement Learning with Variational Causal Reasoningcited by 33
  34. 2023Goal Representations for Instruction Following: A Semi-Supervised Language Interface to Controlcited by 32
  35. 2021Metrics and continuity in reinforcement learningcited by 26
  36. 2024Closing the Gap between TD Learning and Supervised Learning -- A Generalisation Point of Viewcited by 26
  37. 2022Generalization with Lossy Affordances: Leveraging Broad Offline Data for Learning Visuomotor Taskscited by 24
  38. 2019Self-supervised Learning of Distance Functions for Goal-Conditioned Reinforcement Learningcited by 18

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 24; 2026: 27.

53 papers.

Sort:

By area:

  1. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  2. 20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilitiescited by 39
  3. 2025Offline Goal-conditioned Reinforcement Learning with Quasimetric Representationscited by 20
  4. 2024Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learningcited by 18
  5. 2025Horizon Generalization in Reinforcement Learningcited by 15
  6. 2025Episodic Novelty Through Temporal Distancecited by 15
  7. 2025Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Followingcited by 13
  8. 2025Normalizing Flows are Capable Models for Continuous Controlcited by 13
  9. 2025Dual Goal Representationscited by 13
  10. 2025Transitive RL: Value Learning via Divide and Conquercited by 12
  11. 2025Self-Predictive Representations for Combinatorial Generalization in Behavioral Cloningcited by 9
  12. 2025Intention-Conditioned Flow Occupancy Modelscited by 7
38 more
  1. 2025Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learningcited by 6
  2. 2025Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learningcited by 5
  3. 2025Contrastive Representations for Temporal Reasoningcited by 5
  4. 2026RynnValue: Scaling Robotic Value Foundation Models with Temporal Distancecited by 5
  5. 2026Laplacian Representations for Decision-Time Planningcited by 4
  6. 2025CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous Queriescited by 3
  7. 2025STAIR: Addressing Stage Misalignment through Temporal-Aligned Preference Reinforcement Learningcited by 3
  8. 2025Demystifying the Mechanisms Behind Emergent Exploration in Goal-conditioned RLcited by 3
  9. 2025Is Temporal Difference Learning the Gold Standard for Stitching in RL?cited by 3
  10. 2025VLD: Visual Language Goal Distance for Reinforcement Learning Navigationcited by 2
  11. 2026Efficient Hierarchical Implicit Flow Q-learning for Offline Goal-conditioned Reinforcement Learningcited by 2
  12. 2025Mixed-Initiative Dialog for Human-Robot Collaborative Manipulationcited by 1
  13. 2025Unifying Agent Interaction and World Information for Multi-agent Coordinationcited by 1
  14. 2025Consistent Zero-Shot Imitation with Contrastive Goal Inferencecited by 1
  15. 2025Periodic Skill Discoverycited by 1
  16. 2026Can We Really Learn One Representation to Optimize All Rewards?cited by 1
  17. 2026Hitting Time Isomorphism for Multi-Stage Planning with Foundation Policiescited by 1
  18. 2026Compositional Transduction with Latent Analogies for Offline Goal-Conditioned Reinforcement Learningcited by 1
  19. 2026Latent Representation Alignment for Offline Goal-Conditioned Reinforcement Learningcited by 1
  20. 2026NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RLcited by 1
  21. 2026Three Steps at a Time: Learning Representations from Action Sequences in Contrastive RLcited by 1
  22. 2026Recursive Value Learning for Long-Horizon Offline Goal-Conditioned RLcited by 1
  23. 2025Learning The Minimum Action Distancecited by 0
  24. 2025Inter-Agent Relative Representations for Multi-Agent Option Discoverycited by 0
  25. 2026Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Explorationcited by 0
  26. 2026Full-Gradient Successor Feature Representationscited by 0
  27. 2026QHyer: Q-conditioned Hybrid Attention-mamba Transformer for Offline Goal-conditioned RLcited by 0
  28. 2026Quality-Aware Exploration Budget Allocation for Cooperative Multi-Agent Reinforcement Learningcited by 0
  29. 2026Switching Successor Measures for Hierarchical Zero-shot Reinforcement Learningcited by 0
  30. 2026Dual Advantage Fieldscited by 0
  31. 2026ChronoForest: Closed-Loop Multi-Tree Diffusion Planning for Efficient Bridge Search and Route Compositioncited by 0
  32. 2026Direction-Conditioned Policies via Compositional Subgoal Scoring for Online Goal-Conditioned Reinforcement Learningcited by 0
  33. 2026Learning Generalizable Skill Policy with Data-Efficient Unsupervised RLcited by 0
  34. 2026Good Rankers, Bad Objectives: Bilinear Contrastive Critics under Expressive Policy Searchcited by 0
  35. 2026Enhancing VLM Reward Models Through Structure-Aware Fine-Tuningcited by 0
  36. 2026PathBridger: Subgoal Bridges for Offline Goal-Conditioned Reinforcement Learningcited by 0
  37. 2026SUN: Reaching for Novelty in Reinforcement Learningcited by 0
  38. 2026GTRL: Grounding Divide-and-Conquer Value Learning with Temporal Differencescited by 0

50 most cited of 53 All 53 (JSON)