mlsanityData to 14 Oct 2026

TLDR: Unsupervised Goal-Conditioned RL via Temporal Distance-Aware Representations

arXiv 2407.08464 ↗First posted 11 Jul 2024cs.LG cs.AIcited by 17

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Unsupervised goal-conditioned reinforcement learning (GCRL) is a promising paradigm for developing diverse robotic skills without external supervision. However, existing unsupervised GCRL methods often struggle to cover a wide range of states in complex environments due to their limited exploration and sparse or noisy rewards for GCRL. To overcome these challenges, we propose a novel unsupervised GCRL method that leverages TemporaL Distance-aware Representations (TLDR). Based on temporal distance, TLDR selects faraway goals to initiate exploration and computes intrinsic exploration rewards and goal-reaching rewards. Specifically, our exploration policy seeks states with large temporal distances (i.e. covering a large state space), while the goal-conditioned policy learns to minimize the temporal distance to the goal (i.e. reaching the goal). Our results in six simulated locomotion environments demonstrate that TLDR significantly outperforms prior unsupervised GCRL methods in achieving a wide range of states.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 39 of its 47 references that are in the map. First posted on arXiv 11 Jul 2024.

Builds on

39 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Foundation Policies with Hilbert Representations23 Feb

Other references · 38

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2016Layer Normalizationcited by 5,669
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  5. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  7. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  8. 2017Hindsight Experience Replaycited by 1,136
  9. 2018Exploration by Random Network Distillationcited by 993
  10. 2018DeepMind Control Suitecited by 956
  11. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  12. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
26 more
  1. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  2. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  3. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  4. 2020Planning to Explore via Self-Supervised World Modelscited by 368
  5. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 366
  6. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 313
  7. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 274
  8. 2016Variational Intrinsic Controlcited by 267
  9. 2020First return, then explorecited by 266
  10. 2019Self-Supervised Exploration via Disagreementcited by 265
  11. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 174
  12. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 174
  13. 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 153
  14. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 139
  15. 2021Discovering and Achieving Goals via World Modelscited by 131
  16. 2018Variational Option Discovery Algorithmscited by 122
  17. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 121
  18. 2020Learning to be Safe: Deep RL with a Safety Criticcited by 116
  19. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 102
  20. 2022Deep Hierarchical Planning from Pixelscited by 101
  21. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 86
  22. 2019Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discoverycited by 68
  23. 2022Lipschitz-constrained Unsupervised Skill Discoverycited by 62
  24. 2022Efficient Planning in a Compact Latent Action Spacecited by 57
  25. 2023Controllability-Aware Unsupervised Skill Discoverycited by 49
  26. 2023Planning Goals for Explorationcited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 6.

17 papers.

Sort:

By area:

  1. 2025Offline Goal-conditioned Reinforcement Learning with Quasimetric Representationscited by 24
  2. 2026Predictive but Not Plannable: RC-aux for Latent World Modelscited by 19
  3. 2025Episodic Novelty Through Temporal Distancecited by 16
  4. 2025State-Covering Trajectory Stitching for Diffusion Plannerscited by 13
  5. 2025Efficient Skill Discovery via Regret-Aware Optimizationcited by 3
  6. 2025Imagine, Verify, Execute: Memory-guided Agentic Exploration with Vision-Language Modelscited by 2
  7. 2025Unifying Agent Interaction and World Information for Multi-agent Coordinationcited by 2
  8. 2026Action from Adjacent Set in Physical Space Outperforms the Best Prediction in World Modelscited by 2
  9. 2025When a Robot is More Capable than a Human: Learning from Constrained Demonstratorscited by 1
  10. 2025Periodic Skill Discoverycited by 1
  11. 2025ASTRO: Adaptive Stitching via Dynamics-Guided Trajectory Rolloutscited by 1
  12. 2025Autonomous Learning From Success and Failure: Goal-Conditioned Supervised Learning with Negative Feedbackcited by 0
5 more
  1. 2025Inter-Agent Relative Representations for Multi-Agent Option Discoverycited by 0
  2. 2026Search Inspired Exploration in Reinforcement Learningcited by 0
  3. 2026Functional Graphs for Predicting and Explaining Goal Failure in Sparse Goal-Conditioned RLcited by 0
  4. 2026ChronoForest: Closed-Loop Multi-Tree Diffusion Planning for Efficient Bridge Search and Route Compositioncited by 0
  5. 2026ChronoSRL: Temporal Geometry for Self-Supervised Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Dual Goal Representations
  2. 2025Test-Time Graph Search for Goal-Conditioned Reinforcement Learning
  3. 2026Do Better Goal Representations Improve Goal-Conditioned Reinforcement Learning?
  4. 2026Unifying Goal-Conditioned RL and Unsupervised Skill Learning via Control-Maximization
  5. 2026120 Minutes and a Laptop: Minimalist Image-goal Navigation via Unsupervised Exploration and Offline RL
  6. 2025VLD: Visual Language Goal Distance for Reinforcement Learning Navigation
  7. 2024Stitching Sub-Trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL
  8. 2025Flattening Hierarchies with Policy Bootstrapping