mlsanityData to 7 Oct 2026

A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals

arXiv 2408.05804 ↗First posted 11 Aug 2024cs.LG cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

In this paper, we present empirical evidence of skills and directed exploration emerging from a simple RL algorithm long before any successful trials are observed. For example, in a manipulation task, the agent is given a single observation of the goal state and learns skills, first for moving its end-effector, then for pushing the block, and finally for picking up and placing the block. These skills emerge before the agent has ever successfully placed the block at the goal location and without the aid of any reward functions, demonstrations, or manually-specified distance metrics. Once the agent has learned to reach the goal state reliably, exploration is reduced. Implementing our method involves a simple modification of prior work and does not require density estimates, ensembles, or any additional hyperparameters. Intuitively, the proposed method seems like it should be terrible at exploration, and we lack a clear theoretical understanding of why it works so effectively, though our experiments provide some hints.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 64 of its 84 references that are in the map. First posted on arXiv 11 Aug 2024.

Builds on

64 of 84 references.

References · 64

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2016Generative Adversarial Imitation Learningcited by 1,576
  7. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  9. 2017Hindsight Experience Replaycited by 1,096
  10. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  11. 2016Deep Exploration via Bootstrapped DQNcited by 518
  12. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
38 more
  1. 2019Learning Latent Plans from Playcited by 310
  2. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  3. 2017Noisy Networks for Explorationcited by 262
  4. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  5. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  6. 2017Inverse Reward Designcited by 246
  7. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  8. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
  9. 2018Semi-parametric Topological Memory for Navigationcited by 203
  10. 2017Teacher-Student Curriculum Learningcited by 197
  11. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 188
  12. 2017Reverse Curriculum Generation for Reinforcement Learningcited by 185
  13. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  14. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  15. 2019Goal-conditioned Imitation Learningcited by 159
  16. 2017Parameter Space Noise for Explorationcited by 158
  17. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
  18. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  19. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  20. 2019Planning with Goal-Conditioned Policiescited by 144
  21. 2017Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Playcited by 139
  22. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
  23. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 118
  24. 2014Generalization and Exploration via Randomized Value Functionscited by 112
  25. 2018Count-Based Exploration with the Successor Representationcited by 112
  26. 2017Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agentscited by 107
  27. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 101
  28. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 99
  29. 2020Learning with AMIGo: Adversarially Motivated Intrinsic Goalscited by 86
  30. 2018The Laplacian in RL: Learning Representations with Efficient Approximationscited by 83
  31. 2017The Uncertainty Bellman Equation and Explorationcited by 81
  32. 2019Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewardscited by 76
  33. 2021Asymmetric self-play for automatic goal discovery in robotic manipulationcited by 71
  34. 2019Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discoverycited by 65
  35. 2019Learning Predictive Models From Observation and Interactioncited by 41
  36. 2020Model-Based Visual Planning with Self-Supervised Functional Distancescited by 41
  37. 2022Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learningcited by 38
  38. 2016Learning to Act by Predicting the Futurecited by 36

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 13.

20 papers.

Sort:

By area:

  1. 2026Reinforcement Learning via Self-Distillationcited by 285
  2. 2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learningcited by 22
  3. 2026World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetrycited by 13
  4. 2025Deep RL Needs Deep Behavior Analysis: Exploring Implicit Planning by Model-Free Agents in Open-Ended Environmentscited by 4
  5. 2025Skill Learning via Policy Diversity Yields Identifiable Representations for Reinforcement Learningcited by 4
  6. 2025Efficient Skill Discovery via Regret-Aware Optimizationcited by 3
  7. 2025Self-Supervised Goal-Reaching Results in Multi-Agent Cooperation and Explorationcited by 3
  8. 2025Demystifying the Mechanisms Behind Emergent Exploration in Goal-conditioned RLcited by 3
  9. 2025What does really matter in image goal navigation?cited by 2
  10. 2026Efficient Hierarchical Implicit Flow Q-learning for Offline Goal-conditioned Reinforcement Learningcited by 2
  11. 2026Action-Sufficient Goal Representationscited by 1
  12. 2026Temporal Representations for Exploration: Learning Complex Exploratory Behavior without Extrinsic Rewardscited by 1
8 more
  1. 2026CoRMA: Contrastive RMA for Contact-Rich Meta-Adaptationcited by 1
  2. 2026Inverting the Bellman Equation: From Q-Values to World Modelscited by 1
  3. 2026Three Steps at a Time: Learning Representations from Action Sequences in Contrastive RLcited by 1
  4. 2026Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search for Hard Explorationcited by 0
  5. 2026Self-Supervised On-Policy Reinforcement Learning via Contrastive Proximal Policy Optimisationcited by 0
  6. 2026Distributionally Robust Multi-Task Reinforcement Learning via Adaptive Task Samplingcited by 0
  7. 2026Direction-Conditioned Policies via Compositional Subgoal Scoring for Online Goal-Conditioned Reinforcement Learningcited by 0
  8. 2026Observation-Grounded Self-Predictive Reinforcement Learning for Visual Continuous Controlcited by 0