mlsanityData to 7 Oct 2026

Demystifying the Mechanisms Behind Emergent Exploration in Goal-conditioned RL

arXiv 2510.14129 ↗First posted 15 Oct 2025cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

In this work, we take a first step toward elucidating the mechanisms behind emergent exploration in unsupervised reinforcement learning. We study Single-Goal Contrastive Reinforcement Learning (SGCRL), a self-supervised algorithm capable of solving challenging long-horizon goal-reaching tasks without external rewards or curricula. We combine theoretical analysis of the algorithm's objective function with controlled experiments to understand what drives its exploration. We show that SGCRL maximizes implicit rewards shaped by its learned representations. These representations automatically modify the reward landscape to promote exploration before reaching the goal and exploitation thereafter. Our experiments also demonstrate that these exploration dynamics arise from learning low-rank representations of the state space rather than from neural network function approximation. Our improved understanding enables us to adapt SGCRL to perform safety-aware exploration.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 42 of its 84 references that are in the map. First posted on arXiv 15 Oct 2025.

Builds on

42 of 84 references.

References · 42

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  3. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  4. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  5. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  6. 2017Hindsight Experience Replaycited by 1,096
  7. 2018Exploration by Random Network Distillationcited by 966
  8. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  9. 2018Is Q-learning Provably Efficient?cited by 485
  10. 2022Using cognitive psychology to understand GPT-3cited by 389
  11. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  12. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
30 more
  1. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  2. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
  3. 2019Planning with Goal-Conditioned Policiescited by 144
  4. 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 142
  5. 2017Visualizing and Understanding Atari Agentscited by 141
  6. 2020RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environmentscited by 135
  7. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 129
  8. 2018State Representation Learning for Control: An Overviewcited by 125
  9. 2014Model-based Reinforcement Learning and the Eluder Dimensioncited by 116
  10. 2016Why is Posterior Sampling Better than Optimism for Reinforcement Learning?cited by 116
  11. 2020Explainability in Deep Reinforcement Learningcited by 96
  12. 2020PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learningcited by 91
  13. 2021A Survey on Interpretable Reinforcement Learningcited by 70
  14. 2016Graying the black box: Understanding DQNscited by 68
  15. 2021Learning Successor States and Goal-Dependent Values: A Mathematical Viewpointcited by 67
  16. 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 53
  17. 2019Information-Theoretic Confidence Bounds for Reinforcement Learningcited by 51
  18. 2022Exploration via Elliptical Episodic Bonusescited by 51
  19. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  20. 2024Comprehensive Overview of Reward Engineering and Shaping in Advancing Reinforcement Learning Applicationscited by 34
  21. 2019Dot-to-Dot: Explainable Hierarchical Reinforcement Learning for Robotic Manipulationcited by 33
  22. 2019Decoupling feature extraction from policy learning: assessing benefits of state representation learning in goal based roboticscited by 21
  23. 2024A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoalscited by 20
  24. 2023Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Datacited by 19
  25. 2020TripleTree: A Versatile Interpretable Representation of Black Box Agents and their Environmentscited by 18
  26. 2025Interpreting Emergent Planning in Model-Free Reinforcement Learningcited by 15
  27. 2014Bayesian Optimal Control of Smoothly Parameterized Systems: The Lazy Posterior Sampling Algorithmcited by 11
  28. 2020Levels of Analysis for Machine Learningcited by 11
  29. 2025Deep RL Needs Deep Behavior Analysis: Exploring Implicit Planning by Model-Free Agents in Open-Ended Environmentscited by 4
  30. 2025Self-Supervised Goal-Reaching Results in Multi-Agent Cooperation and Explorationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Three Steps at a Time: Learning Representations from Action Sequences in Contrastive RLcited by 1
  2. 2026Self-Supervised On-Policy Reinforcement Learning via Contrastive Proximal Policy Optimisationcited by 0
  3. 2026Arrive and Survive: Scaling Safe Goal-Conditioned Policy Learning from One-Bit Failure Signalscited by 0