mlsanityData to 7 Oct 2026

Unifying Goal-Conditioned RL and Unsupervised Skill Learning via Control-Maximization

arXiv 2605.06145 ↗First posted 7 May 2026cs.LG cs.AI cs.SYcited by 0

Not on the map. Why?

Abstract (arXiv)

Unsupervised pretraining has driven empirical advances in goal-conditioned reinforcement learning (GCRL), but its theoretical foundations remain poorly understood. In particular, an influential class of methods, mutual information skill learning (MISL), discovers behaviorally diverse skills that can later be used for downstream goal-reaching. However, it remains a theoretical mystery why skills learned through MISL should support goal-reaching. A subtle challenge is that both GCRL and MISL are umbrella terms: different GCRL tasks use distinct criteria for measuring goal-reaching performance, while different MISL methods optimize distinct notions of behavioral diversity. We address this challenge and unify GCRL and MISL as instances of control maximization. We identify three canonical GCRL formulations and prove that they are fundamentally inequivalent: they can induce incompatible optimal policies even in the same environment. Nevertheless, they all share a common interpretation: a well-performing goal-conditioned policy is one whose future trajectory is highly sensitive to the commanded goal, with the precise notion of sensitivity determined by the GCRL formulation. Noting that MISL objectives can be understood as measures of skill-sensitivity akin to goal-sensitivity, we show that MISL objectives are bounded by formulation-specific downstream goal-sensitivities. These bounds establish a precise correspondence between MISL methods and downstream GCRL tasks: for every GCRL formulation, there exists a matching MISL objective for which more diverse skills afford greater downstream goal sensitivity. Our results thus lay a theoretical foundation for RL pretraining and have important practical implications, such as suggesting which pretraining objectives to use when a user cares about a specific class of downstream tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 44 of its 71 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

44 of 71 references.

References · 44

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  6. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  7. 2017Hindsight Experience Replaycited by 1,096
  8. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  9. 2019On Variational Bounds of Mutual Informationcited by 550
  10. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  11. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  12. 2016Variational Intrinsic Controlcited by 262
32 more
  1. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  2. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  3. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  4. 2017Hamilton-Jacobi Reachability: A Brief Overview and Recent Advancescited by 186
  5. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  6. 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 142
  7. 2021Discovering and Achieving Goals via World Modelscited by 131
  8. 2018Variational Option Discovery Algorithmscited by 121
  9. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
  10. 2018Unsupervised Control Through Non-Parametric Discriminative Rewardscited by 102
  11. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 99
  12. 2018Temporal Difference Models: Model-Free Deep RL for Model-Based Controlcited by 89
  13. 2020DeepReach: A Deep Learning Approach to High-Dimensional Reachabilitycited by 81
  14. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  15. 2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 66
  16. 2013Empowerment -- an Introductioncited by 65
  17. 2022Lipschitz-constrained Unsupervised Skill Discoverycited by 60
  18. 2019What Can Learned Intrinsic Rewards Capture?cited by 36
  19. 2021The Information Geometry of Unsupervised Reinforcement Learningcited by 36
  20. 2022INFOrmation Prioritization through EmPOWERment in Visual Model-Based RLcited by 27
  21. 2018Many-Goals Reinforcement Learningcited by 22
  22. 2024Accelerating Goal-Conditioned RL Algorithms and Researchcited by 18
  23. 2024Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learningcited by 18
  24. 2019A Unified Bellman Optimality Principle Combining Reward Maximization and Empowermentcited by 15
  25. 2023f-Policy Gradients: A General Framework for Goal Conditioned RL using f-Divergencescited by 14
  26. 2025Dual Goal Representationscited by 13
  27. 2024Learning to Assist Humans without Inferring Rewardscited by 12
  28. 2025Plasticity as the Mirror of Empowermentcited by 11
  29. 2023Hierarchical Empowerment: Towards Tractable Empowerment-Based Skill Learningcited by 7
  30. 2025Skill Learning via Policy Diversity Yields Identifiable Representations for Reinforcement Learningcited by 4
  31. 2019Model-Free Stochastic Reachability Using Kernel Distribution Embeddingscited by 3
  32. 2024Reachability Analysis for Black-Box Dynamical Systemscited by 3

Built on by

Not cited yet.