mlsanityData to 7 Oct 2026

Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learning

arXiv 2412.08021 ↗First posted 11 Dec 2024cs.LG cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning has the potential of lifting several of the key challenges in reinforcement learning today, such as exploration, representation learning, and reward design. Recent work (METRA) has effectively argued that moving away from mutual information and instead optimizing a certain Wasserstein distance is important for good performance. In this paper, we argue that the benefits seen in that paper can largely be explained within the existing framework of mutual information skill learning (MISL). Our analysis suggests a new MISL method (contrastive successor features) that retains the excellent performance of METRA with fewer moving parts, and highlights connections between skill learning, contrastive representation learning, and successor features. Finally, through careful ablation studies, we provide further insight into some of the key ingredients for both our method and METRA.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 64 of its 79 references that are in the map. First posted on arXiv 11 Dec 2024.

Builds on

64 of 79 references.

References · 64

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  8. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  10. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  11. 2016Generative Adversarial Imitation Learningcited by 1,576
  12. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
38 more
  1. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  2. 2018Exploration by Random Network Distillationcited by 966
  3. 2018DeepMind Control Suitecited by 935
  4. 2016Deep Variational Information Bottleneckcited by 915
  5. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  6. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  7. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  8. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  9. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  10. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  11. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  12. 2019On Variational Bounds of Mutual Informationcited by 550
  13. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  14. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  15. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  16. 2019On Mutual Information Maximization for Representation Learningcited by 315
  17. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  18. 2021How to Train Your Energy-Based Modelscited by 271
  19. 2016Variational Intrinsic Controlcited by 262
  20. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  21. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  22. 2019Efficient Exploration via State Marginal Matchingcited by 186
  23. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  24. 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 142
  25. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 131
  26. 2021Discovering and Achieving Goals via World Modelscited by 131
  27. 2017Stochastic Neural Networks for Hierarchical Reinforcement Learningcited by 124
  28. 2020Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skillscited by 120
  29. 2021APS: Active Pretraining with Successor Featurescited by 109
  30. 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 103
  31. 2018Unsupervised Control Through Non-Parametric Discriminative Rewardscited by 102
  32. 2018Universal Successor Features Approximatorscited by 102
  33. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 99
  34. 2021MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Researchcited by 96
  35. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  36. 2024Craftax: A Lightning-Fast Benchmark for Open-Ended Reinforcement Learningcited by 80
  37. 2016Deep Successor Reinforcement Learningcited by 72
  38. 2019Wasserstein Dependency Measure for Representation Learningcited by 69

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 12.

18 papers.

Sort:

By area:

  1. 2026World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetrycited by 13
  2. 2025Intention-Conditioned Flow Occupancy Modelscited by 7
  3. 2025Contrastive Representations for Temporal Reasoningcited by 5
  4. 2025Skill Learning via Policy Diversity Yields Identifiable Representations for Reinforcement Learningcited by 4
  5. 2025VendiRL: A Framework for Self-Supervised Reinforcement Learning of Diversely Diverse Skillscited by 2
  6. 2026Learning Task-Sufficient World Models by Synergizing Agentic Exploration and Structured Modelingcited by 2
  7. 2025Information-Theoretic Policy Pre-Training with Empowermentcited by 1
  8. 2025Periodic Skill Discoverycited by 1
  9. 2026Can We Really Learn One Representation to Optimize All Rewards?cited by 1
  10. 2026UpSkill: Mutual Information Skill Learning for Structured Response Diversity in LLMscited by 1
  11. 2026Temporal Representations for Exploration: Learning Complex Exploratory Behavior without Extrinsic Rewardscited by 1
  12. 2026Regularized Latent Dynamics Prediction is a Strong Baseline For Behavioral Foundation Modelscited by 1
6 more
  1. 2026Occupancy Reward Shaping: Improving Credit Assignment for Offline Goal-Conditioned Reinforcement Learningcited by 1
  2. 2026COLLIE: Guiding Skill Discovery in Semantically Coherent Latent Spacecited by 1
  3. 2026Multi-Agent Empowerment and Emergence of Complex Behavior in Groupscited by 0
  4. 2026Unifying Goal-Conditioned RL and Unsupervised Skill Learning via Control-Maximizationcited by 0
  5. 2026Learning to Perceive the World Through Control: Empowerment-Based Representation Learningcited by 0
  6. 2026Learning Generalizable Skill Policy with Data-Efficient Unsupervised RLcited by 0