# Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learning

arXiv: https://arxiv.org/abs/2412.08021 · HTML: https://mlsanity.com/paper/2412.08021/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 64 of its 79 references that are in the map.
- First posted on arXiv 11 Dec 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 30 of 64)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [BridgeData V2: A Dataset for Robot Learning at Scale](https://mlsanity.com/paper/2308.12952/) (2023)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020)
- [Gradient Descent Finds Global Minima of Deep Neural Networks](https://mlsanity.com/paper/1811.03804/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [VIMA: General Robot Manipulation with Multimodal Prompts](https://mlsanity.com/paper/2210.03094/) (2022)
- [VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training](https://mlsanity.com/paper/2210.00030/) (2022)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [On Mutual Information Maximization for Representation Learning](https://mlsanity.com/paper/1907.13625/) (2019)
- [Dynamics-Aware Unsupervised Discovery of Skills](https://mlsanity.com/paper/1907.01657/) (2019)
- [How to Train Your Energy-Based Models](https://mlsanity.com/paper/2101.03288/) (2021)

## Built on by (most cited first, 21 of 21)

- [World Action Verifier: Self-Improving World Models via Forward-Inverse Asymmetry](https://mlsanity.com/paper/2604.01985/) (2026)
- [Intention-Conditioned Flow Occupancy Models](https://mlsanity.com/paper/2506.08902/) (2025)
- [Contrastive Representations for Temporal Reasoning](https://mlsanity.com/paper/2508.13113/) (2025)
- [Skill Learning via Policy Diversity Yields Identifiable Representations for Reinforcement Learning](https://mlsanity.com/paper/2507.14748/) (2025)
- [Can We Really Learn One Representation to Optimize All Rewards?](https://mlsanity.com/paper/2602.11399/) (2026)
- [VendiRL: A Framework for Self-Supervised Reinforcement Learning of Diversely Diverse Skills](https://mlsanity.com/paper/2509.02930/) (2025)
- [Information-Theoretic Policy Pre-Training with Empowerment](https://mlsanity.com/paper/2510.05996/) (2025)
- [Temporal Representations for Exploration: Learning Complex Exploratory Behavior without Extrinsic Rewards](https://mlsanity.com/paper/2603.02008/) (2026)
- [Regularized Latent Dynamics Prediction is a Strong Baseline For Behavioral Foundation Models](https://mlsanity.com/paper/2603.15857/) (2026)
- [Unifying Goal-Conditioned RL and Unsupervised Skill Learning via Control-Maximization](https://mlsanity.com/paper/2605.06145/) (2026)
- [Learning Task-Sufficient World Models by Synergizing Agentic Exploration and Structured Modeling](https://mlsanity.com/paper/2607.04409/) (2026)
- [Periodic Skill Discovery](https://mlsanity.com/paper/2511.03187/) (2025)
- [UpSkill: Mutual Information Skill Learning for Structured Response Diversity in LLMs](https://mlsanity.com/paper/2602.22296/) (2026)
- [Occupancy Reward Shaping: Improving Credit Assignment for Offline Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2604.20627/) (2026)
- [Learning to Perceive the World Through Control: Empowerment-Based Representation Learning](https://mlsanity.com/paper/2605.30656/) (2026)
- [COLLIE: Guiding Skill Discovery in Semantically Coherent Latent Space](https://mlsanity.com/paper/2606.00950/) (2026)
- [Learning Generalizable Skill Policy with Data-Efficient Unsupervised RL](https://mlsanity.com/paper/2607.00392/) (2026)
- [The Geometry of Empowerment](https://mlsanity.com/paper/2610.07796/) (2026)
- [Multi-Agent Empowerment and Emergence of Complex Behavior in Groups](https://mlsanity.com/paper/2604.21155/) (2026)
- [Mastering Atari 2600 Games with Discovered Options](https://mlsanity.com/paper/2610.03604/) (2026)
- [Scaling Empowered Agents](https://mlsanity.com/paper/2610.13002/) (2026)

## Abstract (arXiv)

Self-supervised learning has the potential of lifting several of the key challenges in reinforcement learning today, such as exploration, representation learning, and reward design. Recent work (METRA) has effectively argued that moving away from mutual information and instead optimizing a certain Wasserstein distance is important for good performance. In this paper, we argue that the benefits seen in that paper can largely be explained within the existing framework of mutual information skill learning (MISL). Our analysis suggests a new MISL method (contrastive successor features) that retains the excellent performance of METRA with fewer moving parts, and highlights connections between skill learning, contrastive representation learning, and successor features. Finally, through careful ablation studies, we provide further insight into some of the key ingredients for both our method and METRA.