# Disentangled Unsupervised Skill Discovery for Efficient Hierarchical Reinforcement Learning

arXiv: https://arxiv.org/abs/2410.11251 · HTML: https://mlsanity.com/paper/2410.11251/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 43 of its 55 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 30 of 43)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Object-Centric Learning with Slot Attention](https://mlsanity.com/paper/2006.15055/) (2020)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [On Variational Bounds of Mutual Information](https://mlsanity.com/paper/1905.06922/) (2019)
- [Dynamics-Aware Unsupervised Discovery of Skills](https://mlsanity.com/paper/1907.01657/) (2019)
- [Weakly-Supervised Disentanglement Without Compromises](https://mlsanity.com/paper/2002.02886/) (2020)
- [Contrastive Learning of Structured World Models](https://mlsanity.com/paper/1911.12247/) (2019)
- [iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks](https://mlsanity.com/paper/2108.03272/) (2021)
- [Deep Reinforcement Learning for Robotics: A Survey of Real-World Successes](https://mlsanity.com/paper/2408.03539/) (2024)
- [Skew-Fit: State-Covering Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/1903.03698/) (2019)
- [SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models](https://mlsanity.com/paper/2210.05861/) (2022)
- [Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills](https://mlsanity.com/paper/2002.03647/) (2020)
- [Unsupervised Control Through Non-Parametric Discriminative Rewards](https://mlsanity.com/paper/1811.11359/) (2018)
- [Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learning](https://mlsanity.com/paper/2007.02832/) (2020)
- [Counterfactual Data Augmentation using Locally Factored Dynamics](https://mlsanity.com/paper/2007.02863/) (2020)
- [Object-Centric Slot Diffusion](https://mlsanity.com/paper/2303.10834/) (2023)
- [METRA: Scalable Unsupervised RL with Metric-Aware Abstraction](https://mlsanity.com/paper/2310.08887/) (2023)
- [Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Survey](https://mlsanity.com/paper/2012.09830/) (2020)
- [Bottom-Up Skill Discovery from Unsegmented Demonstrations for Long-Horizon Robot Manipulation](https://mlsanity.com/paper/2109.13841/) (2021)
- [CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery](https://mlsanity.com/paper/2202.00161/) (2022)
- [The Option Keyboard: Combining Skills in Reinforcement Learning](https://mlsanity.com/paper/2106.13105/) (2021)
- [Causal Dynamics Learning for Task-Independent State Abstraction](https://mlsanity.com/paper/2206.13452/) (2022)
- [Lipschitz-constrained Unsupervised Skill Discovery](https://mlsanity.com/paper/2202.00914/) (2022)
- [FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuning](https://mlsanity.com/paper/2409.16578/) (2024)

## Built on by (most cited first, 9 of 9)

- [SkiLD: Unsupervised Skill Discovery Guided by Factor Interactions](https://mlsanity.com/paper/2410.18416/) (2024)
- [Self-Improving Skill Learning for Robust Skill-based Meta-Reinforcement Learning](https://mlsanity.com/paper/2502.03752/) (2025)
- [DETACH: Cross-domain Learning for Long-Horizon Tasks via Mixture of Disentangled Experts](https://mlsanity.com/paper/2508.07842/) (2025)
- [From Pixels to Factors: Learning Independently Controllable State Variables for Reinforcement Learning](https://mlsanity.com/paper/2510.02484/) (2025)
- [Learning Task-Sufficient World Models by Synergizing Agentic Exploration and Structured Modeling](https://mlsanity.com/paper/2607.04409/) (2026)
- [Imitation Learning Based on Disentangled Representation Learning of Behavioral Characteristics](https://mlsanity.com/paper/2509.04737/) (2025)
- [Periodic Skill Discovery](https://mlsanity.com/paper/2511.03187/) (2025)
- [SUSD: Structured Unsupervised Skill Discovery through State Factorization](https://mlsanity.com/paper/2602.01619/) (2026)
- [Focused Skill Discovery: Learning to Control Specific State Variables while Minimizing Side Effects](https://mlsanity.com/paper/2510.04901/) (2025)

## Abstract (arXiv)

A hallmark of intelligent agents is the ability to learn reusable skills purely from unsupervised interaction with the environment. However, existing unsupervised skill discovery methods often learn entangled skills where one skill variable simultaneously influences many entities in the environment, making downstream skill chaining extremely challenging. We propose Disentangled Unsupervised Skill Discovery (DUSDi), a method for learning disentangled skills that can be efficiently reused to solve downstream tasks. DUSDi decomposes skills into disentangled components, where each skill component only affects one factor of the state space. Importantly, these skill components can be concurrently composed to generate low-level actions, and efficiently chained to tackle downstream tasks through hierarchical Reinforcement Learning. DUSDi defines a novel mutual-information-based objective to enforce disentanglement between the influences of different skill components, and utilizes value factorization to optimize this objective efficiently. Evaluated in a set of challenging environments, DUSDi successfully learns disentangled skills, and significantly outperforms previous skill discovery methods when it comes to applying the learned skills to solve downstream tasks. Code and skills visualization at jiahenghu.github.io/DUSDi-site/.