mlsanityData to 7 Oct 2026

Hierarchical Empowerment: Towards Tractable Empowerment-Based Skill Learning

arXiv 2307.02728 ↗First posted 6 Jul 2023cs.LG cs.AI cs.ROcited by 7

Not on the map. Why?

Abstract (arXiv)

General purpose agents will require large repertoires of skills. Empowerment -- the maximum mutual information between skills and states -- provides a pathway for learning large collections of distinct skills, but mutual information is difficult to optimize. We introduce a new framework, Hierarchical Empowerment, that makes computing empowerment more tractable by integrating concepts from Goal-Conditioned Hierarchical Reinforcement Learning. Our framework makes two specific contributions. First, we introduce a new variational lower bound on mutual information that can be used to compute empowerment over short horizons. Second, we introduce a hierarchical architecture for computing empowerment over exponentially longer time scales. We verify the contributions of the framework in a series of simulated robotics tasks. In a popular ant navigation domain, our four level agents are able to learn skills that cover a surface area over two orders of magnitude larger than prior work.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 37 of its 66 references that are in the map. First posted on arXiv 6 Jul 2023.

Builds on

37 of 66 references.

References · 37

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  3. 2017Hindsight Experience Replaycited by 1,096
  4. 2018Exploration by Random Network Distillationcited by 966
  5. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  6. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  7. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  8. 2016The Option-Critic Architecturecited by 433
  9. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 375
  10. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  11. 2016Variational Intrinsic Controlcited by 262
  12. 2017Count-Based Exploration with Neural Density Modelscited by 255
25 more
  1. 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
  2. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  3. 2019Efficient Exploration via State Marginal Matchingcited by 186
  4. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 171
  5. 2015Learning Continuous Control Policies by Stochastic Value Gradientscited by 156
  6. 2015Variational Information Maximisation for Intrinsically Motivated Reinforcement Learningcited by 142
  7. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 131
  8. 2017Stochastic Neural Networks for Hierarchical Reinforcement Learningcited by 124
  9. 2018Variational Option Discovery Algorithmscited by 121
  10. 2020Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skillscited by 120
  11. 2018Unsupervised Control Through Non-Parametric Discriminative Rewardscited by 102
  12. 2020Maximum Entropy Gain Exploration for Long Horizon Multi-goal Reinforcement Learningcited by 100
  13. 2018CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learningcited by 78
  14. 2019Keeping Your Distance: Solving Sparse Reward Tasks Using Self-Balancing Shaped Rewardscited by 76
  15. 2013Empowerment -- an Introductioncited by 65
  16. 2022Lipschitz-constrained Unsupervised Skill Discoverycited by 60
  17. 2021Hierarchical Reinforcement Learning By Discovering Intrinsic Optionscited by 57
  18. 2013Active Learning of Inverse Models with Intrinsically Motivated Goal Exploration in Robotscited by 56
  19. 2021Learning more skills through optimistic explorationcited by 37
  20. 2020Relative Variational Intrinsic Controlcited by 34
  21. 2021Unsupervised Skill Discovery with Bottleneck Option Learningcited by 28
  22. 2023Flipping Coins to Estimate Pseudocounts for Exploration in Reinforcement Learningcited by 28
  23. 2017Learning Multi-Level Hierarchies with Hindsightcited by 19
  24. 2021Braxlines: Fast and Interactive Toolkit for RL-driven Behavior Engineering beyond Reward Maximizationcited by 11
  25. 2021HAC Explore: Accelerating Exploration with Hierarchical Reinforcement Learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 1; 2026: 2.

7 papers.

  1. 2025Plasticity as the Mirror of Empowermentcited by 11
  2. 2024SkiLD: Unsupervised Skill Discovery Guided by Factor Interactionscited by 9
  3. 2024Learning Action-based Representations Using Invariancecited by 5
  4. 2024MENTOR: Guiding Hierarchical Reinforcement Learning with Human Feedback and Dynamic Distance Constraintcited by 1
  5. 2024Latent-Predictive Empowerment: Measuring Empowerment without a Simulatorcited by 1
  6. 2026C-STEP: Continuous Space-Time Empowerment for Physics-informed Safe Reinforcement Learning of Mobile Agentscited by 0
  7. 2026Unifying Goal-Conditioned RL and Unsupervised Skill Learning via Control-Maximizationcited by 0