# Stochastic Neural Networks for Hierarchical Reinforcement Learning

arXiv: https://arxiv.org/abs/1704.03012 · HTML: https://mlsanity.com/paper/1704.03012/

## Key facts

- Cited by 124 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2017.

## Built on by (most cited first, 30 of 124)

- [PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training](https://mlsanity.com/paper/2106.05091/) (2021)
- [Goal-conditioned Imitation Learning](https://mlsanity.com/paper/1906.05838/) (2019)
- [Deep Hierarchical Planning from Pixels](https://mlsanity.com/paper/2206.04114/) (2022)
- [METRA: Scalable Unsupervised RL with Metric-Aware Abstraction](https://mlsanity.com/paper/2310.08887/) (2023)
- [Reward is enough for convex MDPs](https://mlsanity.com/paper/2106.00661/) (2021)
- [Modular Deep Learning](https://mlsanity.com/paper/2302.11529/) (2023)
- [CIC: Contrastive Intrinsic Control for Unsupervised Skill Discovery](https://mlsanity.com/paper/2202.00161/) (2022)
- [Hierarchical Reinforcement Learning By Discovering Intrinsic Options](https://mlsanity.com/paper/2101.06521/) (2021)
- [Retrieval-Augmented Reinforcement Learning](https://mlsanity.com/paper/2202.08417/) (2022)
- [Landmark-Guided Subgoal Generation in Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2110.13625/) (2021)
- [Hierarchical Skills for Efficient Exploration](https://mlsanity.com/paper/2110.10809/) (2021)
- [The Information Geometry of Unsupervised Reinforcement Learning](https://mlsanity.com/paper/2110.02719/) (2021)
- [Self-optimizing Feature Generation via Categorical Hashing Representation and Hierarchical Reinforcement Crossing](https://mlsanity.com/paper/2309.04612/) (2023)
- [Unsupervised Skill Discovery with Bottleneck Option Learning](https://mlsanity.com/paper/2106.14305/) (2021)
- [An information-theoretic perspective on intrinsic motivation in reinforcement learning: a survey](https://mlsanity.com/paper/2209.08890/) (2022)
- [Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning](https://mlsanity.com/paper/2111.03189/) (2021)
- [Learning Transferable Motor Skills with Hierarchical Latent Mixture Policies](https://mlsanity.com/paper/2112.05062/) (2021)
- [Behavior Contrastive Learning for Unsupervised Skill Discovery](https://mlsanity.com/paper/2305.04477/) (2023)
- [Diverse Auto-Curriculum is Critical for Successful Real-World Multiagent Learning Systems](https://mlsanity.com/paper/2102.07659/) (2021)
- [Causality-driven Hierarchical Structure Discovery for Reinforcement Learning](https://mlsanity.com/paper/2210.06964/) (2022)
- [Pick Your Battles: Interaction Graphs as Population-Level Objectives for Strategic Diversity](https://mlsanity.com/paper/2110.04041/) (2021)
- [Leveraging Approximate Symbolic Models for Reinforcement Learning via Skill Diversity](https://mlsanity.com/paper/2202.02886/) (2022)
- [Learning Options via Compression](https://mlsanity.com/paper/2212.04590/) (2022)
- [Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learning](https://mlsanity.com/paper/2102.13515/) (2021)
- [Hierarchical Reinforcement Learning with Timed Subgoals](https://mlsanity.com/paper/2112.03100/) (2021)
- [CompoSuite: A Compositional Reinforcement Learning Benchmark](https://mlsanity.com/paper/2207.04136/) (2022)
- [Versatile Skill Control via Self-supervised Adversarial Imitation of Unlabeled Mixed Motions](https://mlsanity.com/paper/2209.07899/) (2022)
- [Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learning](https://mlsanity.com/paper/2412.08021/) (2024)
- [Discovering Diverse Solutions in Deep Reinforcement Learning by Maximizing State-Action-Based Mutual Information](https://mlsanity.com/paper/2103.07084/) (2021)
- [Solving Compositional Reinforcement Learning Problems via Task Reduction](https://mlsanity.com/paper/2103.07607/) (2021)

## Abstract (arXiv)

Deep reinforcement learning has achieved many impressive results in recent years. However, tasks with sparse rewards or long horizons continue to pose significant challenges. To tackle these important problems, we propose a general framework that first learns useful skills in a pre-training environment, and then leverages the acquired skills for learning faster in downstream tasks. Our approach brings together some of the strengths of intrinsic motivation and hierarchical methods: the learning of useful skill is guided by a single proxy reward, the design of which requires very minimal domain knowledge about the downstream tasks. Then a high-level policy is trained on top of these skills, providing a significant improvement of the exploration and allowing to tackle sparse rewards in the downstream tasks. To efficiently pre-train a large span of skills, we use Stochastic Neural Networks combined with an information-theoretic regularizer. Our experiments show that this combination is effective in learning a wide span of interpretable skills in a sample-efficient way, and can significantly boost the learning performance uniformly across a wide range of downstream tasks.