# Hierarchical Kickstarting for Skill Transfer in Reinforcement Learning

arXiv: https://arxiv.org/abs/2207.11584 · HTML: https://mlsanity.com/paper/2207.11584/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 33 references that are in the map.
- First posted on arXiv 23 Jul 2022.

## Builds on (most cited first, 24 of 24)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [The Option-Critic Architecture](https://mlsanity.com/paper/1609.05140/) (2016)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [Population Based Training of Neural Networks](https://mlsanity.com/paper/1711.09846/) (2017)
- [FeUdal Networks for Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1703.01161/) (2017)
- [Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation](https://mlsanity.com/paper/1604.06057/) (2016)
- [Distral: Robust Multitask Reinforcement Learning](https://mlsanity.com/paper/1707.04175/) (2017)
- [Benchmarking the Spectrum of Agent Capabilities](https://mlsanity.com/paper/2109.06780/) (2021)
- [The NetHack Learning Environment](https://mlsanity.com/paper/2006.13760/) (2020)
- [ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learning](https://mlsanity.com/paper/1605.02097/) (2016)
- [Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning](https://mlsanity.com/paper/1511.06342/) (2015)
- [Effective Diversity in Population Based Reinforcement Learning](https://mlsanity.com/paper/2002.00632/) (2020)
- [MiniHack the Planet: A Sandbox for Open-Ended Reinforcement Learning Research](https://mlsanity.com/paper/2109.13202/) (2021)
- [Meta Learning Shared Hierarchies](https://mlsanity.com/paper/1710.09767/) (2017)
- [Distilling Policy Distillation](https://mlsanity.com/paper/1902.02186/) (2019)
- [Kickstarting Deep Reinforcement Learning](https://mlsanity.com/paper/1803.03835/) (2018)
- [TorchBeast: A PyTorch Platform for Distributed RL](https://mlsanity.com/paper/1910.03552/) (2019)
- [The MineRL 2020 Competition on Sample Efficient Reinforcement Learning using Human Priors](https://mlsanity.com/paper/2101.11071/) (2021)
- [CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents](https://mlsanity.com/paper/2110.10067/) (2021)

## Built on by (most cited first, 7 of 7)

- [Large Language Model as a Policy Teacher for Training Reinforcement Learning Agents](https://mlsanity.com/paper/2311.13373/) (2023)
- [Dungeons and Data: A Large-Scale NetHack Dataset](https://mlsanity.com/paper/2211.00539/) (2022)
- [LLM Augmented Hierarchical Agents](https://mlsanity.com/paper/2311.05596/) (2023)
- [Developing Driving Strategies Efficiently: A Skill-Based Hierarchical Reinforcement Learning Approach](https://mlsanity.com/paper/2302.02179/) (2023)
- [MIRA: Memory-Integrated Reinforcement Learning Agent with Limited LLM Guidance](https://mlsanity.com/paper/2602.17930/) (2026)
- [Hierarchical Behaviour Spaces](https://mlsanity.com/paper/2604.24558/) (2026)
- [Sample Efficient Hierarchical Reinforcement Learning via Best Policy Identification](https://mlsanity.com/paper/2607.29294/) (2026)

## Abstract (arXiv)

Practising and honing skills forms a fundamental component of how humans learn, yet artificial agents are rarely specifically trained to perform them. Instead, they are usually trained end-to-end, with the hope being that useful skills will be implicitly learned in order to maximise discounted return of some extrinsic reward function. In this paper, we investigate how skills can be incorporated into the training of reinforcement learning (RL) agents in complex environments with large state-action spaces and sparse rewards. To this end, we created SkillHack, a benchmark of tasks and associated skills based on the game of NetHack. We evaluate a number of baselines on this benchmark, as well as our own novel skill-based method Hierarchical Kickstarting (HKS), which is shown to outperform all other evaluated methods. Our experiments show that learning with a prior knowledge of useful skills can significantly improve the performance of agents on complex problems. We ultimately argue that utilising predefined skills provides a useful inductive bias for RL problems, especially those with large state-action spaces and sparse rewards.