# Kickstarting Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1803.03835 · HTML: https://mlsanity.com/paper/1803.03835/

## Key facts

- Cited by 81 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Mar 2018.

## Built on by (most cited first, 30 of 81)

- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Teaching language models to support answers with verified quotes](https://mlsanity.com/paper/2203.11147/) (2022)
- [Open-Ended Learning Leads to Generally Capable Agents](https://mlsanity.com/paper/2107.12808/) (2021)
- [Understanding plasticity in neural networks](https://mlsanity.com/paper/2303.01486/) (2023)
- [Human-Timescale Adaptation in an Open-Ended Task Space](https://mlsanity.com/paper/2301.07608/) (2023)
- [Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning](https://mlsanity.com/paper/2304.13653/) (2023)
- [Universal Humanoid Motion Representations for Physics-Based Control](https://mlsanity.com/paper/2310.04582/) (2023)
- [Automated Reinforcement Learning (AutoRL): A Survey and Open Problems](https://mlsanity.com/paper/2201.03916/) (2022)
- [Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progress](https://mlsanity.com/paper/2206.01626/) (2022)
- [BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning](https://mlsanity.com/paper/2506.17211/) (2025)
- [Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search](https://mlsanity.com/paper/2502.02508/) (2025)
- [In value-based deep reinforcement learning, a pruned network is a good network](https://mlsanity.com/paper/2402.12479/) (2024)
- [Alchemy: A benchmark and analysis toolkit for meta-reinforcement learning agents](https://mlsanity.com/paper/2102.02926/) (2021)
- [Lifelong Robotic Reinforcement Learning by Retaining Experiences](https://mlsanity.com/paper/2109.09180/) (2021)
- [RLx2: Training a Sparse Deep Reinforcement Learning Model from Scratch](https://mlsanity.com/paper/2205.15043/) (2022)
- [Large Language Model as a Policy Teacher for Training Reinforcement Learning Agents](https://mlsanity.com/paper/2311.13373/) (2023)
- [Evolutionary Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.04150/) (2023)
- [On Lottery Tickets and Minimal Task Representations in Deep Reinforcement Learning](https://mlsanity.com/paper/2105.01648/) (2021)
- [Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem](https://mlsanity.com/paper/2402.02868/) (2024)
- [Pretraining in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2211.03959/) (2022)
- [TGRL: An Algorithm for Teacher Guided Reinforcement Learning](https://mlsanity.com/paper/2307.03186/) (2023)
- [Real-Time Simulated Avatar from Head-Mounted Sensors](https://mlsanity.com/paper/2403.06862/) (2024)
- [Learning Robot Soccer from Egocentric Vision with Deep Reinforcement Learning](https://mlsanity.com/paper/2405.02425/) (2024)
- [When LLMs Play the Telephone Game: Cultural Attractors as Conceptual Tools to Evaluate LLMs in Multi-turn Settings](https://mlsanity.com/paper/2407.04503/) (2024)
- [Dungeons and Data: A Large-Scale NetHack Dataset](https://mlsanity.com/paper/2211.00539/) (2022)
- [Knowledge Distillation Performs Partial Variance Reduction](https://mlsanity.com/paper/2305.17581/) (2023)
- [Guarded Policy Optimization with Imperfect Online Demonstrations](https://mlsanity.com/paper/2303.01728/) (2023)
- [How to Spend Your Robot Time: Bridging Kickstarting and Offline Reinforcement Learning for Vision-based Robotic Manipulation](https://mlsanity.com/paper/2205.03353/) (2022)
- [Proximal Policy Distillation](https://mlsanity.com/paper/2407.15134/) (2024)
- [Bridging SFT and DPO for Diffusion Model Alignment with Self-Sampling Preference Optimization](https://mlsanity.com/paper/2410.05255/) (2024)

## Abstract (arXiv)

We present a method for using previously-trained 'teacher' agents to kickstart the training of a new 'student' agent. To this end, we leverage ideas from policy distillation and population based training. Our method places no constraints on the architecture of the teacher or student agents, and it regulates itself to allow the students to surpass their teachers in performance. We show that, on a challenging and computationally-intensive multi-task benchmark (DMLab-30), kickstarted training improves the data efficiency of new agents, making it significantly easier to iterate on their design. We also show that the same kickstarting pipeline can allow a single student agent to leverage multiple 'expert' teachers which specialize on individual tasks. In this setting kickstarting yields surprisingly large gains, with the kickstarted agent matching the performance of an agent trained from scratch in almost 10x fewer steps, and surpassing its final performance by 42 percent. Kickstarting is conceptually simple and can easily be incorporated into reinforcement learning experiments.