# Pretraining & Reinforcement Learning: Sharpening the Axe Before Cutting the Tree

arXiv: https://arxiv.org/abs/2110.02497 · HTML: https://mlsanity.com/paper/2110.02497/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 14 references that are in the map.
- First posted on arXiv 6 Oct 2021.

## Builds on (most cited first, 11 of 11)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [Using Pre-Training Can Improve Model Robustness and Uncertainty](https://mlsanity.com/paper/1901.09960/) (2019)
- [Pretrained Transformers Improve Out-of-Distribution Robustness](https://mlsanity.com/paper/2004.06100/) (2020)
- [Self-Supervised Policy Adaptation during Deployment](https://mlsanity.com/paper/2007.04309/) (2020)
- [Pre-training Neural Networks with Human Demonstrations for Deep Reinforcement Learning](https://mlsanity.com/paper/1709.04083/) (2017)
- [Pre-training with Non-expert Human Demonstration for Deep Reinforcement Learning](https://mlsanity.com/paper/1812.08904/) (2018)
- [Deep Reinforcement Learning with Pre-training for Time-efficient Training of Automatic Speech Recognition](https://mlsanity.com/paper/2005.11172/) (2020)

## Built on by (most cited first, 1 of 1)

- [SMART: Self-supervised Multi-task pretrAining with contRol Transformers](https://mlsanity.com/paper/2301.09816/) (2023)

## Abstract (arXiv)

Pretraining is a common technique in deep learning for increasing performance and reducing training time, with promising experimental results in deep reinforcement learning (RL). However, pretraining requires a relevant dataset for training. In this work, we evaluate the effectiveness of pretraining for RL tasks, with and without distracting backgrounds, using both large, publicly available datasets with minimal relevance, as well as case-by-case generated datasets labeled via self-supervision. Results suggest filters learned during training on less relevant datasets render pretraining ineffective, while filters learned during training on the in-distribution datasets reliably reduce RL training time and improve performance after 80k RL training steps. We further investigate, given a limited number of environment steps, how to optimally divide the available steps into pretraining and RL training to maximize RL performance. Our code is available on GitHub