# Pretraining in Deep Reinforcement Learning: A Survey

arXiv: https://arxiv.org/abs/2211.03959 · HTML: https://mlsanity.com/paper/2211.03959/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 174 of its 216 references that are in the map.
- First posted on arXiv 8 Nov 2022.

## Builds on (most cited first, 30 of 174)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Density estimation using Real NVP](https://mlsanity.com/paper/1605.08803/) (2016)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)

## Built on by (most cited first, 18 of 18)

- [Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards](https://mlsanity.com/paper/2306.04488/) (2023)
- [A Survey on Transformers in Reinforcement Learning](https://mlsanity.com/paper/2301.03044/) (2023)
- [Future-conditioned Unsupervised Pretraining for Decision Transformer](https://mlsanity.com/paper/2305.16683/) (2023)
- [End-to-End (Instance)-Image Goal Navigation through Correspondence as an Emergent Phenomenon](https://mlsanity.com/paper/2309.16634/) (2023)
- [Learning to Discover Skills through Guidance](https://mlsanity.com/paper/2310.20178/) (2023)
- [Constrained Ensemble Exploration for Unsupervised Skill Discovery](https://mlsanity.com/paper/2405.16030/) (2024)
- [Learning Adversarial Low-rank Markov Decision Processes with Unknown Transition and Full-information Feedback](https://mlsanity.com/paper/2311.07876/) (2023)
- [S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learning](https://mlsanity.com/paper/2408.12130/) (2024)
- [Introspective Action Advising for Interpretable Transfer Learning](https://mlsanity.com/paper/2306.12314/) (2023)
- [Self-supervised Pretraining for Decision Foundation Model: Formulation, Pipeline and Challenges](https://mlsanity.com/paper/2401.00031/) (2023)
- [Information-Theoretic Policy Pre-Training with Empowerment](https://mlsanity.com/paper/2510.05996/) (2025)
- [Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments](https://mlsanity.com/paper/2304.09825/) (2023)
- [COLLIE: Guiding Skill Discovery in Semantically Coherent Latent Space](https://mlsanity.com/paper/2606.00950/) (2026)
- [A Central Motor System Inspired Pre-training Reinforcement Learning for Robotic Control](https://mlsanity.com/paper/2311.07822/) (2023)
- [Value Explicit Pretraining for Learning Transferable Representations](https://mlsanity.com/paper/2312.12339/) (2023)
- [A Non-Monolithic Policy Approach of Offline-to-Online Reinforcement Learning](https://mlsanity.com/paper/2410.23737/) (2024)
- [Local Pairwise Distance Matching for Backpropagation-Free Reinforcement Learning](https://mlsanity.com/paper/2507.11367/) (2025)
- [Pretraining in Actor-Critic Reinforcement Learning for Locomotion](https://mlsanity.com/paper/2510.12363/) (2025)

## Abstract (arXiv)

The past few years have seen rapid progress in combining reinforcement learning (RL) with deep learning. Various breakthroughs ranging from games to robotics have spurred the interest in designing sophisticated RL algorithms and systems. However, the prevailing workflow in RL is to learn tabula rasa, which may incur computational inefficiency. This precludes continuous deployment of RL algorithms and potentially excludes researchers without large-scale computing resources. In many other areas of machine learning, the pretraining paradigm has shown to be effective in acquiring transferable knowledge, which can be utilized for a variety of downstream tasks. Recently, we saw a surge of interest in Pretraining for Deep RL with promising results. However, much of the research has been based on different experimental settings. Due to the nature of RL, pretraining in this field is faced with unique challenges and hence requires new design principles. In this survey, we seek to systematically review existing works in pretraining for deep reinforcement learning, provide a taxonomy of these methods, discuss each sub-field, and bring attention to open problems and future directions.