mlsanityData to 14 Oct 2026

Pretraining in Deep Reinforcement Learning: A Survey

arXiv 2211.03959 ↗First posted 8 Nov 2022cs.LG cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

The past few years have seen rapid progress in combining reinforcement learning (RL) with deep learning. Various breakthroughs ranging from games to robotics have spurred the interest in designing sophisticated RL algorithms and systems. However, the prevailing workflow in RL is to learn tabula rasa, which may incur computational inefficiency. This precludes continuous deployment of RL algorithms and potentially excludes researchers without large-scale computing resources. In many other areas of machine learning, the pretraining paradigm has shown to be effective in acquiring transferable knowledge, which can be utilized for a variety of downstream tasks. Recently, we saw a surge of interest in Pretraining for Deep RL with promising results. However, much of the research has been based on different experimental settings. Due to the nature of RL, pretraining in this field is faced with unique challenges and hence requires new design principles. In this survey, we seek to systematically review existing works in pretraining for deep reinforcement learning, provide a taxonomy of these methods, discuss each sub-field, and bring attention to open problems and future directions.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 174 of its 216 references that are in the map. First posted on arXiv 8 Nov 2022.

Builds on

174 of 216 references.

References · 174

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  11. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  12. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
38 more
  1. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  2. 2013Auto-Encoding Variational Bayescited by 4,053
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  5. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  6. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  8. 2016Density estimation using Real NVPcited by 2,042
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  10. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  11. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  12. 2018World Modelscited by 1,513
  13. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  14. 2016Progressive Neural Networkscited by 1,361
  15. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  16. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  17. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  18. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  19. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  20. 2018Exploration by Random Network Distillationcited by 993
  21. 2018DeepMind Control Suitecited by 956
  22. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 913
  23. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 901
  24. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  25. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  26. 2022A Generalist Agentcited by 808
  27. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  28. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  29. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  30. 2019Solving Rubik's Cube with a Robot Handcited by 734
  31. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  32. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  33. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  34. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  35. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
  36. 2019Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmscited by 567
  37. 2019On Variational Bounds of Mutual Informationcited by 559
  38. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511

50 most cited of 174 All 174 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 11; 2024: 3; 2025: 3; 2026: 1.

18 papers.

Sort:

By area:

  1. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 243
  2. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  3. 2023Future-conditioned Unsupervised Pretraining for Decision Transformercited by 26
  4. 2023End-to-End (Instance)-Image Goal Navigation through Correspondence as an Emergent Phenomenoncited by 16
  5. 2023Learning to Discover Skills through Guidancecited by 8
  6. 2024Constrained Ensemble Exploration for Unsupervised Skill Discoverycited by 8
  7. 2023Learning Adversarial Low-rank Markov Decision Processes with Unknown Transition and Full-information Feedbackcited by 4
  8. 2024S-EPOA: Overcoming the Indistinguishability of Segments with Skill-Driven Preference-Based Reinforcement Learningcited by 4
  9. 2023Introspective Action Advising for Interpretable Transfer Learningcited by 2
  10. 2023Self-supervised Pretraining for Decision Foundation Model: Formulation, Pipeline and Challengescited by 2
  11. 2025Information-Theoretic Policy Pre-Training with Empowermentcited by 2
  12. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environmentscited by 1
6 more
  1. 2026COLLIE: Guiding Skill Discovery in Semantically Coherent Latent Spacecited by 1
  2. 2023A Central Motor System Inspired Pre-training Reinforcement Learning for Robotic Controlcited by 0
  3. 2023Value Explicit Pretraining for Learning Transferable Representationscited by 0
  4. 2024A Non-Monolithic Policy Approach of Offline-to-Online Reinforcement Learningcited by 0
  5. 2025Local Pairwise Distance Matching for Backpropagation-Free Reinforcement Learningcited by 0
  6. 2025Pretraining in Actor-Critic Reinforcement Learning for Locomotioncited by 0