mlsanityData to 7 Oct 2026

Pretraining Representations for Data-Efficient Reinforcement Learning

arXiv 2106.04799 ↗First posted 9 Jun 2021cs.LGcited by 105

Not on the map. Why?

Abstract (arXiv)

Data efficiency is a key challenge for deep reinforcement learning. We address this problem by using unlabeled data to pretrain an encoder which is then finetuned on a small amount of task-specific data. To encourage learning representations which capture diverse aspects of the underlying MDP, we employ a combination of latent dynamics modelling and unsupervised goal-conditioned RL. When limited to 100k steps of interaction on Atari games (equivalent to two hours of human experience), our approach significantly surpasses prior work combining offline representation pretraining with task-specific finetuning, and compares favourably with other pretraining methods that require orders of magnitude more data. Our approach shows particular promise when combined with larger models as well as more diverse, task-aligned observational data -- approaching human-level performance and data-efficiency on Atari in our best setting. We provide code associated with this work at https://github.com/mila-iqia/SGI.

Key facts. Cited by 105 papers in the map (data through 7 Oct 2026). Builds on 57 of its 72 references that are in the map. First posted on arXiv 9 Jun 2021.

Builds on

57 of 72 references.

References · 57

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  10. 2020Scaling Laws for Neural Language Modelscited by 6,411
  11. 2016Layer Normalizationcited by 5,579
  12. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
38 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  2. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  3. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  4. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  5. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  6. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  7. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  8. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  9. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  10. 2017Hindsight Experience Replaycited by 1,096
  11. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  12. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  13. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  14. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  15. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  16. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 733
  17. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  18. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  19. 2019Model-Based Reinforcement Learning for Ataricited by 495
  20. 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 470
  21. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  22. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  23. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  24. 2018Large-Scale Study of Curiosity-Driven Learningcited by 277
  25. 2021Scaling Laws for Transfercited by 265
  26. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  27. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  28. 2021Reinforcement Learning with Prototypical Representationscited by 179
  29. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 170
  30. 2019Unsupervised State Representation Learning in Ataricited by 144
  31. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 131
  32. 2018State Representation Learning for Control: An Overviewcited by 125
  33. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 121
  34. 2019When to use parametric models in reinforcement learning?cited by 118
  35. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 101
  36. 2017Building machines that adapt and compute like brainscited by 99
  37. 2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 90
  38. 2020Deep Reinforcement and InfoMax Learningcited by 79

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 25; 2023: 33; 2024: 19; 2025: 16; 2026: 5.

105 papers.

Sort:

By area:

  1. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  2. 2023A Cookbook of Self-Supervised Learningcited by 248
  3. 2023Getting aligned on representational alignmentcited by 142
  4. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  5. 2025FLARE: Robot Learning with Implicit World Modelingcited by 105
  6. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  7. 2023Multi-View Masked World Models for Visual Robotic Manipulationcited by 78
  8. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Modelscited by 77
  9. 2022Offline Visual Representation Learning for Embodied Navigationcited by 74
  10. 2021DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representationscited by 73
  11. 2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 66
  12. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 58
38 more
  1. 2023TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learningcited by 56
  2. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 51
  3. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 51
  4. 2022Masked Autoencoding for Scalable and Generalizable Decision Makingcited by 44
  5. 2023SMART: Self-supervised Multi-task pretrAining with contRol Transformerscited by 44
  6. 2024DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Controlcited by 43
  7. 2022Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Modelscited by 42
  8. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  9. 2023Proto-Value Networks: Scaling Representation Learning with Auxiliary Taskscited by 41
  10. 2022Mask-based Latent Reconstruction for Reinforcement Learningcited by 39
  11. 2021The Information Geometry of Unsupervised Reinforcement Learningcited by 36
  12. 2023Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitationcited by 30
  13. 2023Future-conditioned Unsupervised Pretraining for Decision Transformercited by 26
  14. 2024CyberDemo: Augmenting Simulated Human Demonstration for Real-World Dexterous Manipulationcited by 25
  15. 2022Joint Embedding Predictive Architectures Focus on Slow Featurescited by 24
  16. 2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 23
  17. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
  18. 2022Planning for Sample Efficient Imitation Learningcited by 20
  19. 2022Temporal Disentanglement of Representations for Improved Generalisation in Reinforcement Learningcited by 19
  20. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  21. 2021Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functionscited by 17
  22. 2024Proto Successor Measure: Representing the Behavior Space of an RL Agentcited by 17
  23. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  24. 2025Language Models Can Learn from Verbal Feedback Without Scalar Rewardscited by 17
  25. 2022Light-weight probing of unsupervised representations for Reinforcement Learningcited by 15
  26. 2022Agent-Controller Representations: Principled Offline RL with Rich Exogenous Informationcited by 15
  27. 2022Contrastive Value Learning: Implicit Models for Simple Offline RLcited by 15
  28. 2022Representation Learning in Deep RL via Discrete Information Bottleneckcited by 13
  29. 2024Open-World Reinforcement Learning over Long Short-Term Imaginationcited by 13
  30. 2024Prioritized Generative Replaycited by 13
  31. 2022Behavior Prior Representation learning for Offline Reinforcement Learningcited by 12
  32. 2023Aligning Robot and Human Representationscited by 11
  33. 2022Perceiving the World: Question-guided Reinforcement Learning for Text-based Gamescited by 10
  34. 2022Discrete Factorial Representations as an Abstraction for Goal Conditioned Reinforcement Learningcited by 10
  35. 2023Value function estimation using conditional diffusion models for controlcited by 10
  36. 2023Understanding and Addressing the Pitfalls of Bisimulation-based Representations in Offline Reinforcement Learningcited by 10
  37. 2024Premier-TACO is a Few-Shot Policy Learner: Pretraining Multitask Representation via Temporal Action-Driven Contrastive Losscited by 10
  38. 2023Cross-domain Random Pre-training with Prototypes for Reinforcement Learningcited by 9

50 most cited of 105 All 105 (JSON)