mlsanityData to 7 Oct 2026

Representation Matters: Offline Pretraining for Sequential Decision Making

arXiv 2102.05815 ↗First posted 11 Feb 2021cs.LG cs.AIcited by 90

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

The recent success of supervised learning methods on ever larger offline datasets has spurred interest in the reinforcement learning (RL) field to investigate whether the same paradigms can be translated to RL algorithms. This research area, known as offline RL, has largely focused on offline policy optimization, aiming to find a return-maximizing policy exclusively from offline data. In this paper, we consider a slightly different approach to incorporating offline data into sequential decision-making. We aim to answer the question, what unsupervised objectives applied to offline datasets are able to learn state representations which elevate performance on downstream tasks, whether those downstream tasks be online RL, imitation learning from expert demonstrations, or even offline policy optimization based on the same offline dataset? Through a variety of experiments utilizing standard offline RL datasets, we find that the use of pretraining with unsupervised learning objectives can dramatically improve the performance of policy learning algorithms that otherwise yield mediocre performance on their own. Extensive ablations further provide insights into what components of these unsupervised objectives -- e.g., reward prediction, continuous or discrete representations, pretraining or finetuning -- are most important and in which settings.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 57 references that are in the map. First posted on arXiv 11 Feb 2021.

Builds on

35 of 57 references.

Closest ancestors · 6

Same direction, newest first

  1. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning26 Oct
  2. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  3. 2020MOPO: Model-based Offline Policy Optimization27 May
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  6. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  8. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  9. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  10. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  11. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  12. 2020Mastering Atari with Discrete World Modelscited by 990
17 more
  1. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  2. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  3. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  4. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  5. 2019Challenges of Real-World Reinforcement Learningcited by 282
  6. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  7. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  8. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  9. 2017Imagination-Augmented Agents for Deep Reinforcement Learningcited by 137
  10. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  11. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  12. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  13. 2017Value Prediction Networkcited by 117
  14. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 107
  15. 2018The Laplacian in RL: Learning Representations with Efficient Approximationscited by 83
  16. 2016Loss is its own Reward: Self-Supervision for Reinforcement Learningcited by 72
  17. 2019Prediction, Consistency, Curvature: Representation Learning for Locally-Linear Controlcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 31; 2023: 30; 2024: 10; 2025: 5; 2026: 2.

90 papers.

Sort:

By area:

  1. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  2. 2022Multi-Game Decision Transformerscited by 212
  3. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  4. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 128
  5. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  6. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
  7. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 103
  8. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
  9. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  10. 2022A Survey on Model-based Reinforcement Learningcited by 65
  11. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  12. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learningcited by 61
38 more
  1. 2021Corruption-Robust Offline Reinforcement Learningcited by 51
  2. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 51
  3. 2023SMART: Self-supervised Multi-task pretrAining with contRol Transformerscited by 44
  4. 2023Masked Trajectory Models for Prediction, Representation, and Controlcited by 40
  5. 2022Spectral Decomposition Representation for Reinforcement Learningcited by 35
  6. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  7. 2021Provable Representation Learning for Imitation with Contrastive Fourier Featurescited by 34
  8. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learningcited by 34
  9. 2022Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Errorcited by 31
  10. 2022Chain of Thought Imitation with Procedure Cloningcited by 30
  11. 2023Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitationcited by 30
  12. 2022Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learningcited by 26
  13. 2023Future-conditioned Unsupervised Pretraining for Decision Transformercited by 26
  14. 2023Policy Pre-training for Autonomous Driving via Self-supervised Geometric Modelingcited by 25
  15. 2022Investigating the Properties of Neural Network Representations in Reinforcement Learningcited by 24
  16. 2022PACT: Perception-Action Causal Transformer for Autoregressive Robotics Pre-Trainingcited by 24
  17. 2022Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humanscited by 23
  18. 2022SAM-RL: Sensing-Aware Model-Based Reinforcement Learning via Differentiable Physics-Based Simulation and Renderingcited by 22
  19. 2022Multi-Task Imitation Learning for Linear Dynamical Systemscited by 22
  20. 2022Learning Options via Compressioncited by 22
  21. 2023A Survey on Physics Informed Reinforcement Learning: Review and Open Problemscited by 22
  22. 2023Accelerating Exploration with Unlabeled Prior Datacited by 21
  23. 2021Cross-Trajectory Representation Learning for Zero-Shot Generalization in RLcited by 20
  24. 2023Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Datacited by 19
  25. 2024Offline Model-Based Optimization via Policy-Guided Gradient Searchcited by 19
  26. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  27. 2023Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RLcited by 18
  28. 2022Learning Bellman Complete Representations for Offline Policy Evaluationcited by 17
  29. 2023Simplified Temporal Consistency Reinforcement Learningcited by 17
  30. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  31. 2021Accelerating Offline Reinforcement Learning Application in Real-Time Bidding and Recommendation: Potential Use of Simulationcited by 16
  32. 2022A Mixture-of-Expert Approach to RL-based Dialogue Managementcited by 16
  33. 2023SNeRL: Semantic-aware Neural Radiance Fields for Reinforcement Learningcited by 16
  34. 2022An Empirical Investigation of Representation Learning for Imitationcited by 15
  35. 2022Agent-Controller Representations: Principled Offline RL with Rich Exogenous Informationcited by 15
  36. 2022Representation Learning in Deep RL via Discrete Information Bottleneckcited by 13
  37. 2024MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learningcited by 13
  38. 2022Behavior Prior Representation learning for Offline Reinforcement Learningcited by 12

50 most cited of 90 All 90 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  2. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  3. 2024Unsupervised-to-Online Reinforcement Learning
  4. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  5. 2021Instabilities of Offline RL with Pre-Trained Neural Representation
  6. 2024Augmenting Offline RL with Unlabeled Data
  7. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  8. 2025A Tutorial: An Intuitive Explanation of Offline Reinforcement Learning Theory