Pre-training with Synthetic Data Helps Offline Reinforcement Learning
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
Recently, it has been shown that for offline deep reinforcement learning (DRL), pre-training Decision Transformer with a large language corpus can improve downstream performance (Reid et al., 2022). A natural question to ask is whether this performance gain can only be achieved with language pre-training, or can be achieved with simpler pre-training schemes which do not involve language. In this paper, we first show that language is not essential for improved performance, and indeed pre-training with synthetic IID data for a small number of updates can match the performance gains from pre-training with a large language corpus; moreover, pre-training with data generated by a one-step Markov chain can further improve the performance. Inspired by these experimental results, we then consider pre-training Conservative Q-Learning (CQL), a popular offline DRL algorithm, which is Q-learning-based and typically employs a Multi-Layer Perceptron (MLP) backbone. Surprisingly, pre-training with simple synthetic data for a small number of updates can also improve CQL, providing consistent performance improvement on D4RL Gym locomotion datasets. The results of this paper not only illustrate the importance of pre-training for offline DRL but also show that the pre-training data can be synthetic and generated with remarkably simple mechanisms.
Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 38 of its 51 references that are in the map. First posted on arXiv 1 Oct 2023.
Builds on
38 of 51 references.
Closest ancestors · 5
Same direction, newest first
- 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
- 2021Representation Matters: Offline Pretraining for Sequential Decision Making11 Feb
- 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
Other references · 33
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2016Pointer Sentinel Mixture Modelscited by 3,096
- 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
- 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
- 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
- 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
- 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
- 2018Do Better ImageNet Models Transfer Better?cited by 565
- 2013DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognitioncited by 424
- 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 245
21 more
- 2016What makes ImageNet good for transfer learning?cited by 192
- 2023Language-Driven Representation Learning for Roboticscited by 165
- 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
- 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 106
- 2021Pre-training without Natural Imagescited by 100
- 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 91
- 2021RRL: Resnet as representation for Reinforcement Learningcited by 91
- 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 77
- 2022On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baselinecited by 56
- 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 52
- 2023A Survey on Transformers in Reinforcement Learningcited by 50
- 2021LIME: Learning Inductive Bias for Primitives of Mathematical Reasoningcited by 46
- 2021Does Pretraining for Summarization Require Knowledge Transfer?cited by 30
- 2022Pretraining with Artificial Language: Studying Transferable Knowledge in Language Modelscited by 28
- 2021Improving Fractal Pre-trainingcited by 26
- 2022Insights into Pre-training via Simpler Synthetic Taskscited by 26
- 2023Future-conditioned Unsupervised Pretraining for Decision Transformercited by 26
- 2021On the Transferability of Pre-trained Language Models: A Study from Artificial Datasetscited by 25
- 2022Reinforcement Learning with Automated Auxiliary Loss Searchcited by 13
- 2022Synthetic Pre-Training Tasks for Neural Machine Translationcited by 8
- 2022On the Effect of Pre-training for Transformer in Different Modality on Offline Reinforcement Learningcited by 7
Built on by
Citing papers per year: 2024: 3; 2025: 5; 2026: 1.
9 papers.
- 2024Guided Trajectory Generation with Diffusion Models for Offline Model-based Optimizationcited by 15
- 2025General Intelligence Requires Reward-based Pretrainingcited by 12
- 2026Procedural Pretraining: Warming Up Language Models with Abstract Datacited by 10
- 2025Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoningcited by 9
- 2025Data Swarms: Optimizable Generation of Synthetic Evaluation Datacited by 2
- 2024Enhancing Sample Efficiency and Exploration in Reinforcement Learning through the Integration of Diffusion Models and Proximal Policy Optimizationcited by 1
- 2024One-shot World Models Using a Transformer Trained on a Synthetic Priorcited by 0
- 2025Minimal Ingredients for Reward Assignment from Expert Demonstrationscited by 0
- 2025If generative AI is the answer, what is the question?cited by 0
- 2026Procedural Pretraining: Warming Up Language Models with Abstract Data29 Jan 2026
- 2025If generative AI is the answer, what is the question?7 Sep 2025
- 2025Minimal Ingredients for Reward Assignment from Expert Demonstrations7 Jun 2025
- 2025Data Swarms: Optimizable Generation of Synthetic Evaluation Data31 May 2025
- 2025Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning28 May 2025
- 2025General Intelligence Requires Reward-based Pretraining26 Feb 2025
- 2024One-shot World Models Using a Transformer Trained on a Synthetic Prior21 Sep 2024
- 2024Enhancing Sample Efficiency and Exploration in Reinforcement Learning through the Integration of Diffusion Models and Proximal Policy Optimization2 Sep 2024
- 2024Guided Trajectory Generation with Diffusion Models for Offline Model-based Optimization29 Jun 2024
Similar, not cited
Close in wording, no citation either way
- 2025Pretraining a Shared Q-Network for Data-Efficient Offline Reinforcement Learning
- 2025Efficient Online RL Fine Tuning with Offline Pre-trained Policy Only
- 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
- 2026Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?
- 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trials
- 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning
- 2023When should we prefer Decision Transformers for Offline Reinforcement Learning?
- 2023Small Dataset, Big Gains: Enhancing Reinforcement Learning by Offline Pre-Training with Model Based Augmentation