mlsanityData to 7 Oct 2026

Bootstrapped Transformer for Offline Reinforcement Learning

arXiv 2206.08569 ↗First posted 17 Jun 2022cs.LG cs.AI cs.ROcited by 37

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Offline reinforcement learning (RL) aims at learning policies from previously collected static trajectory data without interacting with the real environment. Recent works provide a novel perspective by viewing offline RL as a generic sequence generation problem, adopting sequence models such as Transformer architecture to model distributions over trajectories, and repurposing beam search as a planning algorithm. However, the training datasets utilized in general offline RL tasks are quite limited and often suffer from insufficient distribution coverage, which could be harmful to training sequence generation models yet has not drawn enough attention in the previous works. In this paper, we propose a novel algorithm named Bootstrapped Transformer, which incorporates the idea of bootstrapping and leverages the learned model to self-generate more offline data to further boost the sequence model training. We conduct extensive experiments on two offline RL benchmarks and demonstrate that our model can largely remedy the existing offline RL training limitations and beat other strong baseline methods. We also analyze the generated pseudo data and the revealed characteristics may shed some light on offline RL training. The codes are available at https://seqml.github.io/bootorl.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 42 of its 68 references that are in the map. First posted on arXiv 17 Jun 2022.

Builds on

42 of 68 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Can Wikipedia Help Offline Reinforcement Learning?28 Jan
  2. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem3 Jun
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  7. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  8. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  10. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  11. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  12. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
27 more
  1. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  2. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  3. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  4. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  5. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  6. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  7. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  8. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  9. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  10. 2017Non-Autoregressive Neural Machine Translationcited by 457
  11. 2020Reinforcement Learning with Augmented Datacited by 452
  12. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  13. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  14. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  15. 2016Dual Learning for Machine Translationcited by 146
  16. 2020Generalization in Reinforcement Learning by Soft Data Augmentationcited by 132
  17. 2020Model-Based Offline Planningcited by 126
  18. 2020Glancing Transformer for Non-Autoregressive Neural Machine Translationcited by 103
  19. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  20. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
  21. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learningcited by 63
  22. 2020Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentationcited by 57
  23. 2021Augmenting Sequential Recommendation with Pseudo-Prior Items via Reversely Pre-training Transformercited by 45
  24. 2021SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policiescited by 42
  25. 2019Lifelong Sequential Modeling with Personalized Memorization for User Response Predictioncited by 41
  26. 2021Universal Trading for Order Execution with Oracle Policy Distillationcited by 29
  27. 2022PerfectDou: Dominating DouDizhu with Perfect Information Distillationcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 16; 2024: 9; 2025: 7; 2026: 3.

37 papers.

Sort:

By area:

  1. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Plannerscited by 147
  2. 2023MetaDiffuser: Diffusion Model as Conditional Planner for Offline Meta-RLcited by 65
  3. 2022Efficient Planning in a Compact Latent Action Spacecited by 57
  4. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  5. 2023Crossway Diffusion: Improving Diffusion-based Visuomotor Policy via Self-supervised Learningcited by 37
  6. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  7. 2024DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitchingcited by 35
  8. 2023Learning to Modulate pre-trained Models in RLcited by 28
  9. 2023Bootstrapped Training of Score-Conditioned Generator for Offline Design of Biological Sequencescited by 23
  10. 2023Graph Decision Transformercited by 20
  11. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  12. 2023Large Sequence Models for Sequential Decision-Making: A Surveycited by 19
25 more
  1. 2023SaFormer: A Conditional Sequence Modeling Approach to Offline Safe Reinforcement Learningcited by 16
  2. 2024Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RLcited by 15
  3. 2024A Large Recurrent Action Model: xLSTM enables Fast Inference for Robotics Taskscited by 14
  4. 2025Generative Models in Decision Making: A Surveycited by 14
  5. 2023Instructed Diffuser with Temporal Condition Guidance for Offline Reinforcement Learningcited by 12
  6. 2023Transformers in Reinforcement Learning: A Surveycited by 11
  7. 2023Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learningcited by 10
  8. 2025Optimal Return-to-Go Guided Decision Transformer for Auto-Bidding in Advertisementcited by 4
  9. 2023VIPeR: Provably Efficient Algorithm for Offline RL with Neural Function Approximationcited by 3
  10. 2023Prioritized Trajectory Replay: A Replay Memory for Data-driven Reinforcement Learningcited by 3
  11. 2024Hierarchical Prompt Decision Transformer: Improving Few-Shot Policy Generalization with Global and Adaptive Guidancecited by 3
  12. 2025Analytic Energy-Guided Policy Optimization for Offline Reinforcement Learningcited by 2
  13. 2025Should We Ever Prefer Decision Transformer for Offline Reinforcement Learning?cited by 2
  14. 2023HIPODE: Enhancing Offline Reinforcement Learning with High-Quality Synthetic Data from a Policy-Decoupled Approachcited by 1
  15. 2023Multi-Objective Decision Transformers for Offline Reinforcement Learningcited by 1
  16. 2024Offline Trajectory Optimization for Offline Reinforcement Learningcited by 1
  17. 2025Online Optimization for Offline Safe Reinforcement Learningcited by 1
  18. 2026Online Finetuning Decision Transformers with Pure RL Gradientscited by 1
  19. 2024Is Offline Decision Making Possible with Only Few Samples? Reliable Decisions in Data-Starved Bandits via Trust Region Enhancementcited by 0
  20. 2024Diverse Randomized Value Functions: A Provably Pessimistic Approach for Offline Reinforcement Learningcited by 0
  21. 2024M3PC: Test-time Model Predictive Control for Pretrained Masked Trajectory Modelcited by 0
  22. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 0
  23. 2025Accelerating Transformers in Online RLcited by 0
  24. 2026Conditional Sequence Modeling for Safe Reinforcement Learningcited by 0
  25. 2026Pessimistic Auxiliary Policy for Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MAGE: Multi-scale Autoregressive Generation for Offline Reinforcement Learning
  2. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  3. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation
  4. 2021The Least Restriction for Offline Reinforcement Learning
  5. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  6. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  7. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  8. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL