mlsanityData to 7 Oct 2026

Future-conditioned Unsupervised Pretraining for Decision Transformer

arXiv 2305.16683 ↗First posted 26 May 2023cs.LGcited by 26

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Recent research in offline reinforcement learning (RL) has demonstrated that return-conditioned supervised learning is a powerful paradigm for decision-making problems. While promising, return conditioning is limited to training data labeled with rewards and therefore faces challenges in learning from unsupervised data. In this work, we aim to utilize generalized future conditioning to enable efficient unsupervised pretraining from reward-free and sub-optimal offline data. We propose Pretrained Decision Transformer (PDT), a conceptually simple approach for unsupervised RL pretraining. PDT leverages future trajectory information as a privileged context to predict actions during training. The ability to make decisions based on both present and future factors enhances PDT's capability for generalization. Besides, this feature can be easily incorporated into a return-conditioned framework for online finetuning, by assigning return values to possible futures and sampling future embeddings based on their respective values. Empirically, PDT outperforms or performs on par with its supervised pretraining counterpart, especially when dealing with sub-optimal data. Further analysis reveals that PDT can extract diverse behaviors from offline data and controllably sample high-return behaviors by online finetuning. Code is available at here.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 42 of its 49 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

42 of 49 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Dichotomy of Control: Separating What You Can Control from What You Cannot24 Oct
  2. 2022Multi-Game Decision Transformers30 May
  3. 2022Online Decision Transformer11 Feb
  4. 2021Generalized Decision Transformer for Offline Hindsight Information Matching19 Nov
  5. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem3 Jun
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2017Neural Discrete Representation Learningcited by 4,650
  9. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  11. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
24 more
  1. 2017Hindsight Experience Replaycited by 1,096
  2. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  3. 2018Exploration by Random Network Distillationcited by 966
  4. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  5. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  6. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  7. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  8. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  9. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  10. 2017VAE with a VampPriorcited by 247
  11. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  12. 2019Efficient Exploration via State Marginal Matchingcited by 186
  13. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  14. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 164
  15. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  16. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124
  17. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  18. 2019Reward-Conditioned Policiescited by 101
  19. 2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 90
  20. 2021Conservative Data Sharing for Multi-Task Offline Reinforcement Learningcited by 64
  21. 2021Controllable and Compositional Generation with Latent-Space Energy-Based Modelscited by 64
  22. 2022Addressing Optimism Bias in Sequence Modeling for Reinforcement Learningcited by 22
  23. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  24. 2021Policy Gradients Incorporating the Futurecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 12; 2025: 7; 2026: 2.

26 papers.

Sort:

By area:

  1. 2023GROOT: Learning to Follow Instructions by Watching Gameplay Videoscited by 42
  2. 2024Reinformer: Max-Return Sequence Modeling for Offline RLcited by 25
  3. 2023Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learningcited by 24
  4. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  5. 2024Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglementcited by 16
  6. 2023Pre-training with Synthetic Data Helps Offline Reinforcement Learningcited by 9
  7. 2024Position: Foundation Agents as the Paradigm Shift for Decision Makingcited by 9
  8. 2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformerscited by 8
  9. 2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modelingcited by 7
  10. 2025Open-World Skill Discovery from Unsegmented Demonstrationscited by 7
  11. 2024Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformercited by 6
  12. 2024Predictive Coding for Decision Transformercited by 6
14 more
  1. 2024Hindsight Preference Learning for Offline Preference-based Reinforcement Learningcited by 3
  2. 2024Unveiling Markov Heads in Pretrained Language Models for Offline Reinforcement Learningcited by 3
  3. 2023Self-supervised Pretraining for Decision Foundation Model: Formulation, Pipeline and Challengescited by 2
  4. 2024Learning Versatile Skills with Curriculum Maskingcited by 2
  5. 2024Enhancing Decision Transformer with Diffusion-Based Trajectory Branch Generationcited by 2
  6. 2025Prompt Tuning Decision Transformers with Structured and Scalable Banditscited by 2
  7. 2025Beyond the Known: Decision Making with Counterfactual Reasoning Decision Transformercited by 1
  8. 2026Online Finetuning Decision Transformers with Pure RL Gradientscited by 1
  9. 2023Model-Based Reinforcement Learning with Multi-Task Offline Pretrainingcited by 0
  10. 2025CooT: Learning to Coordinate In-Context with Coordination Transformerscited by 0
  11. 2025Decision Transformer-Based Drone Trajectory Planning with Dynamic Safety-Efficiency Trade-Offscited by 0
  12. 2025Adjusting the Output of Decision Transformer with Action Gradientcited by 0
  13. 2025PADiff: Predictive and Adaptive Diffusion Policies for Ad Hoc Teamworkcited by 0
  14. 2026Guided Riemannian Optimization (GuRO): Bridging Model Predictive Control and Decision Transformerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Critic-Guided Decision Transformer for Offline Reinforcement Learning
  2. 2024Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learning
  3. 2024Foundation Policies with Hilbert Representations
  4. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining
  5. 2026Beyond Autoregressive RTG: Conditioning via Injection Outside Sequential Modeling in Decision Transformer
  6. 2024Unsupervised-to-Online Reinforcement Learning
  7. 2026In-Context Reinforcement Learning From Suboptimal Historical Data
  8. 2025Prompting Decision Transformers for Zero-Shot Reach-Avoid Policies