# Future-conditioned Unsupervised Pretraining for Decision Transformer

arXiv: https://arxiv.org/abs/2305.16683 · HTML: https://mlsanity.com/paper/2305.16683/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 42 of its 49 references that are in the map.
- First posted on arXiv 26 May 2023.

## Placed in

- Offline RL and imitation learning from data / [Decision transformers and sequence modeling](https://mlsanity.com/offlinerl/decision-transformers-and-sequence-modeling/)

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Large Batch Optimization for Deep Learning: Training BERT in 76 minutes](https://mlsanity.com/paper/1904.00962/) (2019)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning](https://mlsanity.com/paper/2208.06193/) (2022)
- [Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos](https://mlsanity.com/paper/2206.11795/) (2022)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [VAE with a VampPrior](https://mlsanity.com/paper/1705.07120/) (2017)
- [Accelerating Reinforcement Learning with Learned Skill Priors](https://mlsanity.com/paper/2010.11944/) (2020)
- [Multi-Game Decision Transformers](https://mlsanity.com/paper/2205.15241/) (2022)
- [Online Decision Transformer](https://mlsanity.com/paper/2202.05607/) (2022)
- [Efficient Exploration via State Marginal Matching](https://mlsanity.com/paper/1906.05274/) (2019)
- [RvS: What is Essential for Offline RL via Supervised Learning?](https://mlsanity.com/paper/2112.10751/) (2021)
- [OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning](https://mlsanity.com/paper/2010.13611/) (2020)

## Built on by (most cited first, 26 of 26)

- [GROOT: Learning to Follow Instructions by Watching Gameplay Videos](https://mlsanity.com/paper/2310.08235/) (2023)
- [Reinformer: Max-Return Sequence Modeling for Offline RL](https://mlsanity.com/paper/2405.08740/) (2024)
- [Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning](https://mlsanity.com/paper/2310.20587/) (2023)
- [Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directions](https://mlsanity.com/paper/2402.13777/) (2024)
- [Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglement](https://mlsanity.com/paper/2410.11448/) (2024)
- [Pre-training with Synthetic Data Helps Offline Reinforcement Learning](https://mlsanity.com/paper/2310.00771/) (2023)
- [Position: Foundation Agents as the Paradigm Shift for Decision Making](https://mlsanity.com/paper/2405.17009/) (2024)
- [Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers](https://mlsanity.com/paper/2410.24108/) (2024)
- [Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2407.04285/) (2024)
- [Open-World Skill Discovery from Unsegmented Demonstrations](https://mlsanity.com/paper/2503.10684/) (2025)
- [Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformer](https://mlsanity.com/paper/2408.01402/) (2024)
- [Predictive Coding for Decision Transformer](https://mlsanity.com/paper/2410.03408/) (2024)
- [Hindsight Preference Learning for Offline Preference-based Reinforcement Learning](https://mlsanity.com/paper/2407.04451/) (2024)
- [Unveiling Markov Heads in Pretrained Language Models for Offline Reinforcement Learning](https://mlsanity.com/paper/2409.06985/) (2024)
- [Learning Versatile Skills with Curriculum Masking](https://mlsanity.com/paper/2410.17744/) (2024)
- [Self-supervised Pretraining for Decision Foundation Model: Formulation, Pipeline and Challenges](https://mlsanity.com/paper/2401.00031/) (2023)
- [Enhancing Decision Transformer with Diffusion-Based Trajectory Branch Generation](https://mlsanity.com/paper/2411.11327/) (2024)
- [Prompt Tuning Decision Transformers with Structured and Scalable Bandits](https://mlsanity.com/paper/2502.04979/) (2025)
- [Beyond the Known: Decision Making with Counterfactual Reasoning Decision Transformer](https://mlsanity.com/paper/2505.09114/) (2025)
- [Online Finetuning Decision Transformers with Pure RL Gradients](https://mlsanity.com/paper/2601.00167/) (2026)
- [Model-Based Reinforcement Learning with Multi-Task Offline Pretraining](https://mlsanity.com/paper/2306.03360/) (2023)
- [CooT: Learning to Coordinate In-Context with Coordination Transformers](https://mlsanity.com/paper/2506.23549/) (2025)
- [Decision Transformer-Based Drone Trajectory Planning with Dynamic Safety-Efficiency Trade-Offs](https://mlsanity.com/paper/2507.21506/) (2025)
- [Adjusting the Output of Decision Transformer with Action Gradient](https://mlsanity.com/paper/2510.05285/) (2025)
- [PADiff: Predictive and Adaptive Diffusion Policies for Ad Hoc Teamwork](https://mlsanity.com/paper/2511.07260/) (2025)
- [Guided Riemannian Optimization (GuRO): Bridging Model Predictive Control and Decision Transformers](https://mlsanity.com/paper/2608.23204/) (2026)

## Abstract (arXiv)

Recent research in offline reinforcement learning (RL) has demonstrated that return-conditioned supervised learning is a powerful paradigm for decision-making problems. While promising, return conditioning is limited to training data labeled with rewards and therefore faces challenges in learning from unsupervised data. In this work, we aim to utilize generalized future conditioning to enable efficient unsupervised pretraining from reward-free and sub-optimal offline data. We propose Pretrained Decision Transformer (PDT), a conceptually simple approach for unsupervised RL pretraining. PDT leverages future trajectory information as a privileged context to predict actions during training. The ability to make decisions based on both present and future factors enhances PDT's capability for generalization. Besides, this feature can be easily incorporated into a return-conditioned framework for online finetuning, by assigning return values to possible futures and sampling future embeddings based on their respective values. Empirically, PDT outperforms or performs on par with its supervised pretraining counterpart, especially when dealing with sub-optimal data. Further analysis reveals that PDT can extract diverse behaviors from offline data and controllably sample high-return behaviors by online finetuning. Code is available at here.