mlsanityData to 7 Oct 2026

Predictive Coding for Decision Transformer

arXiv 2410.03408 ↗First posted 4 Oct 2024cs.LGcited by 6

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Recent work in offline reinforcement learning (RL) has demonstrated the effectiveness of formulating decision-making as return-conditioned supervised learning. Notably, the decision transformer (DT) architecture has shown promise across various domains. However, despite its initial success, DTs have underperformed on several challenging datasets in goal-conditioned RL. This limitation stems from the inefficiency of return conditioning for guiding policy learning, particularly in unstructured and suboptimal datasets, resulting in DTs failing to effectively learn temporal compositionality. Moreover, this problem might be further exacerbated in long-horizon sparse-reward tasks. To address this challenge, we propose the Predictive Coding for Decision Transformer (PCDT) framework, which leverages generalized future conditioning to enhance DT methods. PCDT utilizes an architecture that extends the DT framework, conditioned on predictive codings, enabling decision-making based on both past and future factors, thereby improving generalization. Through extensive experiments on eight datasets from the AntMaze and FrankaKitchen environments, our proposed method achieves performance on par with or surpassing existing popular value-based and transformer-based methods in offline goal-conditioned RL. Furthermore, we also evaluate our method on a goal-reaching task with a physical robot.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 43 of its 54 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

43 of 54 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targets24 Jun
  2. 2023Future-conditioned Unsupervised Pretraining for Decision Transformer26 May
  3. 2023Masked Trajectory Models for Prediction, Representation, and Control4 May
  4. 2022Masked Autoencoding for Scalable and Generalizable Decision Making23 Nov
  5. 2022Dichotomy of Control: Separating What You Can Control from What You Cannot24 Oct
  6. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL8 Sep
  7. 2022Prompting Decision Transformer for Few-Shot Policy Generalization27 Jun
  8. 2022Online Decision Transformer11 Feb
  9. 2021Generalized Decision Transformer for Offline Hindsight Information Matching19 Nov
  10. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem3 Jun
  11. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  10. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  11. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  12. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
20 more
  1. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  2. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  3. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  4. 2017End-to-end Driving via Conditional Imitation Learningcited by 479
  5. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  6. 2020Attention is All You Need in Speech Separationcited by 184
  7. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  8. 2019Goal-conditioned Imitation Learningcited by 159
  9. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  10. 2019Training Agents using Upside-Down Reinforcement Learningcited by 106
  11. 2019Reward-Conditioned Policiescited by 101
  12. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
  13. 2019Urban Driving with Conditional Imitation Learningcited by 84
  14. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 73
  15. 2023Reinforcement Learning from Passive Data via Latent Intentionscited by 64
  16. 2017Data-efficient Deep Reinforcement Learning for Dexterous Manipulationcited by 61
  17. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
  18. 2022Efficient Planning in a Compact Latent Action Spacecited by 57
  19. 2022UniMASK: Unified Inference in Sequential Decision Problemscited by 23
  20. 2021Hindsight Goal Ranking on Replay Buffer for Sparse Reward Environmentcited by 7

Built on by

Cited by year

Citing papers per year: 2025: 6.

6 papers.

  1. 2025TARO: Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning for Synchronized Video-to-Audio Synthesiscited by 10
  2. 2025Policy Learning from Large Vision-Language Model Feedback without Reward Modelingcited by 4
  3. 2025Sample Efficient Reinforcement Learning via Large Vision Language Model Distillationcited by 3
  4. 2025Occlusion-robust Stylization for Drawing-based 3D Animationcited by 3
  5. 2025Reward Generation via Large Vision-Language Model in Offline Reinforcement Learningcited by 2
  6. 2025Accelerating Multi-modal LLM Gaming Performance via Input Prediction and Mishit Correctioncited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2023Critic-Guided Decision Transformer for Offline Reinforcement Learning
  2. 2024Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learning
  3. 2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers
  4. 2025Adjusting the Output of Decision Transformer with Action Gradient
  5. 2026Beyond Autoregressive RTG: Conditioning via Injection Outside Sequential Modeling in Decision Transformer
  6. 2025Should We Ever Prefer Decision Transformer for Offline Reinforcement Learning?
  7. 2023Rethinking Decision Transformer via Hierarchical Reinforcement Learning
  8. 2026Online Finetuning Decision Transformers with Pure RL Gradients