mlsanityData to 7 Oct 2026

Planning Transformer: Long-Horizon Offline Reinforcement Learning with Planning Tokens

arXiv 2409.09513 ↗First posted 14 Sep 2024cs.LG cs.AI cs.CLcited by 4

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Supervised learning approaches to offline reinforcement learning, particularly those utilizing the Decision Transformer, have shown effectiveness in continuous environments and for sparse rewards. However, they often struggle with long-horizon tasks due to the high compounding error of auto-regressive models. To overcome this limitation, we go beyond next-token prediction and introduce Planning Tokens, which contain high-level, long time-scale information about the agent's future. Predicting dual time-scale tokens at regular intervals enables our model to use these long-horizon Planning Tokens as a form of implicit planning to guide its low-level policy and reduce compounding error. This architectural modification significantly enhances performance on long-horizon tasks, establishing a new state-of-the-art in complex D4RL environments. Additionally, we demonstrate that Planning Tokens improve the interpretability of the model's policy through the interpretable plan visualisations and attention map.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 31 of its 49 references that are in the map. First posted on arXiv 14 Sep 2024.

Builds on

31 of 49 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targets24 Jun
  2. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem3 Jun
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 28

  1. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  5. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  7. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  8. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  9. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  10. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  11. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  12. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
16 more
  1. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  2. 2016The Option-Critic Architecturecited by 433
  3. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  4. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  5. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 328
  6. 2022Transformers are Sample-Efficient World Modelscited by 317
  7. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  8. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  9. 2021Discovering and Achieving Goals via World Modelscited by 131
  10. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 85
  11. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  12. 2022Efficient Planning in a Compact Latent Action Spacecited by 57
  13. 2018Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddingscited by 52
  14. 2019Combating the Compounding-Error Problem with a Multi-step Modelcited by 52
  15. 2021Vector Quantized Models for Planningcited by 47
  16. 2016Strategic Attentive Writer for Learning Macro-Actionscited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025DMWM: Dual-Mind World Model with Long-Term Imaginationcited by 10
  2. 2025CHD: Coupled Hierarchical Diffusion for Long-Horizon Taskscited by 6
  3. 2025Generative Sequential Notification Optimization via Multi-Objective Decision Transformerscited by 0
  4. 2026Decision Titan: Test-Time Training for Long-Term Memory in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Unified token representations for sequential decision models
  2. 2024Latent Plan Transformer for Trajectory Abstraction: Planning as Latent Space Inference
  3. 2026In-Context Planning with Latent Temporal Abstractions
  4. 2021Generalized Decision Transformer for Offline Hindsight Information Matching
  5. 2023Multi-Objective Decision Transformers for Offline Reinforcement Learning
  6. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learning
  7. 2026IPD: Boosting Sequential Policy with Imaginary Planning Distillation in Offline Reinforcement Learning
  8. 2026The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation