mlsanityData to 7 Oct 2026

Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targets

arXiv 2306.14069 ↗First posted 24 Jun 2023cs.LGcited by 22

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Despite the recent advancements in offline reinforcement learning via supervised learning (RvS) and the success of the decision transformer (DT) architecture in various domains, DTs have fallen short in several challenging benchmarks. The root cause of this underperformance lies in their inability to seamlessly connect segments of suboptimal trajectories. To overcome this limitation, we present a novel approach to enhance RvS methods by integrating intermediate targets. We introduce the Waypoint Transformer (WT), using an architecture that builds upon the DT framework and conditioned on automatically-generated waypoints. The results show a significant increase in the final return compared to existing RvS methods, with performance on par or greater than existing state-of-the-art temporal difference learning-based methods. Additionally, the performance and stability improvements are largest in the most challenging environments and data configurations, including AntMaze Large Play/Diverse and Kitchen Mixed/Partial.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 49 references that are in the map. First posted on arXiv 24 Jun 2023.

Builds on

36 of 49 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL8 Sep
  2. 2022Prompting Decision Transformer for Few-Shot Policy Generalization27 Jun
  3. 2022Online Decision Transformer11 Feb
  4. 2021Generalized Decision Transformer for Offline Hindsight Information Matching19 Nov
  5. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem3 Jun
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2017Hindsight Experience Replaycited by 1,096
  9. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  10. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  11. 2018Image Transformercited by 770
  12. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
18 more
  1. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  2. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  3. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 184
  4. 2019Goal-conditioned Imitation Learningcited by 159
  5. 2021Offline RL Without Off-Policy Evaluationcited by 150
  6. 2020Model-Based Offline Planningcited by 126
  7. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124
  8. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  9. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  10. 2019Training Agents using Upside-Down Reinforcement Learningcited by 106
  11. 2019Reward-Conditioned Policiescited by 101
  12. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
  13. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 73
  14. 2020Rewriting History with Inverse RL: Hindsight Inference for Policy Improvementcited by 59
  15. 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
  16. 2021Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learningcited by 32
  17. 2021Model-Based Offline Planning with Trajectory Pruningcited by 30
  18. 2022Data-Efficient Pipeline for Offline Reinforcement Learning with Limited Datacited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 12; 2025: 2; 2026: 6.

22 papers.

Sort:

By area:

  1. 2024Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglementcited by 16
  2. 2024Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RLcited by 15
  3. 2023Rethinking Decision Transformer via Hierarchical Reinforcement Learningcited by 14
  4. 2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformerscited by 8
  5. 2024Predictive Coding for Decision Transformercited by 6
  6. 2024Understanding the Training and Generalization of Pretrained Transformer for Sequential Decision Makingcited by 4
  7. 2024Planning Transformer: Long-Horizon Offline Reinforcement Learning with Planning Tokenscited by 4
  8. 2024Unveiling Markov Heads in Pretrained Language Models for Offline Reinforcement Learningcited by 3
  9. 2023Self-supervised Pretraining for Decision Foundation Model: Formulation, Pipeline and Challengescited by 2
  10. 2024Combining Supervised Learning and Reinforcement Learning for Multi-Label Classification Tasks with Partial Labelscited by 2
  11. 2024Physics-informed Imitative Reinforcement Learning for Real-world Drivingcited by 2
  12. 2024Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learningcited by 2
10 more
  1. 2025Decision SpikeFormer: Spike-Driven Transformer for Decision Makingcited by 1
  2. 2026Decoupling Return-to-Go for Efficient Decision Transformercited by 1
  3. 2024Decision MetaMamba: Enhancing Selective SSM in Offline RL with Heterogeneous Sequence Mixingcited by 0
  4. 2024Adaptformer: Sequence models as adaptive iterative plannerscited by 0
  5. 2025Model-Based Offline Reinforcement Learning with Reliability-Guaranteed Sequence Modelingcited by 0
  6. 2026GAS: Enhancing Reward-Cost Balance of Generative Model-assisted Offline Safe RLcited by 0
  7. 2026Decision MetaMamba: Enhancing Selective SSM in Offline RL with Heterogeneous Sequence Mixingcited by 0
  8. 2026Beyond Autoregressive RTG: Conditioning via Injection Outside Sequential Modeling in Decision Transformercited by 0
  9. 2026Neuro-Symbolic Injection of LTLf Constraints in Autoregressive Reinforcement Learning Policiescited by 0
  10. 2026Improving Offline Goal-Conditioned Reinforcement Learning via Selective Reward Stimulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Critic-Guided Decision Transformer for Offline Reinforcement Learning
  2. 2024Enhancing Decision Transformer with Diffusion-Based Trajectory Branch Generation
  3. 2024Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learning
  4. 2025Should We Ever Prefer Decision Transformer for Offline Reinforcement Learning?
  5. 2026MAGE: Multi-scale Autoregressive Generation for Offline Reinforcement Learning
  6. 2022Bootstrapped Transformer for Offline Reinforcement Learning
  7. 2024Goal-Conditioned Decision Transformer for Multi-Goal Offline Reinforcement Learning
  8. 2022Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectories