mlsanityData to 7 Oct 2026

Beyond Autoregressive RTG: Conditioning via Injection Outside Sequential Modeling in Decision Transformer

arXiv 2605.06104 ↗First posted 7 May 2026cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Decision Transformer (DT) formulates offline reinforcement learning as autoregressive sequence modeling, achieving promising results by predicting actions from a sequence of Return-to-Go (RTG), state, and action tokens. However, RTG is a scalar that summarizes future rewards, containing far less information than typical state or action vectors, yet it consumes the same computational budget per token. Worse, the self-attention cost of Transformers grows quadratically with sequence length, so including RTG as a separate token adds unnecessary overhead. We propose SlimDT, which removes RTG from the autoregressive sequence. Instead, we inject RTG information into the state representations before the sequential modeling step, allowing the Transformer to process only a compact (state, action) sequence. This reduces the sequence length by one-third, directly improving inference efficiency. On the D4RL benchmark, SlimDT surpasses standard DT across various tasks and achieves performance comparable to existing state-of-the-art methods. Decoupling a sparse conditioning signal from an information-rich sequence thus yields both computational gains and higher task performance.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 37 of its 42 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

37 of 42 references.

Closest ancestors · 16

Same direction, newest first

  1. 2026Decoupling Return-to-Go for Efficient Decision Transformer22 Jan
  2. 2026Online Finetuning Decision Transformers with Pure RL Gradients1 Jan
  3. 2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers31 Oct
  4. 2024Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RL8 Jun
  5. 2024Decision Mamba: Reinforcement Learning via Hybrid Selective Sequence Modeling31 May
  6. 2023Critic-Guided Decision Transformer for Offline Reinforcement Learning21 Dec
  7. 2023Rethinking Decision Transformer via Hierarchical Reinforcement Learning1 Nov
  8. 2023Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Making4 Oct
  9. 2023ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioning12 Sep
  10. 2023Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targets24 Jun
  11. 2022Dichotomy of Control: Separating What You Can Control from What You Cannot24 Oct
  12. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL8 Sep
4 more
  1. 2022You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environments31 May
  2. 2022Online Decision Transformer11 Feb
  3. 2021Generalized Decision Transformer for Offline Hindsight Information Matching19 Nov
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  3. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  4. 2022Scalable Diffusion Models with Transformerscited by 5,931
  5. 2016Layer Normalizationcited by 5,579
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  7. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  9. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  10. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  11. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  12. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
9 more
  1. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  2. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  3. 2021COMBO: Conservative Offline Model-Based Policy Optimizationcited by 365
  4. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  5. 2017Modulating early visual processing by languagecited by 167
  6. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
  7. 2021d3rlpy: An Offline Deep Reinforcement Learning Librarycited by 84
  8. 2023Elastic Decision Transformercited by 66
  9. 2021StARformer: Transformer with State-Action-Reward Representations for Visual Reinforcement Learningcited by 27

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Predictive Coding for Decision Transformer
  2. 2024Enhancing Decision Transformer with Diffusion-Based Trajectory Branch Generation
  3. 2026In-Context Reinforcement Learning From Suboptimal Historical Data
  4. 2024Q-value Regularized Transformer for Offline Reinforcement Learning
  5. 2023Future-conditioned Unsupervised Pretraining for Decision Transformer
  6. 2025Should We Ever Prefer Decision Transformer for Offline Reinforcement Learning?
  7. 2024Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformer
  8. 2025Adjusting the Output of Decision Transformer with Action Gradient