mlsanityData to 7 Oct 2026

Adjusting the Output of Decision Transformer with Action Gradient

arXiv 2510.05285 ↗First posted 6 Oct 2025cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Decision Transformer (DT), which integrates reinforcement learning (RL) with the transformer model, introduces a novel approach to offline RL. Unlike classical algorithms that take maximizing cumulative discounted rewards as objective, DT instead maximizes the likelihood of actions. This paradigm shift, however, presents two key challenges: stitching trajectories and extrapolation of action. Existing methods, such as substituting specific tokens with predictive values and integrating the Policy Gradient (PG) method, address these challenges individually but fail to improve performance stably when combined due to inherent instability. To address this, we propose Action Gradient (AG), an innovative methodology that directly adjusts actions to fulfill a function analogous to that of PG, while also facilitating efficient integration with token prediction techniques. AG utilizes the gradient of the Q-value with respect to the action to optimize the action. The empirical results demonstrate that our method can significantly enhance the performance of DT-based algorithms, with some results achieving state-of-the-art levels.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 30 of its 39 references that are in the map. First posted on arXiv 6 Oct 2025.

Builds on

30 of 39 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers31 Oct
  2. 2024In-Context Decision Transformer: Reinforcement Learning via Hierarchical Chain-of-Thought31 May
  3. 2024Q-value Regularized Transformer for Offline Reinforcement Learning27 May
  4. 2024Reinformer: Max-Return Sequence Modeling for Offline RL14 May
  5. 2023Critic-Guided Decision Transformer for Offline Reinforcement Learning21 Dec
  6. 2023Rethinking Decision Transformer via Hierarchical Reinforcement Learning1 Nov
  7. 2023Future-conditioned Unsupervised Pretraining for Decision Transformer26 May
  8. 2023Graph Decision Transformer7 Mar
  9. 2022Hierarchical Decision Transformer21 Sep
  10. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL8 Sep
  11. 2022Online Decision Transformer11 Feb
  12. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem3 Jun
1 more
  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  5. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  7. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  8. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  9. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  10. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  11. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
  12. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
5 more
  1. 2018Deep Reinforcement Learning and the Deadly Triadcited by 170
  2. 2021Offline RL Without Off-Policy Evaluationcited by 150
  3. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 117
  4. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 73
  5. 2023Elastic Decision Transformercited by 66

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Should We Ever Prefer Decision Transformer for Offline Reinforcement Learning?
  2. 2024Predictive Coding for Decision Transformer
  3. 2025A Comparison Between Decision Transformers and Traditional Offline Reinforcement Learning Algorithms
  4. 2023ACT: Empowering Decision Transformer with Dynamic Programming via Advantage Conditioning
  5. 2026Online Finetuning Decision Transformers with Pure RL Gradients
  6. 2024Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformer
  7. 2024Enhancing Decision Transformer with Diffusion-Based Trajectory Branch Generation
  8. 2023Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Making