mlsanityData to 14 Oct 2026

Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers

arXiv 2410.24108 ↗First posted 31 Oct 2024cs.LG cs.AIcited by 8

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

also in 1 more
Abstract (arXiv)

Decision Transformers have recently emerged as a new and compelling paradigm for offline Reinforcement Learning (RL), completing a trajectory in an autoregressive way. While improvements have been made to overcome initial shortcomings, online finetuning of decision transformers has been surprisingly under-explored. The widely adopted state-of-the-art Online Decision Transformer (ODT) still struggles when pretrained with low-reward offline data. In this paper, we theoretically analyze the online-finetuning of the decision transformer, showing that the commonly used Return-To-Go (RTG) that's far from the expected return hampers the online fine-tuning process. This problem, however, is well-addressed by the value function and advantage of standard RL algorithms. As suggested by our analysis, in our experiments, we hence find that simply adding TD3 gradients to the finetuning process of ODT effectively improves the online finetuning performance of ODT, especially if ODT is pretrained with low-reward offline data. These findings provide new directions to further improve decision transformers.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 67 of its 76 references that are in the map. First posted on arXiv 31 Oct 2024.

Builds on

67 of 76 references.

Closest ancestors · 14

Same direction, newest first

  1. 2023Rethinking Decision Transformer via Hierarchical Reinforcement Learning1 Nov
  2. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining12 Oct
  3. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions18 Sep
  4. 2023Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targets24 Jun
  5. 2023Decision S4: Efficient Sequence-Based RL via State Spaces Layers8 Jun
  6. 2023Future-conditioned Unsupervised Pretraining for Decision Transformer26 May
  7. 2023Hyper-Decision Transformer for Efficient Online Policy Adaptation17 Apr
  8. 2023Skill Decision Transformer31 Jan
  9. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL8 Sep
  10. 2022You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environments31 May
  11. 2022Multi-Game Decision Transformers30 May
  12. 2022Online Decision Transformer11 Feb
2 more
  1. 2021Generalized Decision Transformer for Offline Hindsight Information Matching19 Nov
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 53

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2017Proximal Policy Optimization Algorithmscited by 15,215
  5. 2016Layer Normalizationcited by 5,669
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  7. 2015Continuous control with deep reinforcement learningcited by 3,407
  8. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  9. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  10. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
  11. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  12. 2016OpenAI Gymcited by 1,736
38 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  2. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  3. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  4. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  6. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  7. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  8. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  9. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  10. 2022Transformers are Sample-Efficient World Modelscited by 328
  11. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  12. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  13. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  14. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 185
  15. 2018Verifiable Reinforcement Learning via Policy Extractioncited by 177
  16. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  17. 2023Transformer-based World Models Are Happy With 100k Interactionscited by 139
  18. 2018Self-Imitation Learningcited by 138
  19. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 116
  20. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RLcited by 115
  21. 2022Jump-Start Reinforcement Learningcited by 112
  22. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  23. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learningcited by 104
  24. 2018Kickstarting Deep Reinforcement Learningcited by 81
  25. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 76
  26. 2022Supported Policy Optimization for Offline Reinforcement Learningcited by 74
  27. 2023Elastic Decision Transformercited by 67
  28. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 63
  29. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learningcited by 40
  30. 2023Tuning computer vision models with task rewardscited by 37
  31. 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
  32. 2021Offline Pre-trained Multi-Agent Decision Transformer: One Big Sequence Model Tackles All SMAC Taskscited by 32
  33. 2021StARformer: Transformer with State-Action-Reward Representations for Visual Reinforcement Learningcited by 27
  34. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
  35. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learningcited by 20
  36. 2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RLcited by 20
  37. 2023Learning from Visual Observation via Offline Pretrained State-to-Go Transformercited by 14
  38. 2023Semi-Offline Reinforcement Learning for Optimized Text Generationcited by 12

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 6.

8 papers.

  1. 2025Behavioral Exploration: Learning to Explore via In-Context Adaptationcited by 5
  2. 2026Online Finetuning Decision Transformers with Pure RL Gradientscited by 1
  3. 2026Decoupling Return-to-Go for Efficient Decision Transformercited by 1
  4. 2025Adjusting the Output of Decision Transformer with Action Gradientcited by 0
  5. 2026Beyond Autoregressive RTG: Conditioning via Injection Outside Sequential Modeling in Decision Transformercited by 0
  6. 2026Return-to-Go Is More Than a Number: Q-Guided Alignment for Return-Conditioned Supervised Learningcited by 0
  7. 2026Hybrid Offline-Online Multi-Agent Decision Transformers for Wireless Resource Managementcited by 0
  8. 2026Vision-Language-Action Autonomous Driving Agent with Language-based Memorycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Critic-Guided Decision Transformer for Offline Reinforcement Learning
  2. 2025A Comparison Between Decision Transformers and Traditional Offline Reinforcement Learning Algorithms
  3. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  4. 2024Predictive Coding for Decision Transformer
  5. 2023When should we prefer Decision Transformers for Offline Reinforcement Learning?
  6. 2026Active Offline-to-Online Reinforcement Learning
  7. 2025Should We Ever Prefer Decision Transformer for Offline Reinforcement Learning?
  8. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data