mlsanityData to 14 Oct 2026

Accelerating Transformers in Online RL

arXiv 2509.26137 ↗First posted 30 Sep 2025cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

The appearance of transformer-based models in Reinforcement Learning (RL) has expanded the horizons of possibilities in robotics tasks, but it has simultaneously brought a wide range of challenges during its implementation, especially in model-free online RL. Some of the existing learning algorithms cannot be easily implemented with transformer-based models due to the instability of the latter. In this paper, we propose a method that uses the Accelerator policy as a transformer's trainer. The Accelerator, a simpler and more stable model, interacts with the environment independently while simultaneously training the transformer through behavior cloning during the first stage of the proposed algorithm. In the second stage, the pretrained transformer starts to interact with the environment in a fully online setting. As a result, this model-free algorithm accelerates the transformer in terms of its performance and helps it to train online in a more stable and faster way. By conducting experiments on both state-based and image-based ManiSkill environments, as well as on MuJoCo tasks in MDP and POMDP settings, we show that applying our algorithm not only enables stable training of transformers but also reduces training time on image-based environments by up to a factor of two. Moreover, it decreases the required replay buffer size in off-policy methods to 10-20 thousand, which significantly lowers the overall computational demands.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 34 of its 42 references that are in the map. First posted on arXiv 30 Sep 2025.

Builds on

34 of 42 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025CTSAC: Curriculum-Based Transformer Soft Actor-Critic for Goal-Oriented Robot Exploration18 Mar

Other references · 33

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  9. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  10. 2022A Generalist Agentcited by 808
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  12. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
21 more
  1. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  2. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  3. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 229
  4. 2022Multi-Game Decision Transformerscited by 214
  5. 2022Recurrent Memory Transformercited by 203
  6. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 131
  7. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  8. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 79
  9. 2022Retrieval-Augmented Reinforcement Learningcited by 53
  10. 2023SMART: Self-supervised Multi-task pretrAining with contRol Transformerscited by 44
  11. 2024STRAP: Robot Sub-Trajectory Retrieval for Augmented Policy Learningcited by 44
  12. 2022Bootstrapped Transformer for Offline Reinforcement Learningcited by 37
  13. 2022Deep Transformer Q-Networks for Partially Observable Reinforcement Learningcited by 36
  14. 2021Efficient Transformers in Reinforcement Learning using Actor-Learner Distillationcited by 35
  15. 2024TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learningcited by 19
  16. 2023Recurrent Action Transformer with Memorycited by 17
  17. 2024Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Ratecited by 10
  18. 2024QT-TDM: Planning With Transformer Dynamics Model and Autoregressive Q-Learningcited by 4
  19. 2025RL + Transformer = A General-Purpose Problem Solvercited by 3
  20. 2024Offline-to-online Reinforcement Learning for Image-based Grasping with Scarce Demonstrationscited by 2
  21. 2025Should We Ever Prefer Decision Transformer for Offline Reinforcement Learning?cited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Transformers in Reinforcement Learning: A Survey
  2. 2025A New Perspective on Transformers in Online Reinforcement Learning for Continuous Control
  3. 2024Advances in Transformers for Robotic Applications: A Review
  4. 2022TransDreamer: Reinforcement Learning with Transformer World Models
  5. 2022How to Spend Your Robot Time: Bridging Kickstarting and Offline Reinforcement Learning for Vision-based Robotic Manipulation
  6. 2025Accelerating Model-Based Reinforcement Learning with State-Space World Models
  7. 2023Self-Confirming Transformer for Belief-Conditioned Adaptation in Offline Multi-Agent Reinforcement Learning
  8. 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretraining