mlsanityData to 7 Oct 2026

A New Perspective on Transformers in Online Reinforcement Learning for Continuous Control

arXiv 2510.13367 ↗First posted 15 Oct 2025cs.LG cs.AI cs.ROcited by 0

Not on the map. Why?

Abstract (arXiv)

Despite their effectiveness and popularity in offline or model-based reinforcement learning (RL), transformers remain underexplored in online model-free RL due to their sensitivity to training setups and model design decisions such as how to structure the policy and value networks, share components, or handle temporal information. In this paper, we show that transformers can be strong baselines for continuous control in online model-free RL. We investigate key design questions: how to condition inputs, share components between actor and critic, and slice sequential data for training. Our experiments reveal stable architectural and training strategies enabling competitive performance across fully and partially observable tasks, and in both vector- and image-based settings. These findings offer practical guidance for applying transformers in online RL.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 25 of its 42 references that are in the map. First posted on arXiv 15 Oct 2025.

Builds on

25 of 42 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  8. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  10. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  11. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  12. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
13 more
  1. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  2. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  3. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 216
  4. 2021Causal Attention for Vision-Language Taskscited by 111
  5. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
  6. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 62
  7. 2019Variational Recurrent Models for Solving Partially Observable Control Taskscited by 46
  8. 2023SMART: Self-supervised Multi-task pretrAining with contRol Transformerscited by 44
  9. 2021Towards mental time travel: a hierarchical memory for reinforcement learning agentscited by 38
  10. 2022Deep Transformer Q-Networks for Partially Observable Reinforcement Learningcited by 35
  11. 2024Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Ratecited by 10
  12. 2025Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learningcited by 7
  13. 2024ReLIC: A Recipe for 64k Steps of In-Context Reinforcement Learning for Embodied AIcited by 3

Built on by

Not cited yet.