mlsanityData to 7 Oct 2026

Transformer in Transformer as Backbone for Deep Reinforcement Learning

arXiv 2212.14538 ↗First posted 30 Dec 2022cs.LG cs.AI cs.ROcited by 6

Not on the map. Why?

Abstract (arXiv)

Designing better deep networks and better reinforcement learning (RL) algorithms are both important for deep RL. This work focuses on the former. Previous methods build the network with several modules like CNN, LSTM and Attention. Recent methods combine the Transformer with these modules for better performance. However, it requires tedious optimization skills to train a network composed of mixed modules, making these methods inconvenient to be used in practice. In this paper, we propose to design pure Transformer-based networks for deep RL, aiming at providing off-the-shelf backbones for both the online and offline settings. Specifically, the Transformer in Transformer (TIT) backbone is proposed, which cascades two Transformers in a very natural way: the inner one is used to process a single observation, while the outer one is responsible for processing the observation history; combining both is expected to extract spatial-temporal representations for good decision-making. Experiments show that TIT can achieve satisfactory performance in different settings consistently.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 45 of its 58 references that are in the map. First posted on arXiv 30 Dec 2022.

Builds on

45 of 58 references.

References · 45

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2017Proximal Policy Optimization Algorithmscited by 14,441
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2016Densely Connected Convolutional Networkscited by 6,691
  9. 2015Continuous control with deep reinforcement learningcited by 3,279
  10. 2015Trust Region Policy Optimizationcited by 2,503
  11. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  12. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
33 more
  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  2. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  3. 2021ViViT: A Video Vision Transformercited by 1,610
  4. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  9. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  10. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  11. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  12. 2021Transformer in Transformercited by 632
  13. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  14. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  15. 2022Transformers are Sample-Efficient World Modelscited by 317
  16. 2015LSTM: A Search Space Odysseycited by 315
  17. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  18. 2022Online Decision Transformercited by 194
  19. 2016Value Iteration Networkscited by 158
  20. 2022TransDreamer: Reinforcement Learning with Transformer World Modelscited by 143
  21. 2021RRL: Resnet as representation for Reinforcement Learningcited by 89
  22. 2020Dreaming: Model-based Reinforcement Learning by Latent Imagination without Reconstructioncited by 66
  23. 2022You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environmentscited by 62
  24. 2020D2RL: Deep Dense Architectures in Reinforcement Learningcited by 40
  25. 2022Transformers are Meta-Reinforcement Learnerscited by 39
  26. 2020Can Increasing Input Dimensionality Improve Deep Reinforcement Learning?cited by 37
  27. 2021Training Larger Networks for Deep Reinforcement Learningcited by 37
  28. 2022Cascade Transformers for End-to-End Person Searchcited by 31
  29. 2015Deep Attention Recurrent Q-Networkcited by 27
  30. 2021CoBERL: Contrastive BERT for Reinforcement Learningcited by 20
  31. 2022Evaluating Vision Transformer Methods for Deep Reinforcement Learning from Pixelscited by 12
  32. 2021DualFormer: Local-Global Stratified Transformer for Efficient Video Recognitioncited by 11
  33. 2022Pretraining the Vision Transformer using self-supervised methods for vision based Deep Reinforcement Learningcited by 5

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 1; 2025: 1.

6 papers.

  1. 2023Imitating Task and Motion Planning with Visuomotor Transformerscited by 63
  2. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  3. 2024Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformerscited by 8
  4. 2023PDiT: Interleaving Perception and Decision-making Transformers for Deep Reinforcement Learningcited by 5
  5. 2025NGENT: Next-Generation AI Agents Must Integrate Multi-Domain Abilities to Achieve Artificial General Intelligencecited by 2
  6. 2023Stackelberg Decision Transformer for Asynchronous Action Coordination in Multi-Agent Systemscited by 1