mlsanityData to 7 Oct 2026

Deep Transformer Q-Networks for Partially Observable Reinforcement Learning

arXiv 2206.01078 ↗First posted 2 Jun 2022cs.LG cs.AIcited by 35

Not on the map. Why?

Abstract (arXiv)

Real-world reinforcement learning tasks often involve some form of partial observability where the observations only give a partial or noisy view of the true state of the world. Such tasks typically require some form of memory, where the agent has access to multiple past observations, in order to perform well. One popular way to incorporate memory is by using a recurrent neural network to access the agent's history. However, recurrent neural networks in reinforcement learning are often fragile and difficult to train, susceptible to catastrophic forgetting and sometimes fail completely as a result. In this work, we propose Deep Transformer Q-Networks (DTQN), a novel architecture utilizing transformers and self-attention to encode an agent's history. DTQN is designed modularly, and we compare results against several modifications to our base model. Our experiments demonstrate the transformer can solve partially observable tasks faster and more stably than previous recurrent approaches.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 41 of its 56 references that are in the map. First posted on arXiv 2 Jun 2022.

Builds on

41 of 56 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2016Layer Normalizationcited by 5,579
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  9. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  11. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  12. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
29 more
  1. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  2. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  3. 2019Attention is not Explanationcited by 916
  4. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  5. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  6. 2019Attention is not not Explanationcited by 629
  7. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  8. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 312
  9. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  10. 2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 259
  11. 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 202
  12. 2022Online Decision Transformercited by 194
  13. 2016ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learningcited by 178
  14. 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
  15. 2019Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actionscited by 124
  16. 2018Deep Variational Reinforcement Learning for POMDPscited by 122
  17. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 115
  18. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  19. 2016Playing FPS Games with Deep Reinforcement Learningcited by 95
  20. 2019Towards Interpretable Reinforcement Learning Using Attention Augmented Agentscited by 82
  21. 2020What Do Position Embeddings Learn? An Empirical Study of Pre-Trained Language Model Positional Encodingcited by 52
  22. 2018On Improving Deep Reinforcement Learning for POMDPscited by 49
  23. 2015Deep Attention Recurrent Q-Networkcited by 27
  24. 2020Discriminative Particle Filter Reinforcement Learning for Complex Partial Observationscited by 27
  25. 2016Learning to Communicate to Solve Riddles with Deep Distributed Recurrent Q-Networkscited by 21
  26. 2021Recurrent Off-policy Baselines for Memory-based Continuous Controlcited by 19
  27. 2020Robotic Table Tennis with Model-Free Reinforcement Learningcited by 13
  28. 2022Towards Flexible Inference in Sequential Decision Problems via Bidirectional Transformerscited by 6
  29. 2019Transformer Based Reinforcement Learning For Gamescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 10; 2024: 6; 2025: 13; 2026: 5.

35 papers.

Sort:

By area:

  1. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  2. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 62
  3. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  4. 2024An Introduction to Centralized Training for Decentralized Execution in Cooperative Multi-Agent Reinforcement Learningcited by 46
  5. 2024Mastering Memory Tasks with World Modelscited by 38
  6. 2025Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learningcited by 34
  7. 2023Recurrent Action Transformer with Memorycited by 17
  8. 2023Transformers in Reinforcement Learning: A Surveycited by 11
  9. 2025Partially Observable Reinforcement Learning with Memory Tracescited by 9
  10. 2025ELMUR: External Layer Memory with Update/Rewrite for Long-Horizon RL Problemscited by 6
  11. 2024Decision Transformer as a Foundation Model for Partially Observable Continuous Controlcited by 5
  12. 2024Unraveling the Complexity of Memory in RL Agents: an Approach for Classification and Evaluationcited by 5
23 more
  1. 2023Bridging Physics-Informed Neural Networks with Reinforcement Learning: Hamilton-Jacobi-Bellman Proximal Policy Optimization (HJBPPO)cited by 4
  2. 2025Optimal Return-to-Go Guided Decision Transformer for Auto-Bidding in Advertisementcited by 4
  3. 2023Actor-Critic Methods using Physics-Informed Neural Networks: Control of a 1D PDE Model for Fluid-Cooled Battery Packscited by 2
  4. 2023Automatic Deduction Path Learning via Reinforcement Learning with Environmental Correctioncited by 2
  5. 2022Decoding surface codes with deep reinforcement learning and probabilistic policy reusecited by 1
  6. 2025ReaCritic: Reasoning Transformer-based DRL Critic-model Scaling For Wireless Networkscited by 1
  7. 2026Memory Retention Is Not Enough to Master Memory Tasks in Reinforcement Learningcited by 1
  8. 2023RL3: Boosting Meta Reinforcement Learning via RL inside RL2cited by 0
  9. 2023Deep Attention Q-Network for Personalized Treatment Recommendationcited by 0
  10. 2024Transforming Game Play: A Comparative Study of DCQN and DTQN Architectures in Reinforcement Learningcited by 0
  11. 2024A transformer-based deep reinforcement learning approach to spatial navigation in a partially observable Morris Water Mazecited by 0
  12. 2025ColorDynamic: Generalizable, Scalable, Real-time, End-to-end Local Planner for Unstructured and Dynamic Environmentscited by 0
  13. 2025Stone Soup Multi-Target Tracking Feature Extraction For Autonomous Search And Track In Deep Reinforcement Learning Environmentcited by 0
  14. 2025BEAC: Imitating Complex Exploration and Task-oriented Behaviors for Invisible Object Nonprehensile Manipulationcited by 0
  15. 2025Do We Need Transformers to Play FPS Video Games?cited by 0
  16. 2025Bi-directional Recurrence Improves Transformer in Partially Observable Markov Decision Processescited by 0
  17. 2025SAINT: Attention-Based Policies for Discrete Combinatorial Action Spacescited by 0
  18. 2025Accelerating Transformers in Online RLcited by 0
  19. 2025A New Perspective on Transformers in Online Reinforcement Learning for Continuous Controlcited by 0
  20. 2026Short-Term-to-Long-Term Memory Transfer for Knowledge Graphs under Partial Observabilitycited by 0
  21. 2026Transformer-Enhanced Reinforcement Learning: Fundamentals and Applications in Communication Networkscited by 0
  22. 2026V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Controlcited by 0
  23. 2026ALER: Adaptive Learnable Experience Rewriting for Reinforcement Learningcited by 0