mlsanityData to 14 Oct 2026

Rethinking Transformers in Solving POMDPs

arXiv 2405.17358 ↗First posted 27 May 2024cs.LG cs.AIcited by 9

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Sequential decision-making algorithms such as reinforcement learning (RL) in real-world scenarios inevitably face environments with partial observability. This paper scrutinizes the effectiveness of a popular architecture, namely Transformers, in Partially Observable Markov Decision Processes (POMDPs) and reveals its theoretical limitations. We establish that regular languages, which Transformers struggle to model, are reducible to POMDPs. This poses a significant challenge for Transformers in learning POMDP-specific inductive biases, due to their lack of inherent recurrence found in other models like RNNs. This paper casts doubt on the prevalent belief in Transformers as sequence models for RL and proposes to introduce a point-wise recurrent structure. The Deep Linear Recurrent Unit (LRU) emerges as a well-suited alternative for Partially Observable RL, with empirical results highlighting the sub-optimal performance of the Transformer and considerable strength of LRU.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 59 of its 82 references that are in the map. First posted on arXiv 27 May 2024.

Builds on

59 of 82 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Mastering Memory Tasks with World Models7 Mar

Other references · 58

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  9. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  10. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  11. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
38 more
  1. 2023Mastering Diverse Domains through World Modelscited by 1,253
  2. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  3. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  4. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  5. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  6. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  7. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 406
  8. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
  9. 2022Transformers are Sample-Efficient World Modelscited by 328
  10. 2023Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspectivecited by 294
  11. 2019Challenges of Real-World Reinforcement Learningcited by 291
  12. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 288
  13. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  14. 2017Gate-Variants of Gated Recurrent Unit (GRU) Neural Networkscited by 242
  15. 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
  16. 2022Neural Networks and the Chomsky Hierarchycited by 217
  17. 2022Multi-Game Decision Transformerscited by 214
  18. 2022Online Decision Transformercited by 196
  19. 2022The Parallelism Tradeoff: Limitations of Log-Precision Transformerscited by 193
  20. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 190
  21. 2022TransDreamer: Reinforcement Learning with Transformer World Modelscited by 143
  22. 2021Saturated Transformers are Constant-Depth Threshold Circuitscited by 139
  23. 2023Transformer-based World Models Are Happy With 100k Interactionscited by 139
  24. 2021Recurrent Model-Free RL Can Be a Strong Baseline for Many POMDPscited by 122
  25. 2023Randomized Positional Encodings Boost Length Generalization of Transformerscited by 117
  26. 2022Overcoming a Theoretical Limitation of Self-Attentioncited by 113
  27. 2023Structured State Space Models for In-Context Reinforcement Learningcited by 111
  28. 2022Block-Recurrent Transformerscited by 110
  29. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  30. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 89
  31. 2020On the Computational Power of Transformers and its Implications in Sequence Modelingcited by 85
  32. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 77
  33. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 63
  34. 2023POPGym: Benchmarking Partially Observable Reinforcement Learningcited by 62
  35. 2018Optimizing Agent Behavior over Long Time Scales by Transporting Valuecited by 58
  36. 2022Your Transformer May Not be as Powerful as You Expectcited by 55
  37. 2019Sequential Neural Networks as Automatacited by 54
  38. 2023Parallelizing non-linear sequential models over the sequence lengthcited by 45

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 2.

9 papers.

  1. 2025Agentic AI: A Comprehensive Survey of Architectures, Applications, and Future Directionscited by 48
  2. 2024TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learningcited by 19
  3. 2024Real-Time Recurrent Learning using Trace Units in Reinforcement Learningcited by 10
  4. 2024Recurrent Natural Policy Gradient for POMDPscited by 8
  5. 2025Towards Understanding Transformers in Learning Random Walkscited by 2
  6. 2025Long-Horizon Model-Based Offline Reinforcement Learning Without Explicit Conservatismcited by 1
  7. 2026Why Linear Recurrent Memory Works in Partially Observable Reinforcement Learningcited by 1
  8. 2025Bi-directional Recurrence Improves Transformer in Partially Observable Markov Decision Processescited by 0
  9. 2026Online Adaptive Reinforcement Learning with Echo State Networks for Non-Stationary Dynamicscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Transformers in Reinforcement Learning: A Survey
  2. 2025Synthetic POMDPs to Challenge Memory-Augmented RL: Memory Demand Structure Modeling
  3. 2026Does Learning Recurrent Networks Matter in Partially Observable RL?
  4. 2024Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Rate
  5. 2026Finite-State Controllers for (Hidden-Model) POMDPs using Deep Reinforcement Learning
  6. 2025Scalable Policy-Based RL Algorithms for POMDPs
  7. 2023Rethinking Decision Transformer via Hierarchical Reinforcement Learning
  8. 2025Inferring Reward Machines and Transition Machines from Partially Observable Markov Decision Processes