mlsanityData to 7 Oct 2026

TransDreamer: Reinforcement Learning with Transformer World Models

arXiv 2202.09481 ↗First posted 19 Feb 2022cs.LGcited by 143

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

The Dreamer agent provides various benefits of Model-Based Reinforcement Learning (MBRL) such as sample efficiency, reusable knowledge, and safe planning. However, its world model and policy networks inherit the limitations of recurrent neural networks and thus an important question is how an MBRL framework can benefit from the recent advances of transformers and what the challenges are in doing so. In this paper, we propose a transformer-based MBRL agent, called TransDreamer. We first introduce the Transformer State-Space Model, a world model that leverages a transformer for dynamics predictions. We then share this world model with a transformer-based policy network and obtain stability in training a transformer-based RL agent. In experiments, we apply the proposed model to 2D visual RL and 3D first-person visual RL tasks both requiring long-range memory access for memory-based reasoning. We show that the proposed model outperforms Dreamer in these complex tasks.

Key facts. Cited by 143 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 39 of its 54 references that are in the map. First posted on arXiv 19 Feb 2022.

Builds on

39 of 54 references.

Closest ancestors · 4

Same direction, newest first

  1. 2020Mastering Atari with Discrete World Models5 Oct
  2. 2020Planning to Explore via Self-Supervised World Models12 May
  3. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec
  4. 2018World Models27 Mar

Other references · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  6. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  9. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  10. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  11. 2018DeepMind Control Suitecited by 935
  12. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
23 more
  1. 2018Image Transformercited by 770
  2. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  3. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 582
  4. 2019Model-Based Reinforcement Learning for Ataricited by 495
  5. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  6. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  7. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  8. 2018Unity: A General Platform for Intelligent Agentscited by 232
  9. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 193
  10. 2019Scaling Autoregressive Video Modelscited by 189
  11. 2021Variational Transformer Networks for Layout Generationcited by 88
  12. 2021Going Beyond Linear Transformers with Recurrent Fast Weight Programmerscited by 84
  13. 2019Meta reinforcement learning as task inferencecited by 75
  14. 2020Reset-Free Lifelong Learning with Skill-Space Planningcited by 38
  15. 2020Variational Transformers for Diverse Response Generationcited by 36
  16. 2021Efficient Transformers in Reinforcement Learning using Actor-Learner Distillationcited by 35
  17. 2021Stochastic Transformer Networks with Linear Competing Units: Application to end-to-end SL Translationcited by 35
  18. 2018Sparse Attentive Backtracking: Temporal CreditAssignment Through Remindingcited by 33
  19. 2021Generative Video Transformer: Can Objects be the Words?cited by 24
  20. 2020Rapid Task-Solving in Novel Environmentscited by 23
  21. 2020MEMO: A Deep Network for Flexible Combination of Episodic Memoriescited by 21
  22. 2021Unsupervised Object-Based Transition Models for 3D Partially Observable Environmentscited by 19
  23. 2020The Monte Carlo Transformer: a stochastic self-attention model for sequence predictioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 12; 2023: 29; 2024: 32; 2025: 36; 2026: 34.

143 papers.

Sort:

By area:

  1. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  2. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  3. 2023Learning Interactive Real-World Simulatorscited by 440
  4. 2022Transformers are Sample-Efficient World Modelscited by 317
  5. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  6. 2024CycleResearcher: Improving Automated Research via Automated Reviewcited by 142
  7. 2023Transformer-based World Models Are Happy With 100k Interactionscited by 137
  8. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  9. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  10. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  11. 2025TesserAct: Learning 4D Embodied World Modelscited by 90
  12. 2023STORM: Efficient Stochastic Transformer based World Models for Reinforcement Learningcited by 88
38 more
  1. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 83
  2. 2024World Models for Autonomous Driving: An Initial Surveycited by 79
  3. 2025RynnVLA-002: A Unified Vision-Language-Action and World Modelcited by 70
  4. 2023SafeDiffuser: Safe Planning with Diffusion Probabilistic Modelscited by 69
  5. 2023Learning to Model the World with Languagecited by 66
  6. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  7. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 62
  8. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  9. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  10. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  11. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
  12. 2023SMART: Self-supervised Multi-task pretrAining with contRol Transformerscited by 44
  13. 2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39
  14. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  15. 2023World Models via Policy-Guided Trajectory Diffusioncited by 34
  16. 2024Efficient World Models with Context-Aware Tokenizationcited by 33
  17. 2024The Essential Role of Causality in Foundation World Models for Embodied AIcited by 32
  18. 2023Facing Off World Model Backbones: RNNs, Transformers, and S4cited by 31
  19. 2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learningcited by 31
  20. 2025World Models for Learning Dexterous Hand-Object Interactions from Human Videoscited by 26
  21. 2024DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Modelcited by 25
  22. 2026World-Gymnast: Training Robots with Reinforcement Learning in a World Modelcited by 25
  23. 2022UniMASK: Unified Inference in Sequential Decision Problemscited by 23
  24. 2025Embodied AI: From LLMs to World Modelscited by 23
  25. 2026τ0-WM: A Unified Video-Action World Model for Robotic Manipulationcited by 21
  26. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  27. 2025Learning Transformer-based World Models with Contrastive Predictive Codingcited by 20
  28. 2023Large Sequence Models for Sequential Decision-Making: A Surveycited by 19
  29. 2023Recurrent Action Transformer with Memorycited by 17
  30. 2024Slot State Space Modelscited by 17
  31. 2023Curious Replay for Model-based Adaptationcited by 16
  32. 2025Chunking the Critic: A Transformer-based Soft Actor-Critic with N-Step Returnscited by 16
  33. 2025StateSpaceDiffuser: Bringing Long Context to Diffusion World Modelscited by 16
  34. 2022Unsupervised Learning of Temporal Abstractions with Slot-based Transformerscited by 14
  35. 2025Improving Transformer World Models for Data-Efficient RLcited by 13
  36. 2022Switch Trajectory Transformer with Distributional Value Approximation for Multi-Task Reinforcement Learningcited by 11
  37. 2023Transformers in Reinforcement Learning: A Surveycited by 11
  38. 2026When World Models Dream Wrong: Physical-Conditioned Adversarial Attacks against World Modelscited by 11

50 most cited of 143 All 143 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Emergent Agentic Transformer from Chain of Hindsight Experience
  2. 2021DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representations
  3. 2023Rethinking Decision Transformer via Hierarchical Reinforcement Learning
  4. 2024Meta-DT: Offline Meta-RL as Conditional Sequence Modeling with World Model Disentanglement
  5. 2026DreamFormer: Dream Imitation with a Transformer World Model for Language-Conditioned Robotic Manipulation
  6. 2025Decision SpikeFormer: Spike-Driven Transformer for Decision Making
  7. 2026Dreamer-CDP: Improving Reconstruction-free World Models Via Continuous Deterministic Representation Prediction
  8. 2022DayDreamer: World Models for Physical Robot Learning