mlsanityData to 7 Oct 2026

Transformer-based World Models Are Happy With 100k Interactions

arXiv 2303.07109 ↗First posted 13 Mar 2023cs.LG cs.AI stat.MLcited by 137

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Deep neural networks have been successful in many reinforcement learning settings. However, compared to human learners they are overly data hungry. To build a sample-efficient world model, we apply a transformer to real-world episodes in an autoregressive manner: not only the compact latent states and the taken actions but also the experienced or predicted rewards are fed into the transformer, so that it can attend flexibly to all three modalities at different time steps. The transformer allows our world model to access previous states directly, instead of viewing them through a compressed recurrent state. By utilizing the Transformer-XL architecture, it is able to learn long-term dependencies while staying computationally efficient. Our transformer-based world model (TWM) generates meaningful, new experience, which is used to train a policy that outperforms previous model-free and model-based reinforcement learning algorithms on the Atari 100k benchmark.

Key facts. Cited by 137 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 28 of its 37 references that are in the map. First posted on arXiv 13 Mar 2023.

Builds on

28 of 37 references.

Closest ancestors · 7

Same direction, newest first

  1. 2022Transformers are Sample-Efficient World Models1 Sep
  2. 2022TransDreamer: Reinforcement Learning with Transformer World Models19 Feb
  3. 2021Mastering Atari Games with Limited Data30 Oct
  4. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem3 Jun
  5. 2020Mastering Atari with Discrete World Models5 Oct
  6. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec
  7. 2018Recurrent World Models Facilitate Policy Evolution4 Sep

Other references · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2013Auto-Encoding Variational Bayescited by 3,980
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  9. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  10. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  11. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  12. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
9 more
  1. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  2. 2020Quantifying Attention Flow in Transformerscited by 759
  3. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  4. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  5. 2019Model-Based Reinforcement Learning for Ataricited by 495
  6. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  7. 2020Reinforcement Learning with Augmented Datacited by 452
  8. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  9. 2019When to use parametric models in reinforcement learning?cited by 118

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 17; 2024: 35; 2025: 55; 2026: 30.

137 papers.

Sort:

By area:

  1. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  2. 2024Genie: Generative Interactive Environmentscited by 706
  3. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 357
  4. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 285
  5. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  6. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  7. 2025Training Agents Inside of Scalable World Modelscited by 122
  8. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  9. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  10. 2023STORM: Efficient Stochastic Transformer based World Models for Reinforcement Learningcited by 88
  11. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 83
  12. 2024World Models for Autonomous Driving: An Initial Surveycited by 79
38 more
  1. 2025RynnVLA-002: A Unified Vision-Language-Action and World Modelcited by 70
  2. 2023Learning to Model the World with Languagecited by 66
  3. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  4. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 62
  5. 2025A Survey: Learning Embodied Intelligence from Physical Simulators and World Modelscited by 50
  6. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  7. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
  8. 2025Pre-Trained Video Generative Models as World Simulatorscited by 43
  9. 2025DynaGuide: Steering Diffusion Polices with Active Dynamic Guidancecited by 39
  10. 2024Mastering Memory Tasks with World Modelscited by 38
  11. 2025GWM: Towards Scalable Gaussian World Models for Robotic Manipulationcited by 37
  12. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  13. 2023World Models via Policy-Guided Trajectory Diffusioncited by 34
  14. 2024The Essential Role of Causality in Foundation World Models for Embodied AIcited by 32
  15. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 32
  16. 2026Temporal Straightening for Latent Planningcited by 32
  17. 2023Facing Off World Model Backbones: RNNs, Transformers, and S4cited by 31
  18. 2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learningcited by 31
  19. 2025World Models for Learning Dexterous Hand-Object Interactions from Human Videoscited by 26
  20. 2023A Generalist Dynamics Model for Controlcited by 24
  21. 2025Embodied AI: From LLMs to World Modelscited by 23
  22. 2023HarmonyDream: Task Harmonization Inside World Modelscited by 22
  23. 2025Learning Transformer-based World Models with Contrastive Predictive Codingcited by 20
  24. 2023Gradient-based Planning with World Modelscited by 18
  25. 2023Recurrent Action Transformer with Memorycited by 17
  26. 2025StateSpaceDiffuser: Bringing Long Context to Diffusion World Modelscited by 16
  27. 2025WorldWeaver: Generating Long-Horizon Video Worlds via Rich Perceptioncited by 14
  28. 2025Improving Transformer World Models for Data-Efficient RLcited by 13
  29. 2025Bootstrapped Model Predictive Controlcited by 13
  30. 2024Language-Guided World Models: A Model-Based Approach to AI Controlcited by 12
  31. 2024Hindsight PRIORs for Reward Learning from Human Preferencescited by 12
  32. 2024UniZero: Generalized and Efficient Planning with Scalable Latent World Modelscited by 12
  33. 2026WorldBench: Benchmarking Physical Understanding of World Models by Isolating Physics Conceptscited by 12
  34. 2023Transformers in Reinforcement Learning: A Surveycited by 11
  35. 2023ReCoRe: Regularized Contrastive Representation Learning of World Modelcited by 11
  36. 2024Improving Token-Based World Models with Parallel Observation Predictioncited by 11
  37. 2024Do Transformer World Models Give Better Policy Gradients?cited by 10
  38. 2024Decentralized Transformers with Centralized Aggregation are Sample-Efficient Multi-Agent World Modelscited by 10

50 most cited of 137 All 137 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2020Smaller World Models for Reinforcement Learning
  2. 2024Efficient World Models with Context-Aware Tokenization
  3. 2026AGI Maze Prediction Datasets: A Compact Benchmark for Learning World Dynamics with Transformers
  4. 2023Emergent Agentic Transformer from Chain of Hindsight Experience
  5. 2026Beyond Autoregressive RTG: Conditioning via Injection Outside Sequential Modeling in Decision Transformer
  6. 2026World Machine: Towards Generative World Modeling for Time-Series
  7. 2026In-Context Reinforcement Learning From Suboptimal Historical Data
  8. 2025RLVR-World: Training World Models with Reinforcement Learning