# Transformer-based World Models Are Happy With 100k Interactions

arXiv: https://arxiv.org/abs/2303.07109 · HTML: https://mlsanity.com/paper/2303.07109/

## Key facts

- Cited by 139 papers in the map (data through 14 Oct 2026).
- Placed in Latent dynamics models and model-based RL, one of 11 directions in World models.
- Builds on 28 of its 37 references that are in the map.
- First posted on arXiv 13 Mar 2023.

## Placed in

- World models / [Latent dynamics models and model-based RL](https://mlsanity.com/wm/latent-dynamics-models-and-model-based-rl/)

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [Reinforcement Learning with Augmented Data](https://mlsanity.com/paper/2004.14990/) (2020)
- [Regularizing Neural Networks by Penalizing Confident Output Distributions](https://mlsanity.com/paper/1701.06548/) (2017)
- [Transformers are Sample-Efficient World Models](https://mlsanity.com/paper/2209.00588/) (2022)
- [Data-Efficient Reinforcement Learning with Self-Predictive Representations](https://mlsanity.com/paper/2007.05929/) (2020)
- [Mastering Atari Games with Limited Data](https://mlsanity.com/paper/2111.00210/) (2021)
- [TransDreamer: Reinforcement Learning with Transformer World Models](https://mlsanity.com/paper/2202.09481/) (2022)
- [When to use parametric models in reinforcement learning?](https://mlsanity.com/paper/1906.05243/) (2019)

## Built on by (most cited first, 30 of 139)

- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [Genie: Generative Interactive Environments](https://mlsanity.com/paper/2402.15391/) (2024)
- [DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planning](https://mlsanity.com/paper/2411.04983/) (2024)
- [Diffusion for World Modeling: Visual Details Matter in Atari](https://mlsanity.com/paper/2405.12399/) (2024)
- [A Survey on Vision-Language-Action Models for Embodied AI](https://mlsanity.com/paper/2405.14093/) (2024)
- [Bigger, Better, Faster: Human-level Atari with human-level efficiency](https://mlsanity.com/paper/2305.19452/) (2023)
- [Training Agents Inside of Scalable World Models](https://mlsanity.com/paper/2509.24527/) (2025)
- [Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods](https://mlsanity.com/paper/2404.00282/) (2024)
- [Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond](https://mlsanity.com/paper/2405.03520/) (2024)
- [STORM: Efficient Stochastic Transformer based World Models for Reinforcement Learning](https://mlsanity.com/paper/2310.09615/) (2023)
- [Pandora: Towards General World Model with Natural Language Actions and Video States](https://mlsanity.com/paper/2406.09455/) (2024)
- [World Models for Autonomous Driving: An Initial Survey](https://mlsanity.com/paper/2403.02622/) (2024)
- [RynnVLA-002: A Unified Vision-Language-Action and World Model](https://mlsanity.com/paper/2511.17502/) (2025)
- [Learning to Model the World with Language](https://mlsanity.com/paper/2308.01399/) (2023)
- [A Comprehensive Survey on World Models for Embodied AI](https://mlsanity.com/paper/2510.16732/) (2025)
- [When Do Transformers Shine in RL? Decoupling Memory from Credit Assignment](https://mlsanity.com/paper/2307.03864/) (2023)
- [A Survey: Learning Embodied Intelligence from Physical Simulators and World Models](https://mlsanity.com/paper/2507.00917/) (2025)
- [World Action Models: The Next Frontier in Embodied AI](https://mlsanity.com/paper/2605.12090/) (2026)
- [Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learning](https://mlsanity.com/paper/2305.18499/) (2023)
- [Pre-Trained Video Generative Models as World Simulators](https://mlsanity.com/paper/2502.07825/) (2025)
- [DynaGuide: Steering Diffusion Polices with Active Dynamic Guidance](https://mlsanity.com/paper/2506.13922/) (2025)
- [GWM: Towards Scalable Gaussian World Models for Robotic Manipulation](https://mlsanity.com/paper/2508.17600/) (2025)
- [Mastering Memory Tasks with World Models](https://mlsanity.com/paper/2403.04253/) (2024)
- [Temporal Straightening for Latent Planning](https://mlsanity.com/paper/2603.12231/) (2026)
- [Beyond Language Modeling: An Exploration of Multimodal Pretraining](https://mlsanity.com/paper/2603.03276/) (2026)
- [Multi-agent Embodied AI: Advances and Future Directions](https://mlsanity.com/paper/2505.05108/) (2025)
- [World Models via Policy-Guided Trajectory Diffusion](https://mlsanity.com/paper/2312.08533/) (2023)
- [The Essential Role of Causality in Foundation World Models for Embodied AI](https://mlsanity.com/paper/2402.06665/) (2024)
- [Facing Off World Model Backbones: RNNs, Transformers, and S4](https://mlsanity.com/paper/2307.02064/) (2023)
- [Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learning](https://mlsanity.com/paper/2402.03570/) (2024)

## Abstract (arXiv)

Deep neural networks have been successful in many reinforcement learning settings. However, compared to human learners they are overly data hungry. To build a sample-efficient world model, we apply a transformer to real-world episodes in an autoregressive manner: not only the compact latent states and the taken actions but also the experienced or predicted rewards are fed into the transformer, so that it can attend flexibly to all three modalities at different time steps. The transformer allows our world model to access previous states directly, instead of viewing them through a compressed recurrent state. By utilizing the Transformer-XL architecture, it is able to learn long-term dependencies while staying computationally efficient. Our transformer-based world model (TWM) generates meaningful, new experience, which is used to train a policy that outperforms previous model-free and model-based reinforcement learning algorithms on the Atari 100k benchmark.