# STORM: Efficient Stochastic Transformer based World Models for Reinforcement Learning

arXiv: https://arxiv.org/abs/2310.09615 · HTML: https://mlsanity.com/paper/2310.09615/

## Key facts

- Cited by 89 papers in the map (data through 14 Oct 2026).
- Placed in Latent dynamics models and model-based RL, one of 11 directions in World models.
- Builds on 29 of its 46 references that are in the map.
- First posted on arXiv 14 Oct 2023.

## Placed in

- World models / [Latent dynamics models and model-based RL](https://mlsanity.com/wm/latent-dynamics-models-and-model-based-rl/)

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [CURL: Contrastive Unsupervised Representations for Reinforcement Learning](https://mlsanity.com/paper/2004.04136/) (2020)
- [Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels](https://mlsanity.com/paper/2004.13649/) (2020)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [Transformers are Sample-Efficient World Models](https://mlsanity.com/paper/2209.00588/) (2022)
- [Data-Efficient Reinforcement Learning with Self-Predictive Representations](https://mlsanity.com/paper/2007.05929/) (2020)
- [Mastering Atari Games with Limited Data](https://mlsanity.com/paper/2111.00210/) (2021)
- [Bigger, Better, Faster: Human-level Atari with human-level efficiency](https://mlsanity.com/paper/2305.19452/) (2023)
- [TransDreamer: Reinforcement Learning with Transformer World Models](https://mlsanity.com/paper/2202.09481/) (2022)
- [Transformer-based World Models Are Happy With 100k Interactions](https://mlsanity.com/paper/2303.07109/) (2023)
- [When to use parametric models in reinforcement learning?](https://mlsanity.com/paper/1906.05243/) (2019)

## Built on by (most cited first, 30 of 89)

- [Diffusion for World Modeling: Visual Details Matter in Atari](https://mlsanity.com/paper/2405.12399/) (2024)
- [CycleResearcher: Improving Automated Research via Automated Review](https://mlsanity.com/paper/2411.00816/) (2024)
- [Training Agents Inside of Scalable World Models](https://mlsanity.com/paper/2509.24527/) (2025)
- [Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyond](https://mlsanity.com/paper/2405.03520/) (2024)
- [World Models for Autonomous Driving: An Initial Survey](https://mlsanity.com/paper/2403.02622/) (2024)
- [A Comprehensive Survey on World Models for Embodied AI](https://mlsanity.com/paper/2510.16732/) (2025)
- [Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Robotics](https://mlsanity.com/paper/2501.10100/) (2025)
- [World Action Models: The Next Frontier in Embodied AI](https://mlsanity.com/paper/2605.12090/) (2026)
- [NavMorph: A Self-Evolving World Model for Vision-and-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2506.23468/) (2025)
- [Pre-Trained Video Generative Models as World Simulators](https://mlsanity.com/paper/2502.07825/) (2025)
- [GWM: Towards Scalable Gaussian World Models for Robotic Manipulation](https://mlsanity.com/paper/2508.17600/) (2025)
- [Multi-agent Embodied AI: Advances and Future Directions](https://mlsanity.com/paper/2505.05108/) (2025)
- [Efficient World Models with Context-Aware Tokenization](https://mlsanity.com/paper/2406.19320/) (2024)
- [Learning Transformer-based World Models with Contrastive Predictive Coding](https://mlsanity.com/paper/2503.04416/) (2025)
- [StateSpaceDiffuser: Bringing Long Context to Diffusion World Models](https://mlsanity.com/paper/2505.22246/) (2025)
- [NeuralOS: Towards Simulating Operating Systems via Neural Generative Models](https://mlsanity.com/paper/2507.08800/) (2025)
- [GeoPredict: Leveraging Predictive Kinematics and 3D Gaussian Geometry for Precise VLA Manipulation](https://mlsanity.com/paper/2512.16811/) (2025)
- [Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modeling](https://mlsanity.com/paper/2603.04553/) (2026)
- [Open-World Reinforcement Learning over Long Short-Term Imagination](https://mlsanity.com/paper/2410.03618/) (2024)
- [Improving Transformer World Models for Data-Efficient RL](https://mlsanity.com/paper/2502.01591/) (2025)
- [Why LLMs Aren't Scientists Yet: Lessons from Four Autonomous Research Attempts](https://mlsanity.com/paper/2601.03315/) (2026)
- [UniZero: Generalized and Efficient Planning with Scalable Latent World Models](https://mlsanity.com/paper/2406.10667/) (2024)
- [Improving Token-Based World Models with Parallel Observation Prediction](https://mlsanity.com/paper/2402.05643/) (2024)
- [Object-Centric World Models from Few-Shot Annotations for Sample-Efficient Reinforcement Learning](https://mlsanity.com/paper/2501.16443/) (2025)
- [Accelerating Model-Based Reinforcement Learning with State-Space World Models](https://mlsanity.com/paper/2502.20168/) (2025)
- [R2-Dreamer: Redundancy-Reduced World Models without Decoders or Augmentation](https://mlsanity.com/paper/2603.18202/) (2026)
- [Discrete Codebook World Models for Continuous Control](https://mlsanity.com/paper/2503.00653/) (2025)
- [Generative World Modelling for Humanoids: 1X World Model Challenge Technical Report](https://mlsanity.com/paper/2510.07092/) (2025)
- [Masked Generative Priors Improve World Models Sequence Modelling Capabilities](https://mlsanity.com/paper/2410.07836/) (2024)
- [The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning](https://mlsanity.com/paper/2411.10175/) (2024)

## Abstract (arXiv)

Recently, model-based reinforcement learning algorithms have demonstrated remarkable efficacy in visual input environments. These approaches begin by constructing a parameterized simulation world model of the real environment through self-supervised learning. By leveraging the imagination of the world model, the agent's policy is enhanced without the constraints of sampling from the real environment. The performance of these algorithms heavily relies on the sequence modeling and generation capabilities of the world model. However, constructing a perfectly accurate model of a complex unknown environment is nearly impossible. Discrepancies between the model and reality may cause the agent to pursue virtual goals, resulting in subpar performance in the real environment. Introducing random noise into model-based reinforcement learning has been proven beneficial. In this work, we introduce Stochastic Transformer-based wORld Model (STORM), an efficient world model architecture that combines the strong sequence modeling and generation capabilities of Transformers with the stochastic nature of variational autoencoders. STORM achieves a mean human performance of 126.7% on the Atari 100k benchmark, setting a new record among state-of-the-art methods that do not employ lookahead search techniques. Moreover, training an agent with 1.85 hours of real-time interaction experience on a single NVIDIA GeForce RTX 3090 graphics card requires only 4.3 hours, showcasing improved efficiency compared to previous methodologies.