mlsanityData to 7 Oct 2026

Transformers are Sample-Efficient World Models

arXiv 2209.00588 ↗First posted 1 Sep 2022cs.LG cs.AI cs.CVcited by 317

On the map: World models → Latent dynamics models and model-based RL

Abstract (arXiv)

Deep reinforcement learning agents are notoriously sample inefficient, which considerably limits their application to real-world problems. Recently, many model-based methods have been designed to address this issue, with learning in the imagination of a world model being one of the most prominent approaches. However, while virtually unlimited interaction with a simulated environment sounds appealing, the world model has to be accurate over extended periods of time. Motivated by the success of Transformers in sequence modeling tasks, we introduce IRIS, a data-efficient agent that learns in a world model composed of a discrete autoencoder and an autoregressive Transformer. With the equivalent of only two hours of gameplay in the Atari 100k benchmark, IRIS achieves a mean human normalized score of 1.046, and outperforms humans on 10 out of 26 games, setting a new state of the art for methods without lookahead search. To foster future research on Transformers and world models for sample-efficient reinforcement learning, we release our code and models at https://github.com/eloialonso/iris.

Key facts. Cited by 317 papers in the map (data through 7 Oct 2026). Placed in Latent dynamics models and model-based RL, one of 11 directions in World models. Builds on 42 of its 64 references that are in the map. First posted on arXiv 1 Sep 2022.

Builds on

42 of 64 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022TransDreamer: Reinforcement Learning with Transformer World Models19 Feb
  2. 2021Mastering Atari Games with Limited Data30 Oct
  3. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem3 Jun
  4. 2020Mastering Atari with Discrete World Models5 Oct
  5. 2019Dream to Control: Learning Behaviors by Latent Imagination3 Dec
  6. 2018Recurrent World Models Facilitate Policy Evolution4 Sep

Other references · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2017Proximal Policy Optimization Algorithmscited by 14,441
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  8. 2017Neural Discrete Representation Learningcited by 4,650
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  12. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
24 more
  1. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  5. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  6. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  7. 2015Prioritized Experience Replaycited by 1,070
  8. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  9. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  10. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  11. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  12. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  13. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  14. 2019Model-Based Reinforcement Learning for Ataricited by 495
  15. 2015Autoencoding beyond pixels using a learned similarity metriccited by 407
  16. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  17. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  18. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  19. 2022Online Decision Transformercited by 194
  20. 2021Benchmarking the Spectrum of Agent Capabilitiescited by 193
  21. 2020Combining Deep Reinforcement Learning and Search for Imperfect-Information Gamescited by 110
  22. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  23. 2021Vector Quantized Models for Planningcited by 47
  24. 2022MineRL Diamond 2021 Competition: Overview, Results, and Lessons Learnedcited by 26

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 45; 2024: 72; 2025: 97; 2026: 98.

317 papers.

Sort:

By area:

  1. 2023Mastering Diverse Domains through World Modelscited by 1,193
  2. 2024Genie: Generative Interactive Environmentscited by 706
  3. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 618
  4. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  5. 2023Learning Interactive Real-World Simulatorscited by 440
  6. 2024DINO-WM: World Models on Pre-trained Visual Features enable Zero-shot Planningcited by 357
  7. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  8. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 285
  9. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  10. 2022Transformers Learn Shortcuts to Automatacited by 260
  11. 2026LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixelscited by 237
  12. 2025GAIA-2: A Controllable Multi-View Generative World Model for Autonomous Drivingcited by 178
38 more
  1. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  2. 2022SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Modelscited by 146
  3. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  4. 2023Transformer-based World Models Are Happy With 100k Interactionscited by 137
  5. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  6. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 122
  7. 2025Training Agents Inside of Scalable World Modelscited by 122
  8. 2023Learning to Act without Actionscited by 116
  9. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  10. 2024Is Sora a World Simulator? A Comprehensive Survey on General World Models and Beyondcited by 111
  11. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 108
  12. 2023Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusioncited by 105
  13. 2024WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environmentcited by 99
  14. 2025TesserAct: Learning 4D Embodied World Modelscited by 90
  15. 2023STORM: Efficient Stochastic Transformer based World Models for Reinforcement Learningcited by 88
  16. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 83
  17. 2024World Models for Autonomous Driving: An Initial Surveycited by 79
  18. 2025RynnVLA-002: A Unified Vision-Language-Action and World Modelcited by 70
  19. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  20. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 62
  21. 2025Long-Context State-Space Video World Modelscited by 59
  22. 2025Back to the Features: DINO as a Foundation for Video World Modelscited by 56
  23. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  24. 2025Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Roboticscited by 47
  25. 2026World Action Models: The Next Frontier in Embodied AIcited by 46
  26. 2023Pre-training Contextualized World Models with In-the-wild Videos for Reinforcement Learningcited by 45
  27. 2023SMART: Self-supervised Multi-task pretrAining with contRol Transformerscited by 44
  28. 2025Pre-Trained Video Generative Models as World Simulatorscited by 43
  29. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  30. 2024FLIP: Flow-Centric Generative Planning as General-Purpose Manipulation World Modelcited by 41
  31. 2024Autoregressive Models in Vision: A Surveycited by 39
  32. 2025DynaGuide: Steering Diffusion Polices with Active Dynamic Guidancecited by 39
  33. 2024Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Searchcited by 38
  34. 2025RLVR-World: Training World Models with Reinforcement Learningcited by 38
  35. 2025GWM: Towards Scalable Gaussian World Models for Robotic Manipulationcited by 37
  36. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  37. 2025PAN: A World Model for General, Actionable, and Long-Horizon World Simulationcited by 36
  38. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35

50 most cited of 317 All 317 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Multi-Game Decision Transformers
  2. 2020Smaller World Models for Reinforcement Learning
  3. 2023Emergent Agentic Transformer from Chain of Hindsight Experience
  4. 2026In-Context Reinforcement Learning From Suboptimal Historical Data
  5. 2026AutoWorldModel-Bench: A State-Centric Benchmark for Automated World-Model Research
  6. 2026Vintix II: Decision Pre-Trained Transformer is a Scalable In-Context Reinforcement Learner
  7. 2022Multi-Agent Reinforcement Learning is a Sequence Modeling Problem
  8. 2024Game On: Towards Language Models as RL Experimenters