mlsanityData to 14 Oct 2026

The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control

arXiv 2412.03568 ↗First posted 4 Dec 2024cs.AIcited by 80

Not on the map. Why?

Cited by 43 in World models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present The Matrix, the first foundational realistic world simulator capable of generating continuous 720p high-fidelity real-scene video streams with real-time, responsive control in both first- and third-person perspectives, enabling immersive exploration of richly dynamic environments. Trained on limited supervised data from AAA games like Forza Horizon 5 and Cyberpunk 2077, complemented by large-scale unsupervised footage from real-world settings like Tokyo streets, The Matrix allows users to traverse diverse terrains -- deserts, grasslands, water bodies, and urban landscapes -- in continuous, uncut hour-long sequences. Operating at 16 FPS, the system supports real-time interactivity and demonstrates zero-shot generalization, translating virtual game environments to real-world contexts where collecting continuous movement data is often infeasible. For example, The Matrix can simulate a BMW X3 driving through an office setting--an environment present in neither gaming data nor real-world sources. This approach showcases the potential of AAA game data to advance robust world models, bridging the gap between simulations and real-world applications in scenarios with limited data.

Key facts. Cited by 80 papers in the map (data through 14 Oct 2026). Builds on 36 of its 52 references that are in the map. First posted on arXiv 4 Dec 2024.

Builds on

36 of 52 references.

References · 36

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  5. 2022Scalable Diffusion Models with Transformerscited by 6,054
  6. 2016Layer Normalizationcited by 5,669
  7. 2024GPT-4o System Cardcited by 4,588
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  9. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  11. 2023Consistency Modelscited by 1,822
  12. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
24 more
  1. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  2. 2023Mastering Diverse Domains through World Modelscited by 1,253
  3. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  4. 2020Mastering Atari with Discrete World Modelscited by 1,007
  5. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  6. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
  7. 2023Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inferencecited by 846
  8. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  9. 2024Genie: Generative Interactive Environmentscited by 747
  10. 2023GAIA-1: A Generative World Model for Autonomous Drivingcited by 637
  11. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  12. 2023Learning Interactive Real-World Simulatorscited by 458
  13. 2023Improved Techniques for Training Consistency Modelscited by 439
  14. 2022DayDreamer: World Models for Physical Robot Learningcited by 400
  15. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  16. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 300
  17. 2024Diffusion Models Are Real-Time Game Enginescited by 296
  18. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  19. 2024GameGen-X: Interactive Open-world Game Video Generationcited by 128
  20. 2024Pandora: Towards General World Model with Natural Language Actions and Video Statescited by 86
  21. 2023StreamDiffusion: A Pipeline-level Solution for Real-time Interactive Generationcited by 58
  22. 2015On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Modelscited by 55
  23. 2023SafeDreamer: Safe Reinforcement Learning with World Modelscited by 54
  24. 2024MuDreamer: Learning Predictive World Models without Reconstructioncited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 44; 2026: 36.

80 papers.

Sort:

By area:

  1. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  2. 2025LongLive: Real-time Interactive Long Video Generationcited by 265
  3. 2025WorldMem: Long-term Consistent World Simulation with Memorycited by 168
  4. 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrievalcited by 163
  5. 2025Video World Models with Long-term Spatial Memorycited by 136
  6. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 130
  7. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 114
  8. 2026Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memorycited by 100
  9. 2025CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Modelscited by 98
  10. 2025Hunyuan-GameCraft: High-dynamic Interactive Game Video Generation with Hybrid History Conditioncited by 98
  11. 2025Aether: Geometric-Aware Unified World Modelingcited by 97
  12. 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modelingcited by 96
38 more
  1. 2025RELIC: Interactive Video World Model with Long-Horizon Memorycited by 96
  2. 2025HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixelscited by 94
  3. 2025Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generationcited by 90
  4. 2025Reward Forcing: Efficient Streaming Video Generation with Rewarded Distribution Matching Distillationcited by 86
  5. 2025Long-Context State-Space Video World Modelscited by 64
  6. 2025OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modelingcited by 62
  7. 2025Hunyuan-GameCraft-2: Instruction-following Interactive Game World Modelcited by 61
  8. 2025Memory Forcing: Spatio-Temporal Memory for Consistent Scene Generation on Minecraftcited by 52
  9. 2025Pre-Trained Video Generative Models as World Simulatorscited by 44
  10. 2025DeepVerse: 4D Autoregressive Video Generation as a World Modelcited by 40
  11. 2025Spatia: Video Generation with Updatable Spatial Memorycited by 40
  12. 2025VRAG: Learning World Models for Interactive Video Generationcited by 38
  13. 2025PAN: A World Model for General, Actionable, and Long-Horizon World Simulationcited by 36
  14. 2026MemRoPE: Training-Free Infinite Video Generation via Evolving Memory Tokenscited by 27
  15. 2025A Survey of Interactive Generative Videocited by 23
  16. 2026MultiWorld: Scalable Multi-Agent Multi-View Video World Modelscited by 23
  17. 2025PlayerOne: Egocentric World Simulatorcited by 20
  18. 2025StateSpaceDiffuser: Bringing Long Context to Diffusion World Modelscited by 19
  19. 2025NeuralOS: Towards Simulating Operating Systems via Neural Generative Modelscited by 19
  20. 2026Omni-WorldBench: Towards a Comprehensive Interaction-Centric Evaluation for World Modelscited by 18
  21. 2025Unifying Autoregressive and Diffusion-Based Sequence Generationcited by 17
  22. 2025WorldWeaver: Generating Long-Horizon Video Worlds via Rich Perceptioncited by 16
  23. 2026LIVE: Long-horizon Interactive Video World Modelingcited by 16
  24. 2026Anchor Forcing: Anchor Memory and Tri-Region RoPE for Interactive Streaming Video Diffusioncited by 16
  25. 2026LongLive-RAG: A General Retrieval-Augmented Framework for Long Video Generationcited by 16
  26. 2025Yan: Foundational Interactive Video Generationcited by 15
  27. 2025Precise Action-to-Video Generation Through Visual Action Promptscited by 15
  28. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  29. 2025BulletTime: Decoupled Control of Time and Camera Pose for Video Generationcited by 14
  30. 2026EgoSim: Egocentric World Simulator for Embodied Interaction Generationcited by 14
  31. 2026Code World Model: Coding Agent as World Braincited by 14
  32. 2025Position: Interactive Generative Video as Next-Generation Game Enginecited by 13
  33. 2026Geometry-Aware Implicit Memory for Video World Modelscited by 13
  34. 2026WorldCam: Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representationcited by 12
  35. 2026Prisma-World: Camera-Controllable Multi-Agent Video World Modelcited by 11
  36. 2026Making Foresight Actionable: Repurposing Representation Alignment in World Action Modelscited by 11
  37. 2025Taming Camera-Controlled Video Generation with Verifiable Geometry Rewardcited by 10
  38. 2026Incantation: Natural Language as the Action Interface for Multi-Entity Video World Modelscited by 10

50 most cited of 80 All 80 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Matrix-game 2.0: An open-source real-time and streaming interactive world model
  2. 2026Matrix-Game 3.5: Enhancing Real-Time Streaming Interactive World Models with Patch Memory
  3. 2025Matrix-Game: Interactive World Foundation Model
  4. 2026AlayaWorld: Interactive Long-Horizon World Modeling -- Full Technical Report
  5. 2026DreamForge-World 0.1 Preview: A Low-Compute Real-Time Controllable World Model
  6. 2026DreamX-World 1.0: A General-Purpose Interactive World Model
  7. 2026GameWAM: A World Action Model for Video Games
  8. 2026PlayWorld: Benchmarking World Models with Agent Players over Long-Horizon Objectives