Matrix-Game 3.5: Enhancing Real-Time Streaming Interactive World Models with Patch Memory
On the map: World models → Video-generation world models and simulators
also in 1 more
Abstract (arXiv)
Interactive world models extend video generation from offline clip synthesis toward persistent simulation of interactive virtual worlds, enabling applications in games, robotics, embodied agents, and XR. Achieving stable long-horizon interactive generation, however, remains challenging, as the model must simultaneously preserve scene geometry, dynamic consistency, and camera control while supporting real-time autoregressive generation. Building upon Matrix-Game 3.0, we present Matrix-Game 3.5, as shown in Figure 1, which advances real-time interactive world generation toward geometry-aware and long-horizon consistent simulation through three key improvements. First, we propose a unified geometry-aware memory framework, whose patch-memory and tiled-PRoPE components introduce no additional learnable parameters, combining explicit 3D patch retrieval with projective camera conditioning to enable geometry-consistent camera control and faithful long-horizon scene recall. Second, we introduce a static-dynamic disentangled world representation that separately models static scene geometry and dynamic subjects, preserving both geometric consistency and subject identity throughout long-horizon generation. Third, we develop a two-stage progressive real-time distillation framework that converts a bidirectional diffusion model into a few-step causal generator through Perceptual Flow Matching and curriculum based Self-Rollout DMD, enabling minute-long real-time interactive generation. Extensive experiments demonstrate that, with a unified training corpus spanning Unreal simulation environments, open-world games, and internet videos, MatrixGame 3.5 achieves strong performance in long-horizon scene recall, precise camera control, subject consistency, prompt-driven world generation, and stable real-time open-world interaction.
Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 54 of its 57 references that are in the map. First posted on arXiv 30 Aug 2026.
Builds on
54 of 57 references.
Closest ancestors · 20
Same direction, newest first
- 2026WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory2 Jul
- 2026ActWorld: From Explorable to Interactive World Model via Action-Aware Memory16 Jun
- 2026DreamX-World 1.0: A General-Purpose Interactive World Model15 Jun
- 2026From Zero to Hero: Training-Free Custom Concept Spawning in World Models1 Jun
- 2026SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformer14 May
- 2026Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory10 Apr
- 2026MosaicMem: Hybrid Spatial Memory for Controllable Video World Models17 Mar
- 2026Advancing Open-source World Models28 Jan
- 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling16 Dec
- 2025RELIC: Interactive Video World Model with Long-Horizon Memory3 Dec
- 2025Matrix-game 2.0: An open-source real-time and streaming interactive world model18 Aug
- 2025DeepVerse: 4D Autoregressive Video Generation as a World Model1 Jun
8 more
- 2025MAGI-1: Autoregressive Video Generation at Scale19 May
- 2025WorldMem: Long-term Consistent World Simulation with Memory16 Apr
- 2025Wan: Open and Advanced Large-Scale Video Generative Models26 Mar
- 2025GameFactory: Creating New Games with Generative Interactive Videos14 Jan
- 2024GameGen-X: Interactive Open-world Game Video Generation1 Nov
- 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text21 Mar
- 2024Genie: Generative Interactive Environments23 Feb
- 2022Scalable Diffusion Models with Transformers19 Dec
Other references · 34
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
- 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
- 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
- 2023One-step Diffusion with Distribution Matching Distillationcited by 978
- 2024Improved Distribution Matching Distillation for Fast Image Synthesiscited by 648
- 2025Depth Anything 3: Recovering the Visual Space from Any Viewscited by 626
- 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 569
- 2024LTX-Video: Realtime Video Latent Diffusioncited by 549
- 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
- 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 401
22 more
- 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
- 2026Gemma 4 Technical Reportcited by 187
- 2025Self-Forcing++: Towards Minute-Scale High-Quality Video Generationcited by 176
- 2026Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generationcited by 159
- 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrievalcited by 157
- 2024Boximator: Generating Rich and Controllable Motions for Video Synthesiscited by 107
- 2025VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed View Memorycited by 102
- 2025Cameras as Relative Positional Encodingcited by 94
- 2026Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generationcited by 44
- 2024DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise Motion Controlcited by 42
- 2025FlashVSR: Towards Real-Time Diffusion-Based Streaming Video Super-Resolutioncited by 40
- 2026Infinite Worlds with Versatile Interactionscited by 37
- 2025Positional Encoding Fieldcited by 20
- 2025Ultralytics YOLO Evolution: An Overview of YOLO27, YOLO26, YOLO11, YOLOv8, and YOLOv5 Object Detectors for Computer Vision and Pattern Recognitioncited by 17
- 2026Self Gradient Forcing: Native Long Video Extrapolationcited by 16
- 2026HiAR: Efficient Autoregressive Long Video Generation via Hierarchical Denoisingcited by 13
- 2026SceneCode: Executable World Programs for Editable Indoor Scenes with Articulated Objectscited by 7
- 2025ContextAnyone: Context-Aware Diffusion for Character-Consistent Text-to-Video Generationcited by 5
- 2026Memento: Reconstruct to Remember for Consistent Long Video Generationcited by 4
- 2026DistillAlign: Coordinating Mode Covering and Mode Seeking in Autoregressive Video Distillationcited by 4
- 2025TGT: Text-Grounded Trajectories for Locally Controlled Video Generationcited by 3
- 2026Perceptual Flow Matching for Few-Step Generative Modelingcited by 3
Built on by
Citing papers per year: 2026: 7.
7 papers.
- 2026SolarWM: Open Data and Scalable Training for Long-Horizon Video World Modelscited by 9
- 2026WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memorycited by 1
- 2026World in World: Explore the World with World Modelscited by 0
- 2026Code Plans, Diffusion Renders: Open-Ended Generative World Modelingcited by 0
- 2026OPIS: An Input-Grounded Benchmark for Multi-Object Memory in Video World Modelscited by 0
- 2026LongTake: Learning to Sustain Dynamics in Long-Horizon Video Generationcited by 0
- 2026LOCI: Spatial Linear Memory for Streaming World Modelscited by 0
- 2026LOCI: Spatial Linear Memory for Streaming World Models30 Sep 2026
- 2026LongTake: Learning to Sustain Dynamics in Long-Horizon Video Generation29 Sep 2026
- 2026OPIS: An Input-Grounded Benchmark for Multi-Object Memory in Video World Models28 Sep 2026
- 2026Code Plans, Diffusion Renders: Open-Ended Generative World Modeling22 Sep 2026
- 2026WorldCrafter: Consistent Video World Model with Implicit 3D-aware Memory21 Sep 2026
- 2026World in World: Explore the World with World Models10 Sep 2026
- 2026SolarWM: Open Data and Scalable Training for Long-Horizon Video World Models2 Sep 2026
Similar, not cited
Close in wording, no citation either way
- 2026Beyond Pixel Histories: World Models with Persistent 3D State
- 2024The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control
- 2026WorldCam: Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representation
- 2026AlayaWorld: Interactive Long-Horizon World Modeling -- Full Technical Report
- 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends
- 2025Memory Forcing: Spatio-Temporal Memory for Consistent Scene Generation on Minecraft
- 2025Video World Models with Long-term Spatial Memory
- 2026ABot-World-0: Infinite Interactive World Rollout on a Single Desktop GPU