mlsanityData to 7 Oct 2026

Matrix-Game 3.5: Enhancing Real-Time Streaming Interactive World Models with Patch Memory

arXiv 2608.29910 ↗First posted 30 Aug 2026cs.CVcited by 7

On the map: World models → Video-generation world models and simulators

also in 1 more
Abstract (arXiv)

Interactive world models extend video generation from offline clip synthesis toward persistent simulation of interactive virtual worlds, enabling applications in games, robotics, embodied agents, and XR. Achieving stable long-horizon interactive generation, however, remains challenging, as the model must simultaneously preserve scene geometry, dynamic consistency, and camera control while supporting real-time autoregressive generation. Building upon Matrix-Game 3.0, we present Matrix-Game 3.5, as shown in Figure 1, which advances real-time interactive world generation toward geometry-aware and long-horizon consistent simulation through three key improvements. First, we propose a unified geometry-aware memory framework, whose patch-memory and tiled-PRoPE components introduce no additional learnable parameters, combining explicit 3D patch retrieval with projective camera conditioning to enable geometry-consistent camera control and faithful long-horizon scene recall. Second, we introduce a static-dynamic disentangled world representation that separately models static scene geometry and dynamic subjects, preserving both geometric consistency and subject identity throughout long-horizon generation. Third, we develop a two-stage progressive real-time distillation framework that converts a bidirectional diffusion model into a few-step causal generator through Perceptual Flow Matching and curriculum based Self-Rollout DMD, enabling minute-long real-time interactive generation. Extensive experiments demonstrate that, with a unified training corpus spanning Unreal simulation environments, open-world games, and internet videos, MatrixGame 3.5 achieves strong performance in long-horizon scene recall, precise camera control, subject consistency, prompt-driven world generation, and stable real-time open-world interaction.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in Video-generation world models and simulators, one of 11 directions in World models. Builds on 54 of its 57 references that are in the map. First posted on arXiv 30 Aug 2026.

Builds on

54 of 57 references.

Closest ancestors · 20

Same direction, newest first

  1. 2026WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory2 Jul
  2. 2026ActWorld: From Explorable to Interactive World Model via Action-Aware Memory16 Jun
  3. 2026DreamX-World 1.0: A General-Purpose Interactive World Model15 Jun
  4. 2026From Zero to Hero: Training-Free Custom Concept Spawning in World Models1 Jun
  5. 2026SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformer14 May
  6. 2026Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory10 Apr
  7. 2026MosaicMem: Hybrid Spatial Memory for Controllable Video World Models17 Mar
  8. 2026Advancing Open-source World Models28 Jan
  9. 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modeling16 Dec
  10. 2025RELIC: Interactive Video World Model with Long-Horizon Memory3 Dec
  11. 2025Matrix-game 2.0: An open-source real-time and streaming interactive world model18 Aug
  12. 2025DeepVerse: 4D Autoregressive Video Generation as a World Model1 Jun
8 more
  1. 2025MAGI-1: Autoregressive Video Generation at Scale19 May
  2. 2025WorldMem: Long-term Consistent World Simulation with Memory16 Apr
  3. 2025Wan: Open and Advanced Large-Scale Video Generative Models26 Mar
  4. 2025GameFactory: Creating New Games with Generative Interactive Videos14 Jan
  5. 2024GameGen-X: Interactive Open-world Game Video Generation1 Nov
  6. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Text21 Mar
  7. 2024Genie: Generative Interactive Environments23 Feb
  8. 2022Scalable Diffusion Models with Transformers19 Dec

Other references · 34

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  5. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,597
  6. 2023One-step Diffusion with Distribution Matching Distillationcited by 978
  7. 2024Improved Distribution Matching Distillation for Fast Image Synthesiscited by 648
  8. 2025Depth Anything 3: Recovering the Visual Space from Any Viewscited by 626
  9. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 569
  10. 2024LTX-Video: Realtime Video Latent Diffusioncited by 549
  11. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  12. 2024From Slow Bidirectional to Fast Autoregressive Video Diffusion Modelscited by 401
22 more
  1. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  2. 2026Gemma 4 Technical Reportcited by 187
  3. 2025Self-Forcing++: Towards Minute-Scale High-Quality Video Generationcited by 176
  4. 2026Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generationcited by 159
  5. 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrievalcited by 157
  6. 2024Boximator: Generating Rich and Controllable Motions for Video Synthesiscited by 107
  7. 2025VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed View Memorycited by 102
  8. 2025Cameras as Relative Positional Encodingcited by 94
  9. 2026Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generationcited by 44
  10. 2024DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise Motion Controlcited by 42
  11. 2025FlashVSR: Towards Real-Time Diffusion-Based Streaming Video Super-Resolutioncited by 40
  12. 2026Infinite Worlds with Versatile Interactionscited by 37
  13. 2025Positional Encoding Fieldcited by 20
  14. 2025Ultralytics YOLO Evolution: An Overview of YOLO27, YOLO26, YOLO11, YOLOv8, and YOLOv5 Object Detectors for Computer Vision and Pattern Recognitioncited by 17
  15. 2026Self Gradient Forcing: Native Long Video Extrapolationcited by 16
  16. 2026HiAR: Efficient Autoregressive Long Video Generation via Hierarchical Denoisingcited by 13
  17. 2026SceneCode: Executable World Programs for Editable Indoor Scenes with Articulated Objectscited by 7
  18. 2025ContextAnyone: Context-Aware Diffusion for Character-Consistent Text-to-Video Generationcited by 5
  19. 2026Memento: Reconstruct to Remember for Consistent Long Video Generationcited by 4
  20. 2026DistillAlign: Coordinating Mode Covering and Mode Seeking in Autoregressive Video Distillationcited by 4
  21. 2025TGT: Text-Grounded Trajectories for Locally Controlled Video Generationcited by 3
  22. 2026Perceptual Flow Matching for Few-Step Generative Modelingcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 7.

7 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Pixel Histories: World Models with Persistent 3D State
  2. 2024The Matrix: Infinite-Horizon World Generation with Real-Time Moving Control
  3. 2026WorldCam: Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representation
  4. 2026AlayaWorld: Interactive Long-Horizon World Modeling -- Full Technical Report
  5. 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends
  6. 2025Memory Forcing: Spatio-Temporal Memory for Consistent Scene Generation on Minecraft
  7. 2025Video World Models with Long-term Spatial Memory
  8. 2026ABot-World-0: Infinite Interactive World Rollout on a Single Desktop GPU