# Matrix-Game: Interactive World Foundation Model

arXiv: https://arxiv.org/abs/2506.18701 · HTML: https://mlsanity.com/paper/2506.18701/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jun 2025.

## Built on by (most cited first, 19 of 19)

- [Memorize-and-Generate: Towards Long-Term Consistency in Real-Time Video Generation](https://mlsanity.com/paper/2512.18741/) (2025)
- [Alaya-EVOKE: From Linear-Scaling Supervision to Endless World](https://mlsanity.com/paper/2608.13546/) (2026)
- [Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application](https://mlsanity.com/paper/2606.12191/) (2026)
- [Marionette: Predicting World States, Rendering Geometry, Painting Appearance](https://mlsanity.com/paper/2608.14530/) (2026)
- [RoboWorld: Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation](https://mlsanity.com/paper/2607.01060/) (2026)
- [How Far Are Vision-Language Models from Constructing the Real World? A Benchmark for Physical Generative Reasoning](https://mlsanity.com/paper/2603.24866/) (2026)
- [DecMem: Towards Minute-Long Consistent World Generation with Decoupled Memory](https://mlsanity.com/paper/2605.31336/) (2026)
- [ForgeWM: Progressive Causal Training for Few-Step Action-Conditioned Video World Models](https://mlsanity.com/paper/2608.14022/) (2026)
- [Can Video World Models Track Unobserved World States?](https://mlsanity.com/paper/2608.30692/) (2026)
- [Game2World Engine: Unlocking In-the-Wild Gameplay Videos for World Model Training](https://mlsanity.com/paper/2608.24680/) (2026)
- [DreamForge-World 0.1 Preview: A Low-Compute Real-Time Controllable World Model](https://mlsanity.com/paper/2606.30292/) (2026)
- [DynaVieW: Schema-Guided World Modeling for Understanding Hierarchical Visual Dynamics](https://mlsanity.com/paper/2607.04112/) (2026)
- [Quo Vadis, World Modeling?](https://mlsanity.com/paper/2608.02713/) (2026)
- [From Generation to Simulation: How Far Are World Models from Being True Simulators?](https://mlsanity.com/paper/2608.23070/) (2026)
- [HappyWorld-Bench](https://mlsanity.com/paper/2609.24308/) (2026)
- [HelixWorld: A Real-time Interactive Audio-Visual World Model](https://mlsanity.com/paper/2609.38123/) (2026)
- [ROWBench: Do Video Models Render What the Program Specifies?](https://mlsanity.com/paper/2610.02205/) (2026)
- [SPW-Nav: A Streaming Panoramic World Model for Language-Guided Navigation](https://mlsanity.com/paper/2610.08941/) (2026)
- [MultiWorldBench: Do Independently Controlled Views Describe One Shared World?](https://mlsanity.com/paper/2610.11723/) (2026)

## Abstract (arXiv)

We introduce Matrix-Game, an interactive world foundation model for controllable game world generation. Matrix-Game is trained using a two-stage pipeline that first performs large-scale unlabeled pretraining for environment understanding, followed by action-labeled training for interactive video generation. To support this, we curate Matrix-Game-MC, a comprehensive Minecraft dataset comprising over 2,700 hours of unlabeled gameplay video clips and over 1,000 hours of high-quality labeled clips with fine-grained keyboard and mouse action annotations. Our model adopts a controllable image-to-world generation paradigm, conditioned on a reference image, motion context, and user actions. With over 17 billion parameters, Matrix-Game enables precise control over character actions and camera movements, while maintaining high visual quality and temporal coherence. To evaluate performance, we develop GameWorld Score, a unified benchmark measuring visual quality, temporal quality, action controllability, and physical rule understanding for Minecraft world generation. Extensive experiments show that Matrix-Game consistently outperforms prior open-source Minecraft world models (including Oasis and MineWorld) across all metrics, with particularly strong gains in controllability and physical consistency. Double-blind human evaluations further confirm the superiority of Matrix-Game, highlighting its ability to generate perceptually realistic and precisely controllable videos across diverse game scenarios. To facilitate future research on interactive image-to-world generation, we will open-source the Matrix-Game model weights and the GameWorld Score benchmark at https://github.com/SkyworkAI/Matrix-Game.