# MineWorld: a Real-Time and Open-Source Interactive World Model on Minecraft

arXiv: https://arxiv.org/abs/2504.08388 · HTML: https://mlsanity.com/paper/2504.08388/

## Key facts

- Cited by 97 papers in the map (data through 14 Oct 2026).
- Placed in Video-generation world models and simulators, one of 11 directions in World models.
- Builds on 33 of its 39 references that are in the map.
- First posted on arXiv 11 Apr 2025.

## Placed in

- World models / [Video-generation world models and simulators](https://mlsanity.com/wm/video-generation-world-models-and-simulators/)

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Root Mean Square Layer Normalization](https://mlsanity.com/paper/1910.07467/) (2019)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [MaskGIT: Masked Generative Image Transformer](https://mlsanity.com/paper/2202.04200/) (2022)
- [WorldSimBench: Towards Video Generation Models as World Simulators](https://mlsanity.com/paper/2410.18072/) (2024)
- [Cosmos World Foundation Model Platform for Physical AI](https://mlsanity.com/paper/2501.03575/) (2025)
- [Genie: Generative Interactive Environments](https://mlsanity.com/paper/2402.15391/) (2024)
- [Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation](https://mlsanity.com/paper/2310.05737/) (2023)
- [VideoPoet: A Large Language Model for Zero-Shot Video Generation](https://mlsanity.com/paper/2312.14125/) (2023)
- [Learning Interactive Real-World Simulators](https://mlsanity.com/paper/2310.06114/) (2023)
- [Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos](https://mlsanity.com/paper/2206.11795/) (2022)
- [Diffusion Models Are Real-Time Game Engines](https://mlsanity.com/paper/2408.14837/) (2024)
- [Sequential Modeling Enables Scalable Learning for Large Vision Models](https://mlsanity.com/paper/2312.00785/) (2023)
- [iVideoGPT: Interactive VideoGPTs are Scalable World Models](https://mlsanity.com/paper/2405.15223/) (2024)
- [Randomized Autoregressive Visual Generation](https://mlsanity.com/paper/2411.00776/) (2024)
- [VidTok: A Versatile and Open-Source Video Tokenizer](https://mlsanity.com/paper/2412.13061/) (2024)
- [aMUSEd: An Open MUSE Reproduction](https://mlsanity.com/paper/2401.01808/) (2024)
- [ZipAR: Parallel Auto-regressive Image Generation through Spatial Locality](https://mlsanity.com/paper/2412.04062/) (2024)
- [VidTwin: Video VAE with Decoupled Structure and Dynamics](https://mlsanity.com/paper/2412.17726/) (2024)

## Built on by (most cited first, 30 of 97)

- [Matrix-game 2.0: An open-source real-time and streaming interactive world model](https://mlsanity.com/paper/2508.13009/) (2025)
- [Training Agents Inside of Scalable World Models](https://mlsanity.com/paper/2509.24527/) (2025)
- [DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos](https://mlsanity.com/paper/2602.06949/) (2026)
- [Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modeling](https://mlsanity.com/paper/2507.07982/) (2025)
- [Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generation](https://mlsanity.com/paper/2506.09350/) (2025)
- [Vid2World: Crafting Video Diffusion Models to Interactive World Models](https://mlsanity.com/paper/2505.14357/) (2025)
- [MotionStream: Real-Time Video Generation with Interactive Motion Controls](https://mlsanity.com/paper/2511.01266/) (2025)
- [OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation](https://mlsanity.com/paper/2505.20292/) (2025)
- [WorldEval: World Model as Real-World Robot Policies Evaluator](https://mlsanity.com/paper/2505.19017/) (2025)
- [A Comprehensive Survey on World Models for Embodied AI](https://mlsanity.com/paper/2510.16732/) (2025)
- [Helios: Real Real-Time Long Video Generation Model](https://mlsanity.com/paper/2603.04379/) (2026)
- [Memory Forcing: Spatio-Temporal Memory for Consistent Scene Generation on Minecraft](https://mlsanity.com/paper/2510.03198/) (2025)
- [VRAG: Learning World Models for Interactive Video Generation](https://mlsanity.com/paper/2505.21996/) (2025)
- [PAN: A World Model for General, Actionable, and Long-Horizon World Simulation](https://mlsanity.com/paper/2511.09057/) (2025)
- [LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey](https://mlsanity.com/paper/2505.00753/) (2025)
- [WorldMark: A Unified Benchmark Suite for Interactive Video World Models](https://mlsanity.com/paper/2604.21686/) (2026)
- [A Survey of Interactive Generative Video](https://mlsanity.com/paper/2504.21853/) (2025)
- [WorldLens: Full-Spectrum Evaluations of Driving World Models in Real World](https://mlsanity.com/paper/2512.10958/) (2025)
- [MultiWorld: Scalable Multi-Agent Multi-View Video World Models](https://mlsanity.com/paper/2604.18564/) (2026)
- [Playing with Transformer at 30+ FPS via Next-Frame Diffusion](https://mlsanity.com/paper/2506.01380/) (2025)
- [PlayerOne: Egocentric World Simulator](https://mlsanity.com/paper/2506.09995/) (2025)
- [Memorize-and-Generate: Towards Long-Term Consistency in Real-Time Video Generation](https://mlsanity.com/paper/2512.18741/) (2025)
- [MobileWorldBench: Towards Semantic World Modeling For Mobile Agents](https://mlsanity.com/paper/2512.14014/) (2025)
- [WorldStereo: Bridging Camera-Guided Video Generation and Scene Reconstruction via 3D Geometric Memories](https://mlsanity.com/paper/2603.02049/) (2026)
- [Omni-WorldBench: Towards a Comprehensive Interaction-Centric Evaluation for World Models](https://mlsanity.com/paper/2603.22212/) (2026)
- [AstraNav-World: World Model for Foresight Control and Consistency](https://mlsanity.com/paper/2512.21714/) (2025)
- [LIVE: Long-horizon Interactive Video World Modeling](https://mlsanity.com/paper/2602.03747/) (2026)
- [Yan: Foundational Interactive Video Generation](https://mlsanity.com/paper/2508.08601/) (2025)
- [Simulating the Visual World with Artificial Intelligence: A Roadmap](https://mlsanity.com/paper/2511.08585/) (2025)
- [Wonder: Video World Model Done Better](https://mlsanity.com/paper/2607.26037/) (2026)

## Abstract (arXiv)

World modeling is a crucial task for enabling intelligent agents to effectively interact with humans and operate in dynamic environments. In this work, we propose MineWorld, a real-time interactive world model on Minecraft, an open-ended sandbox game which has been utilized as a common testbed for world modeling. MineWorld is driven by a visual-action autoregressive Transformer, which takes paired game scenes and corresponding actions as input, and generates consequent new scenes following the actions. Specifically, by transforming visual game scenes and actions into discrete token ids with an image tokenizer and an action tokenizer correspondingly, we consist the model input with the concatenation of the two kinds of ids interleaved. The model is then trained with next token prediction to learn rich representations of game states as well as the conditions between states and actions simultaneously. In inference, we develop a novel parallel decoding algorithm that predicts the spatial redundant tokens in each frame at the same time, letting models in different scales generate 4 to 7 frames per second and enabling real-time interactions with game players. In evaluation, we propose new metrics to assess not only visual quality but also the action following capacity when generating new scenes, which is crucial for a world model. Our comprehensive evaluation shows the efficacy of MineWorld, outperforming SoTA open-sourced diffusion based world models significantly. The code and model have been released.