# World2Minecraft: Occupancy-Driven Simulated Scenes Construction

arXiv: https://arxiv.org/abs/2604.27578 · HTML: https://mlsanity.com/paper/2604.27578/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 37 of its 45 references that are in the map.
- First posted on arXiv 30 Apr 2026.

## Builds on (most cited first, 30 of 37)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments](https://mlsanity.com/paper/1711.07280/) (2017)
- [Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents](https://mlsanity.com/paper/2302.01560/) (2023)
- [MonoScene: Monocular 3D Semantic Scene Completion](https://mlsanity.com/paper/2112.00726/) (2021)
- [LayoutGPT: Compositional Visual Planning and Generation with Large Language Models](https://mlsanity.com/paper/2305.15393/) (2023)
- [WonderWorld: Interactive 3D Scene Generation from a Single Image](https://mlsanity.com/paper/2406.09394/) (2024)
- [Scan2CAD: Learning CAD Model Alignment in RGB-D Scans](https://mlsanity.com/paper/1811.11187/) (2018)
- [LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models](https://mlsanity.com/paper/2412.02193/) (2024)
- [3D-GPT: Procedural 3D Modeling with Large Language Models](https://mlsanity.com/paper/2310.12945/) (2023)
- [Symphonize 3D Semantic Scene Completion with Contextual Instance Queries](https://mlsanity.com/paper/2306.15670/) (2023)
- [NDC-Scene: Boost Monocular 3D Semantic Scene Completion in Normalized Device Coordinates Space](https://mlsanity.com/paper/2309.14616/) (2023)
- [I-Design: Personalized LLM Interior Designer](https://mlsanity.com/paper/2404.02838/) (2024)
- [ROCA: Robust CAD Model Retrieval and Alignment from a Single Image](https://mlsanity.com/paper/2112.01988/) (2021)
- [StyleShot: A Snapshot on Any Style](https://mlsanity.com/paper/2407.01414/) (2024)
- [DrivingForward: Feed-forward 3D Gaussian Splatting for Driving Scene Reconstruction from Flexible Surround-view Input](https://mlsanity.com/paper/2409.12753/) (2024)
- [EmbodiedOcc: Embodied 3D Occupancy Prediction for Vision-based Online Scene Understanding](https://mlsanity.com/paper/2412.04380/) (2024)
- [GROOT: Learning to Follow Instructions by Watching Gameplay Videos](https://mlsanity.com/paper/2310.08235/) (2023)
- [SceneCraft: Layout-Guided 3D Scene Generation](https://mlsanity.com/paper/2410.09049/) (2024)
- [Learning Domain Invariant Prompt for Vision-Language Models](https://mlsanity.com/paper/2212.04196/) (2022)
- [MCU: An Evaluation Framework for Open-Ended Game Agents](https://mlsanity.com/paper/2310.08367/) (2023)
- [FastLGS: Speeding up Language Embedded Gaussians with Feature Grid Mapping](https://mlsanity.com/paper/2406.01916/) (2024)
- [Monocular Occupancy Prediction for Scalable Indoor Scenes](https://mlsanity.com/paper/2407.11730/) (2024)
- [JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mouse](https://mlsanity.com/paper/2503.16365/) (2025)
- [Boundary-Aware Geometric Encoding for Semantic Segmentation of Point Clouds](https://mlsanity.com/paper/2101.02381/) (2021)
- [Omni-supervised Point Cloud Segmentation via Gradual Receptive Field Component Reasoning](https://mlsanity.com/paper/2105.10203/) (2021)

## Abstract (arXiv)

Embodied intelligence requires high-fidelity simulation environments to support perception and decision-making, yet existing platforms often suffer from data contamination and limited flexibility. To mitigate this, we propose World2Minecraft to convert real-world scenes into structured Minecraft environments based on 3D semantic occupancy prediction. In the reconstructed scenes, we can effortlessly perform downstream tasks such as Vision-Language Navigation(VLN). However, we observe that reconstruction quality heavily depends on accurate occupancy prediction, which remains limited by data scarcity and poor generalization in existing models. We introduce a low-cost, automated, and scalable data acquisition pipeline for creating customized occupancy datasets, and demonstrate its effectiveness through MinecraftOcc, a large-scale dataset featuring 100,165 images from 156 richly detailed indoor scenes. Extensive experiments show that our dataset provides a critical complement to existing datasets and poses a significant challenge to current SOTA methods. These findings contribute to improving occupancy prediction and highlight the value of World2Minecraft in providing a customizable and editable platform for personalized embodied AI research. Project page:https://world2minecraft.github.io/.