# Matrix-3D: Omnidirectional Explorable 3D World Generation

arXiv: https://arxiv.org/abs/2508.08086 · HTML: https://mlsanity.com/paper/2508.08086/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 62 of its 71 references that are in the map.
- First posted on arXiv 11 Aug 2025.

## Builds on (most cited first, 30 of 62)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [LRM: Large Reconstruction Model for Single Image to 3D](https://mlsanity.com/paper/2311.04400/) (2023)
- [MotionCtrl: A Unified and Flexible Motion Controller for Video Generation](https://mlsanity.com/paper/2312.03641/) (2023)
- [VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding](https://mlsanity.com/paper/2501.13106/) (2025)
- [Exploiting Diffusion Prior for Real-World Image Super-Resolution](https://mlsanity.com/paper/2305.07015/) (2023)
- [CameraCtrl: Enabling Camera Control for Text-to-Video Generation](https://mlsanity.com/paper/2404.02101/) (2024)
- [CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assets](https://mlsanity.com/paper/2406.13897/) (2024)
- [CAT3D: Create Anything in 3D with Multi-View Diffusion Models](https://mlsanity.com/paper/2405.10314/) (2024)
- [Depth Pro: Sharp Monocular Metric Depth in Less Than a Second](https://mlsanity.com/paper/2410.02073/) (2024)
- [ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis](https://mlsanity.com/paper/2409.02048/) (2024)
- [A Survey of Autonomous Driving: Common Practices and Emerging Technologies](https://mlsanity.com/paper/1906.05113/) (2019)
- [SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion](https://mlsanity.com/paper/2403.12008/) (2024)
- [GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting](https://mlsanity.com/paper/2404.19702/) (2024)
- [GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control](https://mlsanity.com/paper/2503.03751/) (2025)
- [Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering](https://mlsanity.com/paper/2106.02634/) (2021)
- [MoGe: Unlocking Accurate Monocular Geometry Estimation for Open-Domain Images with Optimal Training Supervision](https://mlsanity.com/paper/2410.19115/) (2024)
- [A Survey of Embodied AI: From Simulators to Research Tasks](https://mlsanity.com/paper/2103.04918/) (2021)
- [TripoSG: High-Fidelity 3D Shape Synthesis using Large-Scale Rectified Flow Models](https://mlsanity.com/paper/2502.06608/) (2025)

## Built on by (most cited first, 30 of 35)

- [Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory](https://mlsanity.com/paper/2604.08995/) (2026)
- [Lyra 2.0: Explorable Generative 3D Worlds](https://mlsanity.com/paper/2604.13036/) (2026)
- [HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds](https://mlsanity.com/paper/2604.14268/) (2026)
- [Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillation](https://mlsanity.com/paper/2509.19296/) (2025)
- [FantasyWorld: Geometry-Consistent World Modeling via Unified Video and 3D Prediction](https://mlsanity.com/paper/2509.21657/) (2025)
- [Unified Camera Positional Encoding for Controlled Video Generation](https://mlsanity.com/paper/2512.07237/) (2025)
- [VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control](https://mlsanity.com/paper/2601.05138/) (2026)
- [WorldStereo: Bridging Camera-Guided Video Generation and Scene Reconstruction via 3D Geometric Memories](https://mlsanity.com/paper/2603.02049/) (2026)
- [Code World Model: Coding Agent as World Brain](https://mlsanity.com/paper/2608.25927/) (2026)
- [Text-to-3D by Stitching a Multi-view Reconstruction Network to a Video Generator](https://mlsanity.com/paper/2510.13454/) (2025)
- [Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends](https://mlsanity.com/paper/2606.01164/) (2026)
- [PanoWorld-X: Generating Explorable Panoramic Worlds via Sphere-Aware Video Diffusion](https://mlsanity.com/paper/2509.24997/) (2025)
- [A Mechanistic View on Video Generation as World Models: State and Dynamics](https://mlsanity.com/paper/2601.17067/) (2026)
- [RLGF: Reinforcement Learning with Geometric Feedback for Autonomous Driving Video Generation](https://mlsanity.com/paper/2509.16500/) (2025)
- [PanoWorld: Real-World Panoramic Generation](https://mlsanity.com/paper/2607.09661/) (2026)
- [Terra: Explorable Native 3D World Model with Point Latents](https://mlsanity.com/paper/2510.14977/) (2025)
- [CRePE: Curved Ray Expectation Positional Encoding for Unified-Camera-Controlled Video Generation](https://mlsanity.com/paper/2605.12938/) (2026)
- [MoVerse: Real-Time Video World Modeling with Panoramic Gaussian Scaffold](https://mlsanity.com/paper/2606.13376/) (2026)
- [SpatialCrafter: Single Image World Modeling with Generative 3D Proxies](https://mlsanity.com/paper/2608.27073/) (2026)
- [MoRight: Motion Control Done Right](https://mlsanity.com/paper/2604.07348/) (2026)
- [Pano2World: End-to-End 3D Generation via Unified Multi-View Sequences](https://mlsanity.com/paper/2607.00832/) (2026)
- [ExGS: Extreme 3D Gaussian Compression with Diffusion Priors](https://mlsanity.com/paper/2509.24758/) (2025)
- [CustomX: Unified Character, Action, and Scene Customization in Video World Models](https://mlsanity.com/paper/2512.17796/) (2025)
- [World Models as an Intermediary between Agents and the Real World](https://mlsanity.com/paper/2602.00785/) (2026)
- [EmoSpace: Immersive Affective Image Generation Guided by Fine-Grained Emotion Prototypes](https://mlsanity.com/paper/2602.11658/) (2026)
- [CM-EVS: Sparse Panoramic RGB-D-Pose Data for Complete Scene Coverage](https://mlsanity.com/paper/2605.15597/) (2026)
- [LAWM-3D: Learning 3D-Aware Latent Actions from Human Videos for Generalizable Robot World Models](https://mlsanity.com/paper/2608.05706/) (2026)
- [EmoWorld: A Decoupled Affective Field for Controllable Emotional Video Generation](https://mlsanity.com/paper/2608.06231/) (2026)
- [StateFlow: Building, Evolving, and Accessing 3D World States for Previsualization](https://mlsanity.com/paper/2608.12314/) (2026)
- [Stepper: Stepwise Immersive Scene Generation with Multiview Panoramas](https://mlsanity.com/paper/2603.28980/) (2026)

## Abstract (arXiv)

Explorable 3D world generation from a single image or text prompt forms a cornerstone of spatial intelligence. Recent works utilize video model to achieve wide-scope and generalizable 3D world generation. However, existing approaches often suffer from a limited scope in the generated scenes. In this work, we propose Matrix-3D, a framework that utilize panoramic representation for wide-coverage omnidirectional explorable 3D world generation that combines conditional video generation and panoramic 3D reconstruction. We first train a trajectory-guided panoramic video diffusion model that employs scene mesh renders as condition, to enable high-quality and geometrically consistent scene video generation. To lift the panorama scene video to 3D world, we propose two separate methods: (1) a feed-forward large panorama reconstruction model for rapid 3D scene reconstruction and (2) an optimization-based pipeline for accurate and detailed 3D scene reconstruction. To facilitate effective training, we also introduce the Matrix-Pano dataset, the first large-scale synthetic collection comprising 116K high-quality static panoramic video sequences with depth and trajectory annotations. Extensive experiments demonstrate that our proposed framework achieves state-of-the-art performance in panoramic video generation and 3D world generation. See more in https://matrix-3d.github.io.