# HY-World 2.0: A Multi-Modal World Model for Reconstructing, Generating, and Simulating 3D Worlds

arXiv: https://arxiv.org/abs/2604.14268 · HTML: https://mlsanity.com/paper/2604.14268/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Placed in Video-generation world models and simulators, one of 11 directions in World models.
- Builds on 73 of its 85 references that are in the map.
- First posted on arXiv 15 Apr 2026.

## Placed in

- World models / [Video-generation world models and simulators](https://mlsanity.com/wm/video-generation-world-models-and-simulators/)

## Builds on (most cited first, 30 of 73)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields](https://mlsanity.com/paper/2111.12077/) (2021)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [VGGT: Visual Geometry Grounded Transformer](https://mlsanity.com/paper/2503.11651/) (2025)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation](https://mlsanity.com/paper/2305.16213/) (2023)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [SAM 3: Segment Anything with Concepts](https://mlsanity.com/paper/2511.16719/) (2025)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [Accelerating 3D Deep Learning with PyTorch3D](https://mlsanity.com/paper/2007.08501/) (2020)
- [Depth Anything 3: Recovering the Visual Space from Any Views](https://mlsanity.com/paper/2511.10647/) (2025)
- [Improved Distribution Matching Distillation for Fast Image Synthesis](https://mlsanity.com/paper/2405.14867/) (2024)
- [Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion](https://mlsanity.com/paper/2506.08009/) (2025)
- [Continuous 3D Perception Model with Persistent State](https://mlsanity.com/paper/2501.12387/) (2025)
- [DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Vision](https://mlsanity.com/paper/2312.16256/) (2023)
- [TartanAir: A Dataset to Push the Limits of Visual SLAM](https://mlsanity.com/paper/2003.14338/) (2020)
- [Q-Align: Teaching LMMs for Visual Scoring via Discrete Text-Defined Levels](https://mlsanity.com/paper/2312.17090/) (2023)
- [PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics](https://mlsanity.com/paper/2311.12198/) (2023)
- [Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass](https://mlsanity.com/paper/2501.13928/) (2025)
- [GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Control](https://mlsanity.com/paper/2503.03751/) (2025)

## Built on by (most cited first, 30 of 44)

- [SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformer](https://mlsanity.com/paper/2605.15178/) (2026)
- [ActWorld: From Explorable to Interactive World Model via Action-Aware Memory](https://mlsanity.com/paper/2606.17730/) (2026)
- [Code World Model: Coding Agent as World Brain](https://mlsanity.com/paper/2608.25927/) (2026)
- [Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trends](https://mlsanity.com/paper/2606.01164/) (2026)
- [WorldClaw: Agentic 3D Open-World Generation at Scale](https://mlsanity.com/paper/2608.05248/) (2026)
- [VideoWeave: Unlocking Geometric Consistency in Video Generation via Joint Geometry-Video Modeling](https://mlsanity.com/paper/2606.14162/) (2026)
- [Compression and Retrieval: Implicit Memory Retrieval for Video World Models](https://mlsanity.com/paper/2606.23105/) (2026)
- [AlayaWorld: Long-Horizon and Playable Video World Generation](https://mlsanity.com/paper/2607.06291/) (2026)
- [Worldscape-MoE: A Unified Mixture-of-Experts World Model for Scalable Heterogeneous Action Control](https://mlsanity.com/paper/2607.03964/) (2026)
- [MoVerse: Real-Time Video World Modeling with Panoramic Gaussian Scaffold](https://mlsanity.com/paper/2606.13376/) (2026)
- [OmniX: Any-view and Any-time 4D Reconstruction via Feed-forward Trajectory Fields](https://mlsanity.com/paper/2607.10840/) (2026)
- [PanoWorld: A Generative Spatial World Model for Consistent Whole-House Panorama Synthesis](https://mlsanity.com/paper/2605.17916/) (2026)
- [Holo-World: Unified Camera, Object and Weather Control for Video World Model](https://mlsanity.com/paper/2606.20083/) (2026)
- [WorldCycle: Self-Verifiable Reinforcement Learning for Long-Horizon Video World Models](https://mlsanity.com/paper/2608.04964/) (2026)
- [AlayaVista: Streaming World Modeling from Panoramic States to Perspective Video](https://mlsanity.com/paper/2609.14462/) (2026)
- [WorldAct: Activating Monolithic 3D Worlds into Interactive-Ready Object-Centric Scenes](https://mlsanity.com/paper/2605.15843/) (2026)
- [Pano2World: End-to-End 3D Generation via Unified Multi-View Sequences](https://mlsanity.com/paper/2607.00832/) (2026)
- [Security of World-Model-Based Embodied AI: A Lifecycle of Threats, Defenses, and Evaluation](https://mlsanity.com/paper/2607.28226/) (2026)
- [WorldSculpt: Generating Compositional Worlds from Grounded Videos](https://mlsanity.com/paper/2609.05416/) (2026)
- [ABot-3DWorld 0: A Universal World Model to Explore Any 3D Space](https://mlsanity.com/paper/2607.11673/) (2026)
- [Apple-π: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence](https://mlsanity.com/paper/2607.16401/) (2026)
- [Mental World Modeling](https://mlsanity.com/paper/2607.27201/) (2026)
- [StateFlow: Building, Evolving, and Accessing 3D World States for Previsualization](https://mlsanity.com/paper/2608.12314/) (2026)
- [PanoPlane: Plane-Aware Panoramic Completion for Sparse-View Indoor 3D Gaussian Splatting](https://mlsanity.com/paper/2605.14135/) (2026)
- [Mind the Gap: Standard 3DGS Evaluation Primarily Measures Near-Trajectory Interpolation](https://mlsanity.com/paper/2607.01556/) (2026)
- [Representation Learning in Diffusion and Flow-based Model: An Application Aspect](https://mlsanity.com/paper/2608.24068/) (2026)
- [N0-Foundation: Towards the Age of Tactile Intelligence](https://mlsanity.com/paper/2608.29601/) (2026)
- [CAER: Causal Action Effect Reweighting for World Model Training](https://mlsanity.com/paper/2608.30897/) (2026)
- [Valerant: An Automatic Navigable Game Map Generator via Action-Conditioned World Model Exploration](https://mlsanity.com/paper/2609.09418/) (2026)
- [World in World: Explore the World with World Models](https://mlsanity.com/paper/2609.11548/) (2026)

## Abstract (arXiv)

We introduce HY-World 2.0, a multi-modal world model framework that advances our prior project HY-World 1.0. HY-World 2.0 accommodates diverse input modalities, including text prompts, single-view images, multi-view images, and videos, and produces 3D world representations. With text or single-view image inputs, the model performs world generation, synthesizing high-fidelity, navigable 3D Gaussian Splatting (3DGS) scenes. This is achieved through a four-stage method: a) Panorama Generation with HY-Pano 2.0, b) Trajectory Planning with WorldNav, c) World Expansion with WorldStereo 2.0, and d) World Composition with WorldMirror 2.0. Specifically, we introduce key innovations to enhance panorama fidelity, enable 3D scene understanding and planning, and upgrade WorldStereo, our keyframe-based view generation model with consistent memory. We also upgrade WorldMirror, a feed-forward model for universal 3D prediction, by refining model architecture and learning strategy, enabling world reconstruction from multi-view images or videos. Also, we introduce WorldLens, a high-performance 3DGS rendering platform featuring a flexible engine-agnostic architecture, automatic IBL lighting, efficient collision detection, and training-rendering co-design, enabling interactive exploration of 3D worlds with character support. Extensive experiments demonstrate that HY-World 2.0 achieves state-of-the-art performance on several benchmarks among open-source approaches, delivering results comparable to the closed-source model Marble. We release all model weights, code, and technical details to facilitate reproducibility and support further research on 3D world models.