Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States
Not on the map. Why?
Abstract (arXiv)
We propose Puffin-World, a unified multimodal architecture that integrates physical understanding, spatial simulation, and 3D world generation and reconstruction without relying on external offline modules. To reliably construct and interact with 3D worlds, our framework jointly models three native world states: physics (gravity field and latitude), geometry (depth), and appearance (image), together with a unified Omni-Camera representation that supports diverse tasks and flexible motions. Beyond modeling these states, we introduce a strategy for propagating physical dynamics across future frames. By grounding absolute camera properties in the real world, Puffin-World enables physically consistent and visually stable world generation. We further couple appearance and geometry within a single generative process, jointly synthesizing each future view and reconstructing its underlying geometry. This unified paradigm enables interleaved closed-loop applications requiring synergy across multiple tasks, including mimic and self-calibrated world exploration. To scale Puffin-World to complex scenarios, we construct Puffin-16M, comprising 15 million vision-language-camera triplets and 1 million trajectories featuring various and challenging motions. To foster further research in this area, we released the code, models, and datasets.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 108 of its 131 references that are in the map. First posted on arXiv 3 Sep 2026.
Builds on
108 of 131 references.
References · 108
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
- 2023Segment Anythingcited by 8,158
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
- 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
- 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
- 2018World Modelscited by 1,444
- 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
- 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
38 more
- 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
- 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
- 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
- 2024Emu3: Next-Token Prediction is All You Needcited by 761
- 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
- 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
- 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 731
- 2025Depth Anything 3: Recovering the Visual Space from Any Viewscited by 626
- 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 593
- 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
- 2025Continuous 3D Perception Model with Persistent Statecited by 527
- 2023DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Visioncited by 514
- 2020TartanAir: A Dataset to Push the Limits of Visual SLAMcited by 494
- 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
- 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
- 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 425
- 2026World Action Models are Zero-shot Policiescited by 411
- 2017Joint 2D-3D-Semantic Data for Indoor Scene Understandingcited by 404
- 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 392
- 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 363
- 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 330
- 2025GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Controlcited by 325
- 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
- 2018DeepMVS: Learning Multi-view Stereopsiscited by 277
- 2024Navigation World Modelscited by 262
- 2025Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformercited by 247
- 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
- 2025Transfer between Modalities with MetaQueriescited by 231
- 2018CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmarkcited by 214
- 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
- 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 200
- 2025Stable Virtual Camera: Generative View Synthesis with Diffusion Modelscited by 179
- 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
- 2024CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generationcited by 163
- 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
- 2026Qwen3.5-Omni Technical Reportcited by 146
- 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 137
50 most cited of 108 All 108 (JSON)
Built on by
Not cited yet.