mlsanityData to 7 Oct 2026

Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States

arXiv 2609.04196 ↗First posted 3 Sep 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

We propose Puffin-World, a unified multimodal architecture that integrates physical understanding, spatial simulation, and 3D world generation and reconstruction without relying on external offline modules. To reliably construct and interact with 3D worlds, our framework jointly models three native world states: physics (gravity field and latitude), geometry (depth), and appearance (image), together with a unified Omni-Camera representation that supports diverse tasks and flexible motions. Beyond modeling these states, we introduce a strategy for propagating physical dynamics across future frames. By grounding absolute camera properties in the real world, Puffin-World enables physically consistent and visually stable world generation. We further couple appearance and geometry within a single generative process, jointly synthesizing each future view and reconstructing its underlying geometry. This unified paradigm enables interleaved closed-loop applications requiring synergy across multiple tasks, including mimic and self-calibrated world exploration. To scale Puffin-World to complex scenarios, we construct Puffin-16M, comprising 15 million vision-language-camera triplets and 1 million trajectories featuring various and challenging motions. To foster further research in this area, we released the code, models, and datasets.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 108 of its 131 references that are in the map. First posted on arXiv 3 Sep 2026.

Builds on

108 of 131 references.

References · 108

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2023Segment Anythingcited by 8,158
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  6. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  7. 2025Qwen3-VL Technical Reportcited by 2,534
  8. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
  9. 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
  10. 2018World Modelscited by 1,444
  11. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  12. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
38 more
  1. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
  2. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
  3. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  4. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  5. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  6. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  7. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
  8. 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 731
  9. 2025Depth Anything 3: Recovering the Visual Space from Any Viewscited by 626
  10. 2020Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understandingcited by 593
  11. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  12. 2025Continuous 3D Perception Model with Persistent Statecited by 527
  13. 2023DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Visioncited by 514
  14. 2020TartanAir: A Dataset to Push the Limits of Visual SLAMcited by 494
  15. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  16. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  17. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 425
  18. 2026World Action Models are Zero-shot Policiescited by 411
  19. 2017Joint 2D-3D-Semantic Data for Indoor Scene Understandingcited by 404
  20. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 392
  21. 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 363
  22. 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 330
  23. 2025GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Controlcited by 325
  24. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  25. 2018DeepMVS: Learning Multi-view Stereopsiscited by 277
  26. 2024Navigation World Modelscited by 262
  27. 2025Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformercited by 247
  28. 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
  29. 2025Transfer between Modalities with MetaQueriescited by 231
  30. 2018CrowdPose: Efficient Crowded Scenes Pose Estimation and A New Benchmarkcited by 214
  31. 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
  32. 2024TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generationcited by 200
  33. 2025Stable Virtual Camera: Generative View Synthesis with Diffusion Modelscited by 179
  34. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
  35. 2024CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generationcited by 163
  36. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  37. 2026Qwen3.5-Omni Technical Reportcited by 146
  38. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 137

50 most cited of 108 All 108 (JSON)

Built on by

Not cited yet.