Thinking with Camera: A Unified Multimodal Model for Camera-Centric Understanding and Generation
Not on the map. Why?
Abstract (arXiv)
Camera-centric understanding and generation are two cornerstones of spatial intelligence, yet they are typically studied in isolation. We present Puffin, a unified camera-centric multimodal model that extends spatial awareness along the camera dimension. Puffin integrates language regression and diffusion-based generation to interpret and create scenes from arbitrary viewpoints. To bridge the modality gap between cameras and vision-language, we introduce a novel paradigm that treats camera as language, enabling thinking with camera. This guides the model to align spatially grounded visual cues with photographic terminology while reasoning across geometric context. Puffin is trained on Puffin-4M, a large-scale dataset of 4 million vision-language-camera triplets. We incorporate both global camera parameters and pixel-wise camera maps, yielding flexible and reliable spatial generation. Experiments demonstrate Puffin superior performance over specialized models for camera-centric generation and understanding. With instruction tuning, Puffin generalizes to diverse cross-view tasks such as spatial imagination, world exploration, and photography guidance. We will release the code, models, dataset pipeline, and benchmark to advance multimodal spatial intelligence research.
Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 70 of its 96 references that are in the map. First posted on arXiv 9 Oct 2025.
Builds on
70 of 96 references.
References · 70
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2023Visual Instruction Tuningcited by 7,792
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,769
- 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
- 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
- 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
- 2025Qwen-Image Technical Reportcited by 973
38 more
- 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 855
- 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
- 2024Emu3: Next-Token Prediction is All You Needcited by 761
- 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 734
- 2018MegaDepth: Learning Single-View Depth Prediction from Internet Photoscited by 731
- 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
- 2020TartanAir: A Dataset to Push the Limits of Visual SLAMcited by 494
- 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
- 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 447
- 2017Joint 2D-3D-Semantic Data for Indoor Scene Understandingcited by 404
- 2024InternLM2 Technical Reportcited by 391
- 2025BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Datasetcited by 363
- 2025GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Controlcited by 325
- 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
- 2025Show-o2: Improved Native Unified Multimodal Modelscited by 242
- 2025Transfer between Modalities with MetaQueriescited by 231
- 2024MetaMorph: Multimodal Understanding and Generation via Instruction Tuningcited by 205
- 2024Cameras as Rays: Pose Estimation via Ray Diffusioncited by 129
- 2022LaMAR: Benchmarking Localization and Mapping for Augmented Realitycited by 98
- 2018LayoutNet: Reconstructing the 3D Room Layout from a Single RGB Imagecited by 94
- 2024RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Modelscited by 88
- 2022Perspective Fields for Single Image Camera Calibrationcited by 75
- 2025Harmonizing Visual Representations for Unified Multimodal Understanding and Generationcited by 74
- 2025Towards Understanding Camera Motions in Any Videocited by 62
- 2025Unified Autoregressive Visual Generation and Understanding with Continuous Tokenscited by 51
- 2024GeoCalib: Learning Single-image Calibration with Geometric Optimizationcited by 50
- 2025TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generationcited by 50
- 2017A Perceptual Measure for Deep Single Image Camera Calibrationcited by 45
- 2025OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generationcited by 45
- 2023Tame a Wild Camera: In-the-Wild Monocular Camera Calibrationcited by 34
- 2021CTRL-C: Camera calibration TRansformer with Line-Classificationcited by 33
- 2018FishEyeRecNet: A Multi-Context Collaborative Deep Network for Fisheye Image Rectificationcited by 32
- 2023Balanced Spherical Grid for Egocentric View Synthesiscited by 29
- 2019Blind Geometric Distortion Correction on Images Through Deep Learningcited by 28
- 2022LGT-Net: Indoor Panoramic Room Layout Estimation with Geometry-Aware Transformer Networkcited by 28
- 2023360Loc: A Dataset and Benchmark for Omnidirectional Visual Localization with Cross-device Queriescited by 28
- 2016Detecting Vanishing Points using Global Image Context in a Non-Manhattan Worldcited by 27
50 most cited of 70 All 70 (JSON)
Built on by
Citing papers per year: 2025: 3; 2026: 16.
19 papers.
Sort:
By area:
- 2025OneStory: Coherent Multi-Shot Video Generation with Adaptive Memorycited by 31
- 2026AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memoriescited by 13
- 2026Prisma-World: Camera-Controllable Multi-Agent Video World Modelcited by 8
- 2025OpenView: Empowering MLLMs with Out-of-view VQAcited by 3
- 2026Venus: Benchmarking and Empowering Multimodal Large Language Models for Aesthetic Guidance and Croppingcited by 3
- 2026Edit in 2D, Verify in 3D: Reinforcement Learning for Multi-view Consistent Scene Editingcited by 3
- 2026World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoningcited by 3
- 2026CameraNoise: Enabling Faithful Camera Control in Video Diffusion through Geometry-Flow-Guided Noise Warpingcited by 3
- 2025GimbalDiffusion: Gravity-Aware Camera Control for Video Generationcited by 2
- 2026Predicting Camera Pose from Perspective Descriptions for Spatial Reasoningcited by 1
- 2026Thinking with Novel Views: A Systematic Analysis of Generative-Augmented Spatial Intelligencecited by 1
- 2026AesFormer: Transform Everyday Photos into Beautiful Memoriescited by 1
7 more
- 2026CameraEditor: Camera-Controlled Image Editing via Video-Prior Sequential Modelingcited by 1
- 2026UniStitch: Unifying Semantic and Geometric Features for Image Stitchingcited by 0
- 2026DeltaCam: Differential Intrinsic Camera Modeling for Video Generationcited by 0
- 2026Rethinking VLM Representation for VLA Initializationcited by 0
- 2026Enhancing In-context Panoramic Generation via Geometric-aware Pretrainingcited by 0
- 2026Puffin-World: Scaling a Unified Multimodal Model with Native 3D World Statescited by 0
- 2026The Camera Inside the Editor: Reading the Implicit Camera of Image Editors with Painted Calibration Patternscited by 0
- 2026The Camera Inside the Editor: Reading the Implicit Camera of Image Editors with Painted Calibration Patterns29 Sep 2026
- 2026Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States3 Sep 2026
- 2026CameraEditor: Camera-Controlled Image Editing via Video-Prior Sequential Modeling1 Sep 2026
- 2026Enhancing In-context Panoramic Generation via Geometric-aware Pretraining9 Jul 2026
- 2026Prisma-World: Camera-Controllable Multi-Agent Video World Model8 Jun 2026
- 2026CameraNoise: Enabling Faithful Camera Control in Video Diffusion through Geometry-Flow-Guided Noise Warping29 May 2026
- 2026Rethinking VLM Representation for VLA Initialization25 May 2026
- 2026DeltaCam: Differential Intrinsic Camera Modeling for Video Generation24 May 2026
- 2026AesFormer: Transform Everyday Photos into Beautiful Memories21 May 2026
- 2026Thinking with Novel Views: A Systematic Analysis of Generative-Augmented Spatial Intelligence11 May 2026
- 2026World2VLM: Distilling World Model Imagination into VLMs for Dynamic Spatial Reasoning29 Apr 2026
- 2026UniStitch: Unifying Semantic and Geometric Features for Image Stitching11 Mar 2026
7 more
- 2026Edit in 2D, Verify in 3D: Reinforcement Learning for Multi-view Consistent Scene Editing3 Mar 2026
- 2026Venus: Benchmarking and Empowering Multimodal Large Language Models for Aesthetic Guidance and Cropping27 Feb 2026
- 2026AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories16 Feb 2026
- 2026Predicting Camera Pose from Perspective Descriptions for Spatial Reasoning5 Feb 2026
- 2025OpenView: Empowering MLLMs with Out-of-view VQA21 Dec 2025
- 2025GimbalDiffusion: Gravity-Aware Camera Control for Video Generation9 Dec 2025
- 2025OneStory: Coherent Multi-Shot Video Generation with Adaptive Memory8 Dec 2025
- 2025OneStory: Coherent Multi-Shot Video Generation with Adaptive Memorycited by 31
- 2026AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memoriescited by 13
- 2025GimbalDiffusion: Gravity-Aware Camera Control for Video Generationcited by 2
- 2026AesFormer: Transform Everyday Photos into Beautiful Memoriescited by 1
- 2026CameraEditor: Camera-Controlled Image Editing via Video-Prior Sequential Modelingcited by 1
- 2026UniStitch: Unifying Semantic and Geometric Features for Image Stitchingcited by 0
- 2026DeltaCam: Differential Intrinsic Camera Modeling for Video Generationcited by 0
- 2026Enhancing In-context Panoramic Generation via Geometric-aware Pretrainingcited by 0
- 2026Puffin-World: Scaling a Unified Multimodal Model with Native 3D World Statescited by 0
- 2026The Camera Inside the Editor: Reading the Implicit Camera of Image Editors with Painted Calibration Patternscited by 0
- 2026The Camera Inside the Editor: Reading the Implicit Camera of Image Editors with Painted Calibration Patterns29 Sep 2026
- 2026Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States3 Sep 2026
- 2026CameraEditor: Camera-Controlled Image Editing via Video-Prior Sequential Modeling1 Sep 2026
- 2026Enhancing In-context Panoramic Generation via Geometric-aware Pretraining9 Jul 2026
- 2026DeltaCam: Differential Intrinsic Camera Modeling for Video Generation24 May 2026
- 2026AesFormer: Transform Everyday Photos into Beautiful Memories21 May 2026
- 2026UniStitch: Unifying Semantic and Geometric Features for Image Stitching11 Mar 2026
- 2026AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memories16 Feb 2026
- 2025GimbalDiffusion: Gravity-Aware Camera Control for Video Generation9 Dec 2025
- 2025OneStory: Coherent Multi-Shot Video Generation with Adaptive Memory8 Dec 2025
Similar, not cited
Close in wording, no citation either way
- 2025Seeing through Imagination: Learning Scene Geometry via Implicit Spatial World Modeling
- 2026SPARGen: Unifying Spatial Perception and Reasoning through Native Multimodal Generation
- 2025SpaceMind: Camera-Guided Modality Fusion for Spatial Reasoning in Vision-Language Models
- 2025SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewards
- 2026On the Generalization Capacities of MLLMs for Spatial Intelligence
- 2025From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D
- 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methods
- 2026OrientSAM: Mitigating Camera-Centric Shortcut in Multimodal Spatial Reasoning via Orientation-Aware Spatial Alignment