How to Spin an Object: First, Get the Shape Right
Not on the map. Why?
Abstract (arXiv)
Image-to-3D models increasingly rely on hierarchical generation to disentangle geometry and texture. However, the design choices underlying these two-stage models--particularly the optimal choice of intermediate geometric representations--remain largely understudied. To investigate this, we introduce unPIC (undo-a-Picture), a modular framework for empirical analysis of image-to-3D pipelines. By factorizing the generation process into a multiview-geometry prior followed by an appearance decoder, unPIC enables a rigorous comparison of intermediate geometry representations. Through this framework, we identify that a specific representation, Camera-Relative Object Coordinates (CROCS), significantly outperforms alternatives such as depth maps, pretrained visual features, and other pointmap-based representations. We demonstrate that CROCS is not only easier for the first-stage geometry prior to predict, but also serves as an effective conditioning signal for ensuring 360-degree consistency during appearance decoding. Another advantage is that CROCS enables fully feedforward, direct 3D point cloud generation without requiring a separate post-hoc reconstruction step. Our unPIC formulation utilizing CROCS achieves superior novel-view quality, geometric accuracy, and multiview consistency; it outperforms leading baselines, including InstantMesh, Direct3D, CAT3D, Free3D, and EscherNet, on datasets of real-world 3D captures like Google Scanned Objects and the Digital Twin Catalog.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 42 of its 59 references that are in the map. First posted on arXiv 13 Dec 2024.
Builds on
42 of 59 references.
References · 42
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
- 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
- 2022Classifier-Free Diffusion Guidancecited by 5,717
- 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
- 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
- 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
- 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
- 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
- 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
- 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
30 more
- 2024Structured 3D Latents for Scalable and Versatile 3D Generationcited by 742
- 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
- 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 555
- 2019Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimationcited by 490
- 2024CLAY: A Controllable Large-scale Generative Model for Creating High-quality 3D Assetscited by 444
- 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 425
- 2024InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Modelscited by 421
- 2021ABO: Dataset and Benchmarks for Real-World 3D Object Understandingcited by 360
- 2023ReconFusion: 3D Reconstruction with Diffusion Priorscited by 317
- 2022Novel View Synthesis with Diffusion Modelscited by 299
- 2024GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Imagecited by 283
- 2024CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Modelcited by 214
- 2024Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformercited by 210
- 2024Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modelingcited by 209
- 2024Probing the 3D Awareness of Visual Foundation Modelscited by 175
- 2023Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perceptioncited by 166
- 2024Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Predictioncited by 163
- 2024MeshLRM: Large Reconstruction Model for High-Quality Meshescited by 138
- 2023SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3Dcited by 134
- 2023ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Imagecited by 109
- 2024EscherNet: A Generative Model for Scalable View Synthesiscited by 94
- 2024MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Modelcited by 87
- 2024Optimal Flow Matching: Learning Straight Trajectories in Just One Stepcited by 74
- 2023Free3D: Consistent Novel View Synthesis without 3D Representationcited by 72
- 2024Advances in 3D Generation: A Surveycited by 62
- 2023Diffusion 3D Features (Diff3F): Decorating Untextured Shapes with Distilled Semantic Featurescited by 55
- 2024SpaRP: Fast 3D Object Reconstruction and Pose Estimation from Sparse Viewscited by 35
- 2024Neural Assets: 3D-Aware Multi-Object Scene Synthesis with Image Diffusion Modelscited by 30
- 2023DORSal: Diffusion for Object-centric Representations of Scenes et alcited by 16
- 2023Leveraging VLM-Based Pipelines to Annotate 3D Objectscited by 10
Built on by
Not cited yet.