# Consistent-1-to-3: Consistent Image to 3D View Synthesis via Geometry-aware Diffusion Models

arXiv: https://arxiv.org/abs/2310.03020 · HTML: https://mlsanity.com/paper/2310.03020/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models.
- Builds on 56 of its 62 references that are in the map.
- First posted on arXiv 4 Oct 2023.

## Placed in

- Diffusion models / [3D, point cloud and scene generation](https://mlsanity.com/diffusion/3d-point-cloud-and-scene-generation/)

## Builds on (most cited first, 30 of 56)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis](https://mlsanity.com/paper/2307.01952/) (2023)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [Vision Transformers for Dense Prediction](https://mlsanity.com/paper/2103.13413/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [Volume Rendering of Neural Implicit Surfaces](https://mlsanity.com/paper/2106.12052/) (2021)
- [MVDream: Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2308.16512/) (2023)
- [Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations](https://mlsanity.com/paper/1906.01618/) (2019)
- [Objaverse-XL: A Universe of 10M+ 3D Objects](https://mlsanity.com/paper/2307.05663/) (2023)
- [IBRNet: Learning Multi-View Image-Based Rendering](https://mlsanity.com/paper/2102.13090/) (2021)
- [Point-E: A System for Generating 3D Point Clouds from Complex Prompts](https://mlsanity.com/paper/2212.08751/) (2022)
- [GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis](https://mlsanity.com/paper/2007.02442/) (2020)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs](https://mlsanity.com/paper/2112.00724/) (2021)
- [Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items](https://mlsanity.com/paper/2204.11918/) (2022)
- [One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization](https://mlsanity.com/paper/2306.16928/) (2023)
- [LION: Latent Point Diffusion Models for 3D Shape Generation](https://mlsanity.com/paper/2210.06978/) (2022)
- [3D Shape Generation and Completion through Point-Voxel Diffusion](https://mlsanity.com/paper/2104.03670/) (2021)
- [Modular Primitives for High-Performance Differentiable Rendering](https://mlsanity.com/paper/2011.03277/) (2020)
- [Shap-E: Generating Conditional 3D Implicit Functions](https://mlsanity.com/paper/2305.02463/) (2023)
- [Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering](https://mlsanity.com/paper/2106.02634/) (2021)
- [RealFusion: 360deg Reconstruction of Any Object from a Single Image](https://mlsanity.com/paper/2302.10663/) (2023)

## Built on by (most cited first, 30 of 79)

- [SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion](https://mlsanity.com/paper/2403.12008/) (2024)
- [One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion](https://mlsanity.com/paper/2311.07885/) (2023)
- [ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2312.02201/) (2023)
- [GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models](https://mlsanity.com/paper/2310.08529/) (2023)
- [LVSM: A Large View Synthesis Model with Minimal 3D Inductive Bias](https://mlsanity.com/paper/2410.17242/) (2024)
- [RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3D](https://mlsanity.com/paper/2311.16918/) (2023)
- [PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Prediction](https://mlsanity.com/paper/2311.12024/) (2023)
- [SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency](https://mlsanity.com/paper/2407.17470/) (2024)
- [MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Views](https://mlsanity.com/paper/2411.04924/) (2024)
- [EscherNet: A Generative Model for Scalable View Synthesis](https://mlsanity.com/paper/2402.03908/) (2024)
- [Free3D: Consistent Novel View Synthesis without 3D Representation](https://mlsanity.com/paper/2312.04551/) (2023)
- [SF3D: Stable Fast 3D Mesh Reconstruction with UV-unwrapping and Illumination Disentanglement](https://mlsanity.com/paper/2408.00653/) (2024)
- [MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Model](https://mlsanity.com/paper/2408.10198/) (2024)
- [Meta 3D TextureGen: Fast and Consistent Texture Generation for 3D Objects](https://mlsanity.com/paper/2407.02430/) (2024)
- [SV4D 2.0: Enhancing Spatio-Temporal Consistency in Multi-View Video Diffusion for High-Quality 4D Generation](https://mlsanity.com/paper/2503.16396/) (2025)
- [DressCode: Autoregressively Sewing and Generating Garments from Text Guidance](https://mlsanity.com/paper/2401.16465/) (2024)
- [Vivid-ZOO: Multi-View Video Generation with Diffusion Model](https://mlsanity.com/paper/2406.08659/) (2024)
- [Text-to-Image Rectified Flow as Plug-and-Play Priors](https://mlsanity.com/paper/2406.03293/) (2024)
- [LLMs Meet Multimodal Generation and Editing: A Survey](https://mlsanity.com/paper/2405.19334/) (2024)
- [SpaRP: Fast 3D Object Reconstruction and Pose Estimation from Sparse Views](https://mlsanity.com/paper/2408.10195/) (2024)
- [MVPaint: Synchronized Multi-View Diffusion for Painting Anything 3D](https://mlsanity.com/paper/2411.02336/) (2024)
- [Cascade-Zero123: One Image to Highly Consistent 3D with Self-Prompted Nearby Views](https://mlsanity.com/paper/2312.04424/) (2023)
- [GeoDream: Disentangling 2D and Geometric Priors for High-Fidelity and Consistent 3D Generation](https://mlsanity.com/paper/2311.17971/) (2023)
- [MVPGS: Excavating Multi-view Priors for Gaussian Splatting from Sparse Input Views](https://mlsanity.com/paper/2409.14316/) (2024)
- [HarmonyView: Harmonizing Consistency and Diversity in One-Image-to-3D](https://mlsanity.com/paper/2312.15980/) (2023)
- [Stable Score Distillation for High-Quality 3D Generation](https://mlsanity.com/paper/2312.09305/) (2023)
- [MVD2: Efficient Multiview 3D Reconstruction for Multiview Diffusion](https://mlsanity.com/paper/2402.14253/) (2024)
- [3DSceneEditor: Controllable 3D Scene Editing with Gaussian Splatting](https://mlsanity.com/paper/2412.01583/) (2024)
- [Learn to Optimize Denoising Scores for 3D Generation: A Unified and Improved Diffusion Prior on NeRF and 3D Gaussian Splatting](https://mlsanity.com/paper/2312.04820/) (2023)
- [Atlas3D: Physically Constrained Self-Supporting Text-to-3D for Simulation and Fabrication](https://mlsanity.com/paper/2405.18515/) (2024)

## Abstract (arXiv)

Zero-shot novel view synthesis (NVS) from a single image is an essential problem in 3D object understanding. While recent approaches that leverage pre-trained generative models can synthesize high-quality novel views from in-the-wild inputs, they still struggle to maintain 3D consistency across different views. In this paper, we present Consistent-1-to-3, which is a generative framework that significantly mitigates this issue. Specifically, we decompose the NVS task into two stages: (i) transforming observed regions to a novel view, and (ii) hallucinating unseen regions. We design a scene representation transformer and view-conditioned diffusion model for performing these two stages respectively. Inside the models, to enforce 3D consistency, we propose to employ epipolor-guided attention to incorporate geometry constraints, and multi-view attention to better aggregate multi-view information. Finally, we design a hierarchy generation paradigm to generate long sequences of consistent views, allowing a full 360-degree observation of the provided object image. Qualitative and quantitative evaluation over multiple datasets demonstrates the effectiveness of the proposed mechanisms against state-of-the-art approaches. Our project page is at https://jianglongye.com/consistent123/