# MVD-Fusion: Single-view 3D via Depth-consistent Multi-view Generation

arXiv: https://arxiv.org/abs/2404.03656 · HTML: https://mlsanity.com/paper/2404.03656/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 55 references that are in the map.
- First posted on arXiv 4 Apr 2024.

## Builds on (most cited first, 30 of 47)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [DreamFusion: Text-to-3D using 2D Diffusion](https://mlsanity.com/paper/2209.14988/) (2022)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [pixelNeRF: Neural Radiance Fields from One or Few Images](https://mlsanity.com/paper/2012.02190/) (2020)
- [Objaverse: A Universe of Annotated 3D Objects](https://mlsanity.com/paper/2212.08051/) (2022)
- [Zero-1-to-3: Zero-shot One Image to 3D Object](https://mlsanity.com/paper/2303.11328/) (2023)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [MVDream: Multi-view Diffusion for 3D Generation](https://mlsanity.com/paper/2308.16512/) (2023)
- [Point-E: A System for Generating 3D Point Clouds from Complex Prompts](https://mlsanity.com/paper/2212.08751/) (2022)
- [Wonder3D: Single Image to 3D using Cross-Domain Diffusion](https://mlsanity.com/paper/2310.15008/) (2023)
- [Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction](https://mlsanity.com/paper/2109.00512/) (2021)
- [Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images](https://mlsanity.com/paper/1804.01654/) (2018)
- [Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items](https://mlsanity.com/paper/2204.11918/) (2022)
- [Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation](https://mlsanity.com/paper/2212.00774/) (2022)
- [3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction](https://mlsanity.com/paper/1604.00449/) (2016)
- [One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization](https://mlsanity.com/paper/2306.16928/) (2023)
- [Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model](https://mlsanity.com/paper/2310.15110/) (2023)
- [Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors](https://mlsanity.com/paper/2306.17843/) (2023)
- [Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior](https://mlsanity.com/paper/2303.14184/) (2023)
- [DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction](https://mlsanity.com/paper/1905.10711/) (2019)
- [Learning Category-Specific Mesh Reconstruction from Image Collections](https://mlsanity.com/paper/1803.07549/) (2018)
- [RealFusion: 360deg Reconstruction of Any Object from a Single Image](https://mlsanity.com/paper/2302.10663/) (2023)
- [One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion](https://mlsanity.com/paper/2311.07885/) (2023)
- [SparseFusion: Distilling View-conditioned Diffusion for 3D Reconstruction](https://mlsanity.com/paper/2212.00792/) (2022)
- [CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Model](https://mlsanity.com/paper/2403.05034/) (2024)

## Built on by (most cited first, 30 of 35)

- [Neural Gaffer: Relighting Any Object via Diffusion](https://mlsanity.com/paper/2406.07520/) (2024)
- [Bolt3D: Generating 3D Scenes in Seconds](https://mlsanity.com/paper/2503.14445/) (2025)
- [MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction Model](https://mlsanity.com/paper/2408.10198/) (2024)
- [Human-3Diffusion: Realistic Avatar Creation via Explicit 3D Consistent Diffusion Models](https://mlsanity.com/paper/2406.08475/) (2024)
- [MaterialMVP: Illumination-Invariant Material Generation via Multi-view PBR Diffusion](https://mlsanity.com/paper/2503.10289/) (2025)
- [Predicting 4D Hand Trajectory from Monocular Videos](https://mlsanity.com/paper/2501.08329/) (2025)
- [Robust CLIP-Based Detector for Exposing Diffusion Model-Generated Images](https://mlsanity.com/paper/2404.12908/) (2024)
- [PCDreamer: Point Cloud Completion Through Multi-view Diffusion Priors](https://mlsanity.com/paper/2411.19036/) (2024)
- [AR-1-to-3: Single Image to Consistent 3D Object Generation via Next-View Prediction](https://mlsanity.com/paper/2503.12929/) (2025)
- [Turbo3D: Ultra-fast Text-to-3D Generation](https://mlsanity.com/paper/2412.04470/) (2024)
- [STAR: Skeleton-aware Text-based 4D Avatar Generation with In-Network Motion Retargeting](https://mlsanity.com/paper/2406.04629/) (2024)
- [VideoPanda: Video Panoramic Diffusion with Multi-view Attention](https://mlsanity.com/paper/2504.11389/) (2025)
- [Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric Diffusion](https://mlsanity.com/paper/2501.18804/) (2025)
- [InstaScene: Towards Complete 3D Instance Decomposition and Reconstruction from Cluttered Scenes](https://mlsanity.com/paper/2507.08416/) (2025)
- [Sampling 3D Gaussian Scenes in Seconds with Latent Diffusion Models](https://mlsanity.com/paper/2406.13099/) (2024)
- [HouseCrafter: Lifting Floorplans to 3D Scenes with 2D Diffusion Model](https://mlsanity.com/paper/2406.20077/) (2024)
- [MVGBench: Comprehensive Benchmark for Multi-view Generation Models](https://mlsanity.com/paper/2507.00006/) (2025)
- [GeoRemover: Removing Objects and Their Causal Visual Artifacts](https://mlsanity.com/paper/2509.18538/) (2025)
- [ARM: Appearance Reconstruction Model for Relightable 3D Generation](https://mlsanity.com/paper/2411.10825/) (2024)
- [Direct and Explicit 3D Generation from a Single Image](https://mlsanity.com/paper/2411.10947/) (2024)
- [GarmentCrafter: Progressive Novel View Synthesis for Single-View 3D Garment Reconstruction and Editing](https://mlsanity.com/paper/2503.08678/) (2025)
- [GaussVideoDreamer: 3D Scene Generation with Video Diffusion and Inconsistency-Aware Gaussian Splatting](https://mlsanity.com/paper/2504.10001/) (2025)
- [Can3Tok: Canonical 3D Tokenization and Latent Modeling of Scene-Level 3D Gaussians](https://mlsanity.com/paper/2508.01464/) (2025)
- [ExploreGS: Explorable 3D Scene Reconstruction with Virtual Camera Samplings and Diffusion Priors](https://mlsanity.com/paper/2508.06014/) (2025)
- [Generating 360deg Video is What You Need For a 3D Scene](https://mlsanity.com/paper/2504.02045/) (2025)
- [Jointly Generating Multi-view Consistent PBR Textures using Collaborative Control](https://mlsanity.com/paper/2410.06985/) (2024)
- [Geometry and Perception Guided Gaussians for Multiview-consistent 3D Generation from a Single Image](https://mlsanity.com/paper/2506.21152/) (2025)
- [GaMO: Geometry-aware Multi-view Diffusion Outpainting for Sparse-View 3D Reconstruction](https://mlsanity.com/paper/2512.25073/) (2025)
- [SoPE: Spherical Coordinate-Based Positional Embedding for Enhancing Spatial Perception of 3D LVLMs](https://mlsanity.com/paper/2602.22716/) (2026)
- [Embodied Crowd Counting](https://mlsanity.com/paper/2503.08367/) (2025)

## Abstract (arXiv)

We present MVD-Fusion: a method for single-view 3D inference via generative modeling of multi-view-consistent RGB-D images. While recent methods pursuing 3D inference advocate learning novel-view generative models, these generations are not 3D-consistent and require a distillation process to generate a 3D output. We instead cast the task of 3D inference as directly generating mutually-consistent multiple views and build on the insight that additionally inferring depth can provide a mechanism for enforcing this consistency. Specifically, we train a denoising diffusion model to generate multi-view RGB-D images given a single RGB input image and leverage the (intermediate noisy) depth estimates to obtain reprojection-based conditioning to maintain multi-view consistency. We train our model using large-scale synthetic dataset Obajverse as well as the real-world CO3D dataset comprising of generic camera viewpoints. We demonstrate that our approach can yield more accurate synthesis compared to recent state-of-the-art, including distillation-based 3D inference and prior multi-view generation methods. We also evaluate the geometry induced by our multi-view depth prediction and find that it yields a more accurate representation than other direct 3D inference approaches.