mlsanityData to 7 Oct 2026

VFusion3D: Learning Scalable 3D Generative Models from Video Diffusion Models

arXiv 2403.12034 ↗First posted 18 Mar 2024cs.CV cs.GR cs.LGcited by 56

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

This paper presents a novel method for building scalable 3D generative models utilizing pre-trained video diffusion models. The primary obstacle in developing foundation 3D generative models is the limited availability of 3D data. Unlike images, texts, or videos, 3D data are not readily accessible and are difficult to acquire. This results in a significant disparity in scale compared to the vast quantities of other types of data. To address this issue, we propose using a video diffusion model, trained with extensive volumes of text, images, and videos, as a knowledge source for 3D data. By unlocking its multi-view generative capabilities through fine-tuning, we generate a large-scale synthetic multi-view dataset to train a feed-forward 3D generative model. The proposed model, VFusion3D, trained on nearly 3M synthetic multi-view data, can generate a 3D asset from a single image in seconds and achieves superior performance when compared to current SOTA feed-forward 3D generative models, with users preferring our results over 90% of the time.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 64 of its 73 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

64 of 73 references.

Closest ancestors · 16

Same direction, newest first

  1. 2023ReconFusion: 3D Reconstruction with Diffusion Priors5 Dec
  2. 2023ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models3 Dec
  3. 2023ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation2 Dec
  4. 2023One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion14 Nov
  5. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model10 Nov
  6. 2023GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models12 Oct
  7. 2023SweetDreamer: Aligning Geometric Priors in 2D Diffusion for Consistent Text-to-3D4 Oct
  8. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Image7 Sep
  9. 2023MVDream: Multi-view Diffusion for 3D Generation31 Aug
  10. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors30 Jun
  11. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization29 Jun
  12. 2023Viewset Diffusion: (0-)Image-Conditioned 3D Generative Models from 2D Data13 Jun
4 more
  1. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  2. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  3. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  4. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  11. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  12. 2022Scalable Diffusion Models with Transformerscited by 5,931
36 more
  1. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
  2. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  3. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  4. 2018Occupancy Networks: Learning 3D Reconstruction in Function Spacecited by 2,037
  5. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  6. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  7. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  8. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  9. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  10. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 767
  11. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 730
  12. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 601
  13. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  14. 2020U2-Net: Going Deeper with Nested U-Structure for Salient Object Detectioncited by 410
  15. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 379
  16. 2023Splatter Image: Ultra-Fast Single-View 3D Reconstructioncited by 359
  17. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278
  18. 2023Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformerscited by 257
  19. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  20. 2023MVImgNet: A Large-scale Dataset of Multi-view Imagescited by 219
  21. 2023DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Modelcited by 213
  22. 2023MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusioncited by 210
  23. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  24. 2023PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape Predictioncited by 165
  25. 2023HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusioncited by 152
  26. 2022Highly Accurate Dichotomous Image Segmentationcited by 110
  27. 2023DreamHuman: Animatable 3D Avatars from Textcited by 102
  28. 2021GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worldscited by 98
  29. 20233D-GPT: Procedural 3D Modeling with Large Language Modelscited by 91
  30. 2023ATT3D: Amortized Text-to-3D Object Synthesiscited by 87
  31. 2024IM-3D: Iterative Multiview Diffusion and Reconstruction for High-Quality 3D Generationcited by 84
  32. 2023ConsistNet: Enforcing 3D Consistency for Multi-view Images Diffusioncited by 65
  33. 2024SPAD : Spatially Aware Multiview Diffuserscited by 62
  34. 2024AGG: Amortized Generative 3D Gaussians for Single Image to 3Dcited by 51
  35. 2022RT-NeRF: Real-Time On-Device Neural Radiance Fields Towards Immersive AR/VR Renderingcited by 35
  36. 2024HexaGen3D: StableDiffusion is just one step away from Fast and Diverse Text-to-3D Generationcited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 31; 2025: 18; 2026: 7.

56 papers.

Sort:

By area:

  1. 2024InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Modelscited by 421
  2. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  3. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  4. 2024SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistencycited by 143
  5. 2024Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformerscited by 137
  6. 2025Test-Time Training Done Rightcited by 121
  7. 2024Training-free Camera Control for Video Generationcited by 119
  8. 2024Wonderland: Navigating 3D Scenes from a Single Imagecited by 98
  9. 2025MEt3R: Measuring Multi-View Consistency in Generated Imagescited by 91
  10. 20243D Gaussian Splatting: Survey, Technologies, Challenges, and Opportunitiescited by 77
  11. 2024TC4D: Trajectory-Conditioned Text-to-4D Generationcited by 76
  12. 2024Meta 3D AssetGen: Text-to-Mesh Generation with High-Quality Geometry, Texture, and PBR Materialscited by 66
38 more
  1. 2025Geo4D: Leveraging Video Generators for Geometric 4D Scene Reconstructioncited by 66
  2. 2024You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scalecited by 65
  3. 2024PartGen: Part-level 3D Generation and Reconstruction with Multi-View Diffusion Modelscited by 60
  4. 2024IDOL: Instant Photorealistic 3D Human Creation from a Single Imagecited by 55
  5. 2025DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian Splat Generationcited by 49
  6. 2024Vivid-ZOO: Multi-View Video Generation with Diffusion Modelcited by 40
  7. 2025PhysAnimator: Physics-Guided Generative Cartoon Animationcited by 40
  8. 2025Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillationcited by 37
  9. 2026Lyra 2.0: Explorable Generative 3D Worldscited by 36
  10. 2025AutoPartGen: Autogressive 3D Part Generation and Discoverycited by 33
  11. 2024Diffusion2: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Modelscited by 24
  12. 2025DSO: Aligning 3D Generators with Simulation Feedback for Physical Soundnesscited by 24
  13. 2024MegaSynth: Scaling Up 3D Scene Reconstruction with Synthesized Datacited by 21
  14. 2024Phidias: A Generative Model for Creating 3D Content from Text, Image, and 3D Conditions with Reference-Augmented Diffusioncited by 19
  15. 2025A Survey on 3D Gaussian Splatting Applications: Segmentation, Editing, and Generationcited by 19
  16. 2024Learning-based Multi-View Stereo: A Surveycited by 18
  17. 2024Atlas Gaussians Diffusion for 3D Generationcited by 15
  18. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applicationscited by 14
  19. 2024DiffusionBlend: Learning 3D Image Prior through Position-aware Diffusion Score Blending for 3D Computed Tomography Reconstructioncited by 13
  20. 2025Edit360: 2D Image Edits to 3D Assets from Any Anglecited by 13
  21. 2024Flex3D: Feed-Forward 3D Generation with Flexible Reconstruction Model and Input View Curationcited by 12
  22. 2024LiftImage3D: Lifting Any Single Image to 3D Gaussians with Video Generation Priorscited by 11
  23. 2025Can Video Diffusion Model Reconstruct 4D Geometry?cited by 10
  24. 2025Gen3DEval: Using vLLMs for Automatic Evaluation of Generated 3D Objectscited by 10
  25. 2024Semantic Score Distillation Sampling for Compositional Text-to-3D Generationcited by 9
  26. 2024KFC-W: Generating 3D-Consistent Videos from Unposed Internet Photoscited by 7
  27. 2024Bootstrap3D: Improving Multi-view Diffusion Model with Synthetic Datacited by 5
  28. 2024DreamBeast: Distilling 3D Fantastical Animals with Part-Aware Knowledge Transfercited by 5
  29. 2025Can3Tok: Canonical 3D Tokenization and Latent Modeling of Scene-Level 3D Gaussianscited by 5
  30. 2026NOVA3R: Non-pixel-aligned Visual Transformer for Amodal 3D Reconstructioncited by 5
  31. 2024Ctrl-GenAug: Controllable Generative Augmentation for Medical Sequence Classificationcited by 4
  32. 2024Meta 3D Gencited by 3
  33. 2025Distilling Multi-view Diffusion Models into 3D Generatorscited by 3
  34. 2024Improving Viewpoint Consistency in 3D Generation via Structure Feature and CLIP Guidancecited by 2
  35. 2024Vid3D: Synthesis of Dynamic 3D Scenes using 2D Video Diffusioncited by 1
  36. 2024MVBoost: Boost 3D Reconstruction with Multi-View Refinementcited by 1
  37. 2025GenVDM: Generating Vector Displacement Maps From a Single Imagecited by 1
  38. 2025Droplet3D: Commonsense Priors from Videos Facilitate 3D Generationcited by 1

50 most cited of 56 All 56 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generation
  2. 20243D-VirtFusion: Synthetic 3D Data Augmentation through Generative Diffusion Models and Controllable Editing
  3. 2025VideoScene: Distilling Video Diffusion Model to Generate 3D Scenes in One Step
  4. 2024GANFusion: Feed-Forward Text-to-3D with Diffusion in GAN Space
  5. 2024V3D: Video Diffusion Models are Effective 3D Generators
  6. 2023WildFusion: Learning 3D-Aware Latent Diffusion Models in View Space
  7. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion
  8. 2024Gen-3Diffusion: Realistic Image-to-3D Generation via 2D & 3D Diffusion Synergy