mlsanityData to 7 Oct 2026

ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis

arXiv 2409.02048 ↗First posted 3 Sep 2024cs.CVcited by 392

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Despite recent advancements in neural 3D reconstruction, the dependence on dense multi-view captures restricts their broader applicability. In this work, we propose ViewCrafter, a novel method for synthesizing high-fidelity novel views of generic scenes from single or sparse images with the prior of video diffusion model. Our method takes advantage of the powerful generation capabilities of video diffusion model and the coarse 3D clues offered by point-based representation to generate high-quality video frames with precise camera pose control. To further enlarge the generation range of novel views, we tailored an iterative view synthesis strategy together with a camera trajectory planning algorithm to progressively extend the 3D clues and the areas covered by the novel views. With ViewCrafter, we can facilitate various applications, such as immersive experiences with real-time rendering by efficiently optimizing a 3D-GS representation using the reconstructed 3D points and the generated novel views, and scene-level text-to-3D generation for more imaginative content creation. Extensive experiments on diverse datasets demonstrate the strong generalization capability and superior performance of our method in synthesizing high-fidelity and consistent novel views.

Key facts. Cited by 392 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 66 of its 78 references that are in the map. First posted on arXiv 3 Sep 2024.

Builds on

66 of 78 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024MultiDiff: Consistent Novel View Synthesis from a Single Image26 Jun
  2. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Models16 May
  3. 2024RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth Diffusion10 Apr
  4. 2023ReconFusion: 3D Reconstruction with Diffusion Priors5 Dec
  5. 2023DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior25 Oct
  6. 2023HiFi-123: Towards High-fidelity One Image to 3D Content Generation10 Oct
  7. 2023Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields19 May
  8. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Models5 Apr
  9. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior24 Mar
  10. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  11. 2022Novel View Synthesis with Diffusion Models6 Oct

Other references · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  9. 2022Classifier-Free Diffusion Guidancecited by 5,717
  10. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  11. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  12. 2015ShapeNet: An Information-Rich 3D Model Repositorycited by 2,880
38 more
  1. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  2. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  3. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
  4. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  5. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  6. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  7. 2024Grounding Image Matching in 3D with MASt3Rcited by 960
  8. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  9. 2020Accelerating 3D Deep Learning with PyTorch3Dcited by 781
  10. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  11. 2023pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstructioncited by 668
  12. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
  13. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  14. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  15. 2024MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Imagescited by 553
  16. 2023DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Visioncited by 514
  17. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  18. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  19. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  20. 2019SynSin: End-to-end View Synthesis from a Single Imagecited by 364
  21. 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 330
  22. 2023FSGS: Real-Time Few-shot View Synthesis using Gaussian Splattingcited by 278
  23. 2020Single-View View Synthesis with Multiplane Imagescited by 263
  24. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 260
  25. 2024DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalizationcited by 256
  26. 2023LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenescited by 228
  27. 2023MVImgNet: A Large-scale Dataset of Multi-view Imagescited by 219
  28. 20203D Photography using Context-aware Layered Depth Inpaintingcited by 206
  29. 2024CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generationcited by 163
  30. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  31. 2023Make-Your-Video: Customized Video Generation Using Textual and Structural Guidancecited by 129
  32. 2021Geometry-Free View Synthesis: Transformers and no 3D Priorscited by 99
  33. 2024MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Modelcited by 87
  34. 2021PixelSynth: Generating a 3D-Consistent Experience from a Single Imagecited by 82
  35. 2024ToonCrafter: Generative Cartoon Interpolationcited by 68
  36. 2023Real-Time Radiance Fields for Single-Image Portrait View Synthesiscited by 64
  37. 2022Single-View View Synthesis in the Wild with Learned Adaptive Multiplane Imagescited by 59
  38. 2023NeU-NBV: Next Best View Planning Using Uncertainty Estimation in Image-Based Neural Renderingcited by 53

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 31; 2025: 179; 2026: 182.

392 papers.

Sort:

By area:

  1. 2025GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Controlcited by 325
  2. 2025Stable Virtual Camera: Generative View Synthesis with Diffusion Modelscited by 179
  3. 2024LVSM: A Large View Synthesis Model with Minimal 3D Inductive Biascited by 178
  4. 2024CAT4D: Create Anything in 4D with Multi-View Video Diffusion Modelscited by 160
  5. 2025Difix3D+: Improving 3D Reconstructions with Single-Step Diffusion Modelscited by 160
  6. 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrievalcited by 157
  7. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 155
  8. 2025WorldMem: Long-term Consistent World Simulation with Memorycited by 152
  9. 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modelingcited by 152
  10. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  11. 2024DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusioncited by 139
  12. 2025Video World Models with Long-term Spatial Memorycited by 124
38 more
  1. 2025TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Modelscited by 120
  2. 2025Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noisecited by 114
  3. 2024MVSplat360: Feed-Forward 360 Scene Synthesis from Sparse Viewscited by 111
  4. 2025VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed View Memorycited by 102
  5. 2024Wonderland: Navigating 3D Scenes from a Single Imagecited by 98
  6. 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modelingcited by 86
  7. 2025Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls for Video Generationcited by 75
  8. 2025SkyReels-A2: Compose Anything in Video Diffusion Transformerscited by 72
  9. 2025SpatialVID: A Large-Scale Video Dataset with Spatial Annotationscited by 72
  10. 2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 71
  11. 2024ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuningcited by 68
  12. 2025Geo4D: Leveraging Video Generators for Geometric 4D Scene Reconstructioncited by 66
  13. 2024You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scalecited by 65
  14. 2025Follow-Your-Creation: Empowering 4D Creation through Video Inpaintingcited by 61
  15. 2025Bolt3D: Generating 3D Scenes in Secondscited by 57
  16. 2024FreeVS: Generative View Synthesis on Free Driving Trajectorycited by 52
  17. 2025MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generationcited by 50
  18. 2026SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformercited by 50
  19. 2024StreetCrafter: Street View Synthesis with Controllable Video Diffusion Modelscited by 49
  20. 2025VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Priorcited by 49
  21. 2025SV4D 2.0: Enhancing Spatio-Temporal Consistency in Multi-View Video Diffusion for High-Quality 4D Generationcited by 44
  22. 2025ATI: Any Trajectory Instruction for Controllable Video Generationcited by 42
  23. 2025GenFusion: Closing the Loop between Reconstruction and Generation via Videoscited by 41
  24. 2025Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulationcited by 41
  25. 2024Efficient Diffusion Models: A Comprehensive Survey from Principles to Practicescited by 39
  26. 2025DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generationcited by 39
  27. 2026NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videoscited by 39
  28. 2024VistaDream: Sampling multiview consistent images for single-view scene reconstructioncited by 38
  29. 2025FlexWorld: Progressively Expanding 3D Scenes for Flexiable-View Synthesiscited by 37
  30. 2025Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillationcited by 37
  31. 2025GeometryCrafter: Consistent Geometry Estimation for Open-world Videos with Diffusion Priorscited by 36
  32. 2025GSFixer: Improving 3D Gaussian Splatting with Reference-Guided Video Diffusion Priorscited by 36
  33. 2026Lyra 2.0: Explorable Generative 3D Worldscited by 36
  34. 2025Reangle-A-Video: 4D Video Generation as Video-to-Video Translationcited by 35
  35. 2024SplatFlow: Multi-View Rectified Flow Model for 3D Gaussian Splatting Synthesiscited by 34
  36. 2025Free4D: Tuning-free 4D Scene Generation with Spatial-Temporal Consistencycited by 34
  37. 2024Prometheus: 3D-Aware Latent Diffusion Models for Feed-Forward Text-to-3D Scene Generationcited by 33
  38. 2025Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Controlcited by 33

50 most cited of 392 All 392 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models
  2. 2023Consistent View Synthesis with Pose-Guided Diffusion Models
  3. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion
  4. 2024NVS-Solver: Video Diffusion Model as Zero-Shot Novel View Synthesizer
  5. 2025DT-NVS: Diffusion Transformers for Novel View Synthesis
  6. 2025VideoFrom3D: 3D Scene Video Generation via Complementary Image and Video Diffusion Models
  7. 2026ArtiFixer: Enhancing and Extending 3D Reconstruction with Auto-Regressive Diffusion Models
  8. 2026Efficient Camera-Controlled Video Generation of Static Scenes via Sparse Diffusion and 3D Rendering