mlsanityData to 7 Oct 2026

4Diffusion: Multi-view Video Diffusion Model for 4D Generation

arXiv 2405.20674 ↗First posted 31 May 2024cs.CVcited by 104

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Current 4D generation methods have achieved noteworthy efficacy with the aid of advanced diffusion generative models. However, these methods lack multi-view spatial-temporal modeling and encounter challenges in integrating diverse prior knowledge from multiple diffusion models, resulting in inconsistent temporal appearance and flickers. In this paper, we propose a novel 4D generation pipeline, namely 4Diffusion, aimed at generating spatial-temporally consistent 4D content from a monocular video. We first design a unified diffusion model tailored for multi-view video generation by incorporating a learnable motion module into a frozen 3D-aware diffusion model to capture multi-view spatial-temporal correlations. After training on a curated dataset, our diffusion model acquires reasonable temporal consistency and inherently preserves the generalizability and spatial consistency of the 3D-aware diffusion model. Subsequently, we propose 4D-aware Score Distillation Sampling loss, which is based on our multi-view video diffusion model, to optimize 4D representation parameterized by dynamic NeRF. This aims to eliminate discrepancies arising from multiple diffusion models, allowing for generating spatial-temporally consistent 4D content. Moreover, we devise an anchor loss to enhance the appearance details and facilitate the learning of dynamic NeRF. Extensive qualitative and quantitative experiments demonstrate that our method achieves superior performance compared to previous methods.

Key facts. Cited by 104 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 59 of its 67 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

59 of 67 references.

Closest ancestors · 24

Same direction, newest first

  1. 2024STAG4D: Spatial-Temporal Anchored Generative 4D Gaussians22 Mar
  2. 2024V3D: Video Diffusion Models are Effective 3D Generators11 Mar
  3. 20234DGen: Grounded 4D Content Generation with Spatial-temporal Consistency28 Dec
  4. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models21 Dec
  5. 2023Mosaic-SDF for 3D Generative Models14 Dec
  6. 2023A Unified Approach for Text- and Image-guided 4D Scene Generation28 Nov
  7. 2023Animate124: Animating One Image to 4D Dynamic Scene24 Nov
  8. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model10 Nov
  9. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusion23 Oct
  10. 2023Consistent123: One Image to Highly Consistent 3D Asset Using Case-Aware Diffusion Priors29 Sep
  11. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Image7 Sep
  12. 2023MVDream: Multi-view Diffusion for 3D Generation31 Aug
12 more
  1. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors30 Jun
  2. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  3. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior24 Mar
  4. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation24 Mar
  5. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  6. 20233DGen: Triplane Latent Diffusion for Textured Mesh Generation9 Mar
  7. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion12 Dec
  8. 2022DiffRF: Rendering-Guided 3D Radiance Field Diffusion2 Dec
  9. 20223D Neural Field Generation using Triplane Diffusion30 Nov
  10. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  11. 2022RenderDiffusion: Image Diffusion for 3D Reconstruction, Inpainting and Generation17 Nov
  12. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  6. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  7. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  8. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  9. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  10. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  11. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  12. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
23 more
  1. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  2. 20234D Gaussian Splatting for Real-Time Dynamic Scene Renderingcited by 1,144
  3. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  4. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  5. 2023K-Planes: Explicit Radiance Fields in Space, Time, and Appearancecited by 742
  6. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 730
  7. 2023HexPlane: A Fast Representation for Dynamic Scenescited by 582
  8. 2023SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenescited by 339
  9. 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 305
  10. 2023Triplane Meets Gaussian Splatting: Fast and Generalizable Single-View 3D Reconstruction with Transformerscited by 257
  11. 2023DreamBooth3D: Subject-Driven Text-to-3D Generationcited by 235
  12. 2023DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Modelcited by 213
  13. 2023Text-To-4D Dynamic Scene Generationcited by 204
  14. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  15. 2023DreamGaussian4D: Generative 4D Gaussian Splattingcited by 191
  16. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 184
  17. 2023Swap Attention in Spatiotemporal Diffusions for Text-to-Video Generationcited by 153
  18. 2023HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusioncited by 152
  19. 2023Consistent4D: Consistent 360deg Dynamic Object Generation from Monocular Videocited by 124
  20. 2023NerfAcc: Efficient Sampling Accelerates NeRFscited by 90
  21. 2023HumanGaussian: Text-Driven 3D Human Generation with Gaussian Splattingcited by 89
  22. 2023ConsistNet: Enforcing 3D Consistency for Multi-view Images Diffusioncited by 65
  23. 2024SC4D: Sparse-Controlled Video-to-4D Generation and Motion Transfercited by 61

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 56; 2026: 32.

104 papers.

Sort:

By area:

  1. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  2. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  3. 2024SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistencycited by 143
  4. 2024SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpointscited by 92
  5. 2024ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuningcited by 68
  6. 2024Animate3D: Animating Any 3D Model with Multi-view Video Diffusioncited by 55
  7. 2024InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Modelscited by 53
  8. 2025SV4D 2.0: Enhancing Spatio-Temporal Consistency in Multi-View Video Diffusion for High-Quality 4D Generationcited by 44
  9. 2025Puppeteer: Rig and Animate Your 3D Modelscited by 41
  10. 2024Vivid-ZOO: Multi-View Video Generation with Diffusion Modelcited by 40
  11. 2025Motion Anything: Any to Motion Generationcited by 40
  12. 2025Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillationcited by 37
38 more
  1. 2025DeepVerse: 4D Autoregressive Video Generation as a World Modelcited by 36
  2. 2025Gaussian Variation Field Diffusion for High-fidelity Video-to-4D Synthesiscited by 33
  3. 2025AnimateAnyMesh: A Feed-Forward 4D Foundation Model for Text-Driven Universal Mesh Animationcited by 31
  4. 2025FlashWorld: High-quality 3D Scene Generation within Secondscited by 30
  5. 2024Puppet-Master: Scaling Interactive Video Generation as a Motion Prior for Part-Level Dynamicscited by 28
  6. 20244Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusioncited by 27
  7. 2025DreamArt: Generating Interactable Articulated Objects from a Single Imagecited by 25
  8. 2024Cavia: Camera-controllable Multi-view Video Diffusion with View-Integrated Attentioncited by 24
  9. 2024Phys4DGen: Physics-Compliant 4D Generation with Multi-Material Composition Perceptioncited by 23
  10. 2025Geometry-aware 4D Video Generation for Robot Manipulationcited by 23
  11. 2025FreeArt3D: Training-Free Articulated Object Generation using 3D Diffusioncited by 22
  12. 2025ShapeGen4D: Towards High Quality 4D Shape Generation from Videoscited by 20
  13. 2024Trans4D: Realistic Geometry-Aware Transition for Compositional Text-to-4D Synthesiscited by 19
  14. 2025Diffuman4D: 4D Consistent Human View Synthesis from Sparse-View Videos with Spatio-Temporal Diffusion Modelscited by 17
  15. 2025MVP4D: Multi-View Portrait Video Diffusion for Animatable 4D Avatarscited by 17
  16. 20244Dynamic: Text-to-4D Generation with Hybrid Priorscited by 15
  17. 2026Motion 3-to-4: 3D Motion Reconstruction for 4D Synthesiscited by 15
  18. 2026ActionMesh: Animated 3D Mesh Generation with Temporal 3D Diffusioncited by 15
  19. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Modelscited by 14
  20. 20254D-LRM: Large Space-Time Reconstruction Model From and To Any View at Any Timecited by 13
  21. 2025Reconstructing 4D Spatial Intelligence: A Surveycited by 13
  22. 20254Real-Video-V2: Fused View-Time Attention and Feedforward Reconstruction for 4D Scene Generationcited by 12
  23. 2025Towards Cross-View Point Correspondence in Vision-Language Modelscited by 12
  24. 2025BulletTime: Decoupled Control of Time and Camera Pose for Video Generationcited by 12
  25. 2026Mesh4D: 4D Mesh Reconstruction and Tracking from Monocular Videocited by 12
  26. 2025Not All Frame Features Are Equal: Video-to-4D Generation via Decoupling Dynamic-Static Featurescited by 10
  27. 2025Can Video Diffusion Model Reconstruct 4D Geometry?cited by 10
  28. 2025Uni4D-LLM: A Unified SpatioTemporal-Aware VLM for 4D Understanding and Generationcited by 10
  29. 2026Action Images: End-to-End Policy Learning via Multiview Video Generationcited by 10
  30. 2025Image Can Bring Your Memory Back: A Novel Multi-Modal Guided Attack against Image Generation Model Unlearningcited by 9
  31. 20254D-Bench: Benchmarking Multi-modal Large Language Models for 4D Object Understandingcited by 8
  32. 2025Splat4D: Diffusion-Enhanced 4D Gaussian Splatting for Temporally and Spatially Consistent Content Creationcited by 8
  33. 2025PhysGM: Large Physical Gaussian Model for Feed-Forward 4D Synthesiscited by 8
  34. 2025Articulate That Object Part (ATOP): 3D Part Articulation via Text and Motion Personalizationcited by 7
  35. 2025In-2-4D: Inbetweening from Two Single-View Images to 4D Generationcited by 7
  36. 2025SS4D: Native 4D Generative Model via Structured Spacetime Latentscited by 7
  37. 2025DIMO: Diverse 3D Motion Generation for Arbitrary Objectscited by 6
  38. 2025ByteLoom: Weaving Geometry-Consistent Human-Object Interactions through Progressive Curriculum Learningcited by 6

50 most cited of 104 All 104 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models
  2. 2024Diffusion2: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models
  3. 2025Zero4D: Training-Free 4D Video Generation From Single Video Using Off-the-Shelf Video Diffusion
  4. 20254DNeX: Feed-Forward 4D Generative Modeling Made Easy
  5. 2024CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models
  6. 2025Diff4Splat: Controllable 4D Scene Generation with Latent Dynamic Reconstruction Models
  7. 2025Free4D: Tuning-free 4D Scene Generation with Spatial-Temporal Consistency
  8. 2025VideoScene: Distilling Video Diffusion Model to Generate 3D Scenes in One Step