mlsanityData to 14 Oct 2026

4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models

arXiv 2406.07472 ↗First posted 11 Jun 2024cs.CVcited by 64

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Existing dynamic scene generation methods mostly rely on distilling knowledge from pre-trained 3D generative models, which are typically fine-tuned on synthetic object datasets. As a result, the generated scenes are often object-centric and lack photorealism. To address these limitations, we introduce a novel pipeline designed for photorealistic text-to-4D scene generation, discarding the dependency on multi-view generative models and instead fully utilizing video generative models trained on diverse real-world datasets. Our method begins by generating a reference video using the video generation model. We then learn the canonical 3D representation of the video using a freeze-time video, delicately generated from the reference video. To handle inconsistencies in the freeze-time video, we jointly learn a per-frame deformation to model these imperfections. We then learn the temporal deformation based on the canonical representation to capture dynamic interactions in the reference video. The pipeline facilitates the generation of dynamic scenes with enhanced photorealism and structural integrity, viewable from multiple perspectives, thereby setting a new standard in 4D scene generation.

Key facts. Cited by 64 papers in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 65 of its 75 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

65 of 75 references.

Closest ancestors · 20

Same direction, newest first

  1. 2024STAG4D: Spatial-Temporal Anchored Generative 4D Gaussians22 Mar
  2. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion18 Mar
  3. 2024Diffusion Priors for Dynamic View Synthesis from Monocular Videos10 Jan
  4. 20234DGen: Grounded 4D Content Generation with Spatial-temporal Consistency28 Dec
  5. 2023Align Your Gaussians: Text-to-4D with Dynamic 3D Gaussians and Composed Diffusion Models21 Dec
  6. 2023A Unified Approach for Text- and Image-guided 4D Scene Generation28 Nov
  7. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Model10 Nov
  8. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusion23 Oct
  9. 2023Consistent123: Improve Consistency for One Image to 3D Object Synthesis12 Oct
  10. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Image7 Sep
  11. 2023MVDream: Multi-view Diffusion for 3D Generation31 Aug
  12. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors30 Jun
8 more
  1. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  2. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation24 Mar
  3. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  4. 2022Point-E: A System for Generating 3D Point Clouds from Complex Prompts16 Dec
  5. 2022Rodin: A Generative Model for Sculpting 3D Digital Avatars Using Diffusion12 Dec
  6. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation1 Dec
  7. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  8. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 45

  1. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  2. 2022Scalable Diffusion Models with Transformerscited by 6,054
  3. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  4. 2022Video Diffusion Modelscited by 2,305
  5. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  6. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  7. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  8. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  9. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,465
  10. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  11. 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,198
  12. 20234D Gaussian Splatting for Real-Time Dynamic Scene Renderingcited by 1,157
33 more
  1. 2018Learning Implicit Fields for Generative Shape Modelingcited by 1,060
  2. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 849
  3. 2023Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstructioncited by 759
  4. 2020Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenescited by 751
  5. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 739
  6. 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 737
  7. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 580
  8. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Modelscited by 579
  9. 2021Dynamic View Synthesis from Dynamic Monocular Videocited by 420
  10. 2020Space-time Neural Irradiance Fields for Free-Viewpoint Videocited by 414
  11. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 385
  12. 2022Fast Dynamic Radiance Fields with Time-Aware Neural Voxelscited by 346
  13. 2023ControlVideo: Training-free Controllable Text-to-Video Generationcited by 327
  14. 2022Expanding Language-Image Pretrained Models for General Video Recognitioncited by 321
  15. 2022SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generationcited by 288
  16. 2020Neural Radiance Flow for 4D View Synthesis and Video Processingcited by 270
  17. 2024CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Modelcited by 215
  18. 2023DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Modelcited by 214
  19. 2023Text-To-4D Dynamic Scene Generationcited by 205
  20. 2023DreamGaussian4D: Generative 4D Gaussian Splattingcited by 194
  21. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 185
  22. 2024Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesiscited by 105
  23. 2021Neural Trajectory Fields for Dynamic Novel View Synthesiscited by 86
  24. 2023InfiniCity: Infinite-Scale City Synthesiscited by 68
  25. 2017Improved Adversarial Systems for 3D Object Generation and Reconstructioncited by 66
  26. 2023Reconstructing Animatable Categories from Videoscited by 47
  27. 2023Flow supervision for Deformable NeRFcited by 42
  28. 2023CoGS: Controllable Gaussian Splattingcited by 32
  29. 2022Cross-Modal 3D Shape Generation and Manipulationcited by 28
  30. 2024Motion-Zero: Zero-Shot Moving Object Control Framework for Diffusion-Based Video Generationcited by 25
  31. 2023Text-Guided Synthesis of Eulerian Cinemagraphscited by 22
  32. 2023FIT: Far-reaching Interleaved Transformerscited by 16
  33. 2024GTR: Improving Large 3D Reconstruction Models through Geometry and Texture Refinementcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 17; 2025: 33; 2026: 14.

64 papers.

Sort:

By area:

  1. 2024CAT4D: Create Anything in 4D with Multi-View Video Diffusion Modelscited by 161
  2. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  3. 2025WorldScore: A Unified Evaluation Benchmark for World Generationcited by 156
  4. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 154
  5. 2024DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusioncited by 140
  6. 2025Controllable Video Generation: A Surveycited by 98
  7. 2024ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuningcited by 70
  8. 2024InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Modelscited by 56
  9. 2025SV4D 2.0: Enhancing Spatio-Temporal Consistency in Multi-View Video Diffusion for High-Quality 4D Generationcited by 48
  10. 2025Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillationcited by 41
  11. 2024ResearchTown: Simulator of Human Research Communitycited by 37
  12. 2024DreamDrive: Generative 4D Scene Modeling from Street View Imagescited by 35
38 more
  1. 2025Free4D: Tuning-free 4D Scene Generation with Spatial-Temporal Consistencycited by 35
  2. 20254DNeX: Feed-Forward 4D Generative Modeling Made Easycited by 34
  3. 20244Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusioncited by 27
  4. 20253D Scene Generation: A Surveycited by 24
  5. 2025EPiC: Efficient Video Camera Control Learning with Precise Anchor-Video Guidancecited by 21
  6. 2025ShapeGen4D: Towards High Quality 4D Shape Generation from Videoscited by 21
  7. 2024Trans4D: Realistic Geometry-Aware Transition for Compositional Text-to-4D Synthesiscited by 19
  8. 2025Compositional Generative Model of Unbounded 4D Citiescited by 17
  9. 2025Zero4D: Training-Free 4D Video Generation From Single Video Using Off-the-Shelf Video Diffusioncited by 16
  10. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Modelscited by 15
  11. 2026AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memoriescited by 15
  12. 20254Real-Video-V2: Fused View-Time Attention and Feedforward Reconstruction for 4D Scene Generationcited by 14
  13. 2025WideRange4D: Enabling High-Quality 4D Reconstruction with Wide-Range Movements and Scenescited by 11
  14. 2024Gaussians-to-Life: Text-Driven Animation of 3D Gaussian Splatting Scenescited by 9
  15. 2025SS4D: Native 4D Generative Model via Structured Spacetime Latentscited by 8
  16. 2024Birth and Death of a Rosecited by 7
  17. 2025Diff4Splat: Controllable 4D Scene Generation with Latent Dynamic Reconstruction Modelscited by 7
  18. 2026RynnWorld-4D: 4D Embodied World Models for Robotic Manipulationcited by 7
  19. 2024DissolveStereo: Coarse Depth Injection for Zero-Shot Stereo Video Generationcited by 6
  20. 2025Where and How to Perturb: On the Design of Perturbation Guidance in Diffusion and Flow Modelscited by 6
  21. 2024Bringing Objects to Life: training-free 4D generation from 3D objects through view consistent noisecited by 5
  22. 2026Choreographing a World of Dynamic Objectscited by 5
  23. 2026UFO-4D: Unposed Feedforward 4D Reconstruction from Two Imagescited by 5
  24. 2026Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithmscited by 5
  25. 2024Human Action CLIPs: Detecting AI-generated Human Motioncited by 4
  26. 2025PromptVFX: Text-Driven Fields for Open-World 3D Gaussian Animationcited by 4
  27. 2025ViDAR: Video Diffusion-Aware 4D Reconstruction From Monocular Inputscited by 4
  28. 2025CoCo4D: Comprehensive and Complex 4D Scene Generationcited by 4
  29. 2025Advances in 4D Representation: Geometry, Motion, and Interactioncited by 4
  30. 2025OmniX: From Unified Panoramic Generation and Perception to Graphics-Ready 3D Scenescited by 4
  31. 2026BiMotion: B-spline Motion for Text-guided Dynamic 3D Character Generationcited by 4
  32. 2024Towards a Science Exocortexcited by 3
  33. 2025FB-4D: Spatial-Temporal Coherent Dynamic 3D Content Generation with Feature Bankscited by 3
  34. 2025PanoLora: Bridging Perspective and Panoramic Video Generation with LoRA Adaptationcited by 3
  35. 2026i-PhysGaussian: Implicit Physical Simulation for 3D Gaussian Splattingcited by 3
  36. 2026Guiding a Diffusion Model by Swapping Its Tokenscited by 3
  37. 2026Sculpt4D: Generating 4D Shapes via Sparse-Attention Diffusion Transformerscited by 3
  38. 2024PaintScene4D: Consistent 4D Scene Generation from Text Promptscited by 2

50 most cited of 64 All 64 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 20244Dynamic: Text-to-4D Generation with Hybrid Priors
  2. 2024SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency
  3. 2026Beyond Pixels: From Video Priors to 4D Worlds
  4. 2024Vid3D: Synthesis of Dynamic 3D Scenes using 2D Video Diffusion
  5. 2025WorldReel: 4D Video Generation with Consistent Geometry and Motion Modeling
  6. 20244Diffusion: Multi-view Video Diffusion Model for 4D Generation
  7. 2024Tex4D: Zero-shot 4D Scene Texturing with Video Diffusion Models
  8. 2024Phy124: Fast Physics-Driven 4D Content Generation from a Single Image