mlsanityData to 7 Oct 2026

VividDream: Generating 3D Scene with Ambient Dynamics

arXiv 2405.20334 ↗First posted 30 May 2024cs.CV cs.GRcited by 21

Not on the map. Why?

Abstract (arXiv)

We introduce VividDream, a method for generating explorable 4D scenes with ambient dynamics from a single input image or text prompt. VividDream first expands an input image into a static 3D point cloud through iterative inpainting and geometry merging. An ensemble of animated videos is then generated using video diffusion models with quality refinement techniques and conditioned on renderings of the static 3D scene from the sampled camera trajectories. We then optimize a canonical 4D scene representation using an animated video ensemble, with per-video motion embeddings and visibility masks to mitigate inconsistencies. The resulting 4D scene enables free-view exploration of a 3D scene with plausible ambient scene dynamics. Experiments demonstrate that VividDream can provide human viewers with compelling 4D experiences generated based on diverse real images and text prompts.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 58 of its 63 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

58 of 63 references.

References · 58

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  5. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,736
  8. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  9. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  10. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  11. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  12. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
38 more
  1. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  2. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  3. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,419
  4. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  5. 2024Depth Anything: Unleashing the Power of Large-Scale Unlabeled Datacited by 1,346
  6. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  7. 2020NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collectionscited by 1,230
  8. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  9. 20234D Gaussian Splatting for Real-Time Dynamic Scene Renderingcited by 1,144
  10. 2023DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creationcited by 837
  11. 2023K-Planes: Explicit Radiance Fields in Space, Time, and Appearancecited by 742
  12. 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 731
  13. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
  14. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  15. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  16. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  17. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  18. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  19. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 330
  20. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  21. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  22. 2022DynIBaR: Neural Dynamic Image-Based Renderingcited by 224
  23. 2023Robust Dynamic Radiance Fieldscited by 193
  24. 2023DreamGaussian4D: Generative 4D Gaussian Splattingcited by 191
  25. 2020Robust Consistent Video Depth Estimationcited by 188
  26. 2024PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generationcited by 185
  27. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 184
  28. 2023SceneScape: Text-Driven Consistent Scene Generationcited by 160
  29. 2023WonderJourney: Going from Anywhere to Everywherecited by 157
  30. 2022FILM: Frame Interpolation for Large Motioncited by 149
  31. 2024GALA3D: Towards Text-to-3D Complex Scene Generation via Layout-guided Generative Gaussian Splattingcited by 104
  32. 2024STAG4D: Spatial-Temporal Anchored Generative 4D Gaussianscited by 102
  33. 2023Generative Image Dynamicscited by 93
  34. 2023A Unified Approach for Text- and Image-guided 4D Scene Generationcited by 82
  35. 2023Set-the-Scene: Global-Local Training for Generating Controllable NeRF Scenescited by 75
  36. 2024DreamScene4D: Dynamic Multi-Object Scene Generation from Monocular Videoscited by 63
  37. 2020Animating Pictures with Eulerian Motion Fieldscited by 54
  38. 2023SceneWiz3D: Towards Text-guided 3D Scene Compositioncited by 54

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 12; 2026: 5.

21 papers.

Sort:

By area:

  1. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  2. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  3. 2025WorldScore: A Unified Evaluation Benchmark for World Generationcited by 143
  4. 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modelingcited by 86
  5. 2025WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actionscited by 43
  6. 2025Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillationcited by 37
  7. 2024DynamicScaler: Seamless and Scalable Video Generation for Panoramic Scenescited by 24
  8. 2025Generative Physical AI in Vision: A Surveycited by 24
  9. 20253D Scene Generation: A Surveycited by 24
  10. 2025Scene Splatter: Momentum 3D Scene Generation from Single Image with Video Diffusion Modelcited by 13
  11. 2025SOPHY: Learning to Generate Simulation-Ready Objects with Physical Materialscited by 5
  12. 2026Video Generation Models as World Models: Efficient Paradigms, Architectures and Algorithmscited by 5
9 more
  1. 2025Optimizing 4D Gaussians for Dynamic Scene Video from Single Landscape Imagescited by 4
  2. 2025OmniX: From Unified Panoramic Generation and Perception to Graphics-Ready 3D Scenescited by 4
  3. 2024PaintScene4D: Consistent 4D Scene Generation from Text Promptscited by 2
  4. 2026VRGaussianAvatar: Integrating 3D Gaussian Avatars into VRcited by 1
  5. 2026AniGS: Bridging Rendering and Diffusion Prior for 3D Scene Animationcited by 1
  6. 2025WonderVerse: Extendable 3D Scene Generation with Video Generative Modelscited by 0
  7. 2025Object-Aware 4D Human Motion Generationcited by 0
  8. 2026S-Avatar: Diffusion-Guided Gaussian Head Avatars from a Single Imagecited by 0
  9. 2026OuroWorld: Bringing Any 3D World Alive as Diverse, Endlessly Looping 3D Cinemagraphscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025DreamJourney: Perpetual View Generation with Video Diffusion Models
  2. 2025CoCo4D: Comprehensive and Complex 4D Scene Generation
  3. 20244Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models
  4. 2024Bringing Objects to Life: training-free 4D generation from 3D objects through view consistent noise
  5. 2024Vid3D: Synthesis of Dynamic 3D Scenes using 2D Video Diffusion
  6. 2025VividCam: Learning Unconventional Camera Motions from Virtual Synthetic Videos
  7. 2024SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency
  8. 2026CP4D: Compositional Physics-aware 4D Scene Generation