mlsanityData to 14 Oct 2026

Controlling Space and Time with Diffusion Models

arXiv 2407.07860 ↗First posted 10 Jul 2024cs.CVcited by 66

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

We present 4DiM, a cascaded diffusion model for 4D novel view synthesis (NVS), supporting generation with arbitrary camera trajectories and timestamps, in natural scenes, conditioned on one or more images. With a novel architecture and sampling procedure, we enable training on a mixture of 3D (with camera pose), 4D (pose+time) and video (time but no pose) data, which greatly improves generalization to unseen images and camera pose trajectories over prior works that focus on limited domains (e.g., object centric). 4DiM is the first-ever NVS method with intuitive metric-scale camera pose control enabled by our novel calibration pipeline for structure-from-motion-posed data. Experiments demonstrate that 4DiM outperforms prior 3D NVS models both in terms of image fidelity and pose alignment, while also enabling the generation of scene dynamics. 4DiM provides a general framework for a variety of tasks including single-image-to-3D, two-image-to-video (interpolation and extrapolation), and pose-conditioned video-to-video translation, which we illustrate qualitatively on a variety of scenes. For an overview see https://4d-diffusion.github.io

Key facts. Cited by 66 papers in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 50 of its 67 references that are in the map. First posted on arXiv 10 Jul 2024.

Builds on

50 of 67 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Models16 May
  2. 2024MVDiffusion++: A Dense High-resolution Multi-view Diffusion Model for Single or Sparse-view 3D Object Reconstruction20 Feb
  3. 2023Zero-Shot Metric Depth with a Field-of-View Conditioned Diffusion Model20 Dec
  4. 2023ReconFusion: 3D Reconstruction with Diffusion Priors5 Dec
  5. 2023Light Field Diffusion for Single-View Novel View Synthesis20 Sep
  6. 2023MVDream: Multi-view Diffusion for 3D Generation31 Aug
  7. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  8. 2023Long-Term Photometric Consistent Novel View Synthesis with Diffusion Models21 Apr
  9. 2023Consistent View Synthesis with Pose-Guided Diffusion Models30 Mar
  10. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  11. 2022Novel View Synthesis with Diffusion Models6 Oct
  12. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 38

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  5. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  7. 2022Scalable Diffusion Models with Transformerscited by 6,054
  8. 2022Classifier-Free Diffusion Guidancecited by 5,847
  9. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  10. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
  11. 2022Video Diffusion Modelscited by 2,305
  12. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,584
26 more
  1. 2021Image Super-Resolution via Iterative Refinementcited by 1,511
  2. 2019Root Mean Square Layer Normalizationcited by 1,480
  3. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  4. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  5. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  6. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  7. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  8. 2021Variational Diffusion Modelscited by 1,153
  9. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
  10. 2019Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representationscited by 864
  11. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 764
  12. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  13. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 652
  14. 2021RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputscited by 615
  15. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 580
  16. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 460
  17. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 457
  18. 2023Simple diffusion: End-to-end diffusion for high resolution imagescited by 422
  19. 2021Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representationscited by 227
  20. 2023MVImgNet: A Large-scale Dataset of Multi-view Imagescited by 220
  21. 2020Real-Time Intermediate Flow Estimation for Video Frame Interpolationcited by 213
  22. 2023MVDiffusion: Enabling Holistic Multi-view Image Generation with Correspondence-Aware Diffusioncited by 212
  23. 2022FILM: Frame Interpolation for Large Motioncited by 153
  24. 2023LDMVFI: Video Frame Interpolation with Latent Diffusion Modelscited by 89
  25. 2024Sora Generates Videos with Stunning Geometrical Consistencycited by 31
  26. 2023DyST: Towards Dynamic Neural Scene Representations on Real-World Videoscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 13; 2025: 38; 2026: 15.

66 papers.

Sort:

By area:

  1. 2025GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Controlcited by 341
  2. 2025World Simulation with Video Foundation Models for Physical AIcited by 200
  3. 2025Stable Virtual Camera: Generative View Synthesis with Diffusion Modelscited by 182
  4. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 178
  5. 2025History-Guided Video Diffusioncited by 171
  6. 2024CAT4D: Create Anything in 4D with Multi-View Video Diffusion Modelscited by 161
  7. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  8. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 154
  9. 2024DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusioncited by 140
  10. 2024ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuningcited by 70
  11. 2024World-consistent Video Diffusion with Explicit 3D Modelingcited by 65
  12. 2025RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Controlcited by 57
38 more
  1. 2024StreetCrafter: Street View Synthesis with Controllable Video Diffusion Modelscited by 49
  2. 2025WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actionscited by 45
  3. 2025Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillationcited by 41
  4. 2025Reangle-A-Video: 4D Video Generation as Video-to-Video Translationcited by 35
  5. 2025Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Controlcited by 34
  6. 2024Scaling 4D Representationscited by 30
  7. 2024Puppet-Master: Scaling Interactive Video Generation as a Motion Prior for Part-Level Dynamicscited by 28
  8. 20244Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusioncited by 27
  9. 2024Cavia: Camera-controllable Multi-view Video Diffusion with View-Integrated Attentioncited by 24
  10. 2025CamCtrl3D: Single-Image Scene Exploration with Precise 3D Camera Controlcited by 21
  11. 2025EPiC: Efficient Video Camera Control Learning with Precise Anchor-Video Guidancecited by 21
  12. 2025Generative Gaussian Splatting: Generating 3D Scenes with Video Diffusion Priorscited by 19
  13. 2025Diffuman4D: 4D Consistent Human View Synthesis from Sparse-View Videos with Spatio-Temporal Diffusion Modelscited by 17
  14. 2025MVP4D: Multi-View Portrait Video Diffusion for Animatable 4D Avatarscited by 17
  15. 2025Zero4D: Training-Free 4D Video Generation From Single Video Using Off-the-Shelf Video Diffusioncited by 16
  16. 2026ActionMesh: Animated 3D Mesh Generation with Temporal 3D Diffusioncited by 16
  17. 2025From Image to Video: An Empirical Study of Diffusion Representationscited by 15
  18. 2026AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memoriescited by 15
  19. 20254Real-Video-V2: Fused View-Time Attention and Feedforward Reconstruction for 4D Scene Generationcited by 14
  20. 20254D-LRM: Large Space-Time Reconstruction Model From and To Any View at Any Timecited by 14
  21. 2025Reconstruct, Inpaint, Test-Time Finetune: Dynamic Novel-view Synthesis from Monocular Videoscited by 14
  22. 2025BulletTime: Decoupled Control of Time and Camera Pose for Video Generationcited by 14
  23. 2025VideoPanda: Video Panoramic Diffusion with Multi-view Attentioncited by 13
  24. 2025SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Timecited by 11
  25. 2026PerpetualWonder: Long-Horizon Action-Conditioned 4D Scene Generationcited by 9
  26. 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trendscited by 8
  27. 2024KFC-W: Generating 3D-Consistent Videos from Unposed Internet Photoscited by 7
  28. 2025Vid-CamEdit: Video Camera Trajectory Editing with Generative Rendering from Estimated Geometrycited by 7
  29. 2025Eye2Eye: A Simple Approach for Monocular-to-Stereo Video Synthesiscited by 6
  30. 2025CausNVS: Autoregressive Multi-view Diffusion for Flexible 3D Novel View Synthesiscited by 6
  31. 2025DIMO: Diverse 3D Motion Generation for Arbitrary Objectscited by 6
  32. 2025InverseCrafter: Efficient Video ReCapture as a Latent Domain Inverse Problemcited by 6
  33. 2025SOPHY: Learning to Generate Simulation-Ready Objects with Physical Materialscited by 5
  34. 2025OracleGS: Grounding Generative Priors for Sparse-View Gaussian Splattingcited by 5
  35. 2026SCoPE: Sightline-Coordinate Positional Encoding for Video Diffusion Transformerscited by 5
  36. 2025Virtually Being: Customizing Camera-Controllable Video Diffusion Models with Multi-View Performance Capturescited by 4
  37. 2025OmniView: An All-Seeing Diffusion Model for 3D and 4D View Synthesiscited by 4
  38. 2025UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesiscited by 4

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Diff4Splat: Controllable 4D Scene Generation with Latent Dynamic Reconstruction Models
  2. 2025Dream4D: Lifting Camera-Controlled I2V towards Spatiotemporally Consistent 4D Generation
  3. 2026Probing into Camera Control of Video Models
  4. 20254DNeX: Feed-Forward 4D Generative Modeling Made Easy
  5. 2026Video Models as Native 4D Renderers: World-Grounded Conditioning from Animated Mesh
  6. 2024Diffusion Priors for Dynamic View Synthesis from Monocular Videos
  7. 2024Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models
  8. 2023ViVid-1-to-3: Novel View Synthesis with Video Diffusion Models