mlsanityData to 14 Oct 2026

DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusion

arXiv 2411.04928 ↗First posted 7 Nov 2024cs.CV cs.AI cs.GRcited by 140

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

In this paper, we introduce DimensionX, a framework designed to generate photorealistic 3D and 4D scenes from just a single image with video diffusion. Our approach begins with the insight that both the spatial structure of a 3D scene and the temporal evolution of a 4D scene can be effectively represented through sequences of video frames. While recent video diffusion models have shown remarkable success in producing vivid visuals, they face limitations in directly recovering 3D/4D scenes due to limited spatial and temporal controllability during generation. To overcome this, we propose ST-Director, which decouples spatial and temporal factors in video diffusion by learning dimension-aware LoRAs from dimension-variant data. This controllable video diffusion approach enables precise manipulation of spatial structure and temporal dynamics, allowing us to reconstruct both 3D and 4D representations from sequential frames with the combination of spatial and temporal dimensions. Additionally, to bridge the gap between generated videos and real-world scenes, we introduce a trajectory-aware mechanism for 3D generation and an identity-preserving denoising strategy for 4D generation. Extensive experiments on various real-world and synthetic datasets demonstrate that DimensionX achieves superior results in controllable video generation, as well as in 3D and 4D scene generation, compared with previous methods.

Key facts. Cited by 140 papers in the map (data through 14 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 61 of its 68 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

61 of 68 references.

Closest ancestors · 17

Same direction, newest first

  1. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis3 Sep
  2. 2024ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Model29 Aug
  3. 2024SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistency24 Jul
  4. 2024Controlling Space and Time with Diffusion Models10 Jul
  5. 20244Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models11 Jun
  6. 2024Unique3D: High-Quality and Efficient 3D Mesh Generation from a Single Image30 May
  7. 2024Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models26 May
  8. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Models16 May
  9. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion18 Mar
  10. 2024V3D: Video Diffusion Models are Effective 3D Generators11 Mar
  11. 2023Sherpa3D: Boosting High-Fidelity Text-to-3D Generation via Coarse 3D Prior11 Dec
  12. 2023Animate124: Animating One Image to 4D Dynamic Scene24 Nov
5 more
  1. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusion23 Oct
  2. 2023MVDream: Multi-view Diffusion for 3D Generation31 Aug
  3. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation25 May
  4. 2022Magic3D: High-Resolution Text-to-3D Content Creation18 Nov
  5. 2022DreamFusion: Text-to-3D using 2D Diffusion29 Sep

Other references · 44

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  3. 2020Denoising Diffusion Implicit Modelscited by 9,781
  4. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  5. 2022Scalable Diffusion Models with Transformerscited by 6,054
  6. 2022Classifier-Free Diffusion Guidancecited by 5,847
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  10. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,415
  11. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  12. 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,032
32 more
  1. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  2. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,822
  3. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,567
  4. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  5. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  6. 20234D Gaussian Splatting for Real-Time Dynamic Scene Renderingcited by 1,157
  7. 2023Objaverse-XL: A Universe of 10M+ 3D Objectscited by 774
  8. 2019Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelinescited by 672
  9. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 580
  10. 2023DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Visioncited by 537
  11. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 460
  12. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  13. 2024OpenVid-1M: A Large-Scale High-Quality Dataset for Text-to-video Generationcited by 341
  14. 2024DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalizationcited by 256
  15. 2024Shape of Motion: 4D Reconstruction from a Single Videocited by 250
  16. 2024A Survey on 3D Gaussian Splattingcited by 224
  17. 2023DreamGaussian4D: Generative 4D Gaussian Splattingcited by 194
  18. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 188
  19. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 185
  20. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 175
  21. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Modelscited by 173
  22. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
  23. 2024CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generationcited by 166
  24. 2024Tora: Trajectory-oriented Diffusion Transformer for Video Generationcited by 164
  25. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  26. 2024Recent Advances in 3D Gaussian Splattingcited by 131
  27. 2024Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Controlcited by 117
  28. 2024MotionClone: Training-Free Motion Cloning for Controllable Video Generationcited by 105
  29. 2024Physics3D: Learning Physical Properties of 3D Gaussians via Video Diffusioncited by 80
  30. 2024DreamReward: Text-to-3D Generation with Human Preferencecited by 56
  31. 2024Vidu4D: Single Generated Video to High-Fidelity 4D Reconstruction with Dynamic Gaussian Surfelscited by 38
  32. 2023AnimatableDreamer: Text-Guided Non-rigid 3D Model Generation and Reconstruction with Canonical Score Distillationcited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 11; 2025: 86; 2026: 43.

140 papers.

Sort:

By area:

  1. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 287
  2. 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modelingcited by 170
  3. 2024CAT4D: Create Anything in 4D with Multi-View Video Diffusion Modelscited by 161
  4. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 154
  5. 2025TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Modelscited by 125
  6. 2024Wonderland: Navigating 3D Scenes from a Single Imagecited by 99
  7. 2025HunyuanWorld 1.0: Generating Immersive, Explorable, and Interactive 3D Worlds from Words or Pixelscited by 94
  8. 2025Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls for Video Generationcited by 77
  9. 2024You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scalecited by 65
  10. 2025Follow-Your-Creation: Empowering 4D Creation through Video Inpaintingcited by 62
  11. 2025EnerVerse: Envisioning Embodied Future Space for Robotics Manipulationcited by 61
  12. 2025Bolt3D: Generating 3D Scenes in Secondscited by 59
38 more
  1. 2025Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signalscited by 53
  2. 2025SV4D 2.0: Enhancing Spatio-Temporal Consistency in Multi-View Video Diffusion for High-Quality 4D Generationcited by 48
  3. 2025WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actionscited by 45
  4. 2025Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillationcited by 41
  5. 2025Video-T1: Test-Time Scaling for Video Generationcited by 40
  6. 2025FantasyWorld: Geometry-Consistent World Modeling via Unified Video and 3D Predictioncited by 38
  7. 2025FlexWorld: Progressively Expanding 3D Scenes for Flexiable-View Synthesiscited by 37
  8. 2025GeometryCrafter: Consistent Geometry Estimation for Open-world Videos with Diffusion Priorscited by 37
  9. 2025Reangle-A-Video: 4D Video Generation as Video-to-Video Translationcited by 35
  10. 2025Free4D: Tuning-free 4D Scene Generation with Spatial-Temporal Consistencycited by 35
  11. 2025Matrix-3D: Omnidirectional Explorable 3D World Generationcited by 35
  12. 2024Prometheus: 3D-Aware Latent Diffusion Models for Feed-Forward Text-to-3D Scene Generationcited by 34
  13. 20254DNeX: Feed-Forward 4D Generative Modeling Made Easycited by 34
  14. 2025FlashWorld: High-quality 3D Scene Generation within Secondscited by 32
  15. 2025Emergent Temporal Correspondences from Video Diffusion Transformerscited by 31
  16. 2026Gen3R: 3D Scene Generation Meets Feed-Forward Reconstructioncited by 31
  17. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Controlcited by 28
  18. 2024DepthLab: From Partial to Completecited by 27
  19. 2025Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Surveycited by 27
  20. 2024Generative Photography: Scene-Consistent Camera Control for Realistic Text-to-Image Synthesiscited by 26
  21. 2025WorldExplorer: Towards Generating Fully Navigable 3D Scenescited by 26
  22. 2025VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generationcited by 25
  23. 20253D Scene Generation: A Surveycited by 24
  24. 2025Perception-as-Control: Fine-grained Controllable Image Animation with 3D-aware Motion Representationcited by 23
  25. 2025Generative Video Motion Editing with 3D Point Trackscited by 23
  26. 2026VideoGPA: Distilling Geometry Priors for 3D-Consistent Video Generationcited by 23
  27. 2024FaceLift: Learning Generalizable Single Image 3D Face Reconstruction from Synthetic Headscited by 22
  28. 2025EPiC: Efficient Video Camera Control Learning with Precise Anchor-Video Guidancecited by 21
  29. 2025ShapeGen4D: Towards High Quality 4D Shape Generation from Videoscited by 21
  30. 2025Positional Encoding Fieldcited by 20
  31. 2025Generative Gaussian Splatting: Generating 3D Scenes with Video Diffusion Priorscited by 19
  32. 2025WorldGen: From Text to Traversable and Interactive 3D Worldscited by 19
  33. 2025GenDoP: Auto-regressive Camera Trajectory Generation as a Director of Photographycited by 18
  34. 2026WorldStereo: Bridging Camera-Guided Video Generation and Scene Reconstruction via 3D Geometric Memoriescited by 18
  35. 2025Compositional Generative Model of Unbounded 4D Citiescited by 17
  36. 2025Skyfall-GS: Synthesizing Immersive 3D Urban Scenes from Satellite Imagerycited by 17
  37. 2025Zero4D: Training-Free 4D Video Generation From Single Video Using Off-the-Shelf Video Diffusioncited by 16
  38. 2025SpatialCrafter: Unleashing the Imagination of Video Diffusion Models for Scene Reconstruction from Limited Observationscited by 16

50 most cited of 140 All 140 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025VideoScene: Distilling Video Diffusion Model to Generate 3D Scenes in One Step
  2. 2024Diffusion2: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models
  3. 20244Diffusion: Multi-view Video Diffusion Model for 4D Generation
  4. 2026Beyond Pixels: From Video Priors to 4D Worlds
  5. 2024Vid3D: Synthesis of Dynamic 3D Scenes using 2D Video Diffusion
  6. 20254DVD: Cascaded Dense-view Video Diffusion Model for High-quality 4D Content Generation
  7. 2025Can Video Diffusion Model Reconstruct 4D Geometry?
  8. 2025VideoFrom3D: 3D Scene Video Generation via Complementary Image and Video Diffusion Models