mlsanityData to 7 Oct 2026

World-consistent Video Diffusion with Explicit 3D Modeling

arXiv 2412.01821 ↗First posted 2 Dec 2024cs.CVcited by 62

On the map: Diffusion models → 3D, point cloud and scene generation

Abstract (arXiv)

Recent advancements in diffusion models have set new benchmarks in image and video generation, enabling realistic visual synthesis across single- and multi-frame contexts. However, these models still struggle with efficiently and explicitly generating 3D-consistent content. To address this, we propose World-consistent Video Diffusion (WVD), a novel framework that incorporates explicit 3D supervision using XYZ images, which encode global 3D coordinates for each image pixel. More specifically, we train a diffusion transformer to learn the joint distribution of RGB and XYZ frames. This approach supports multi-task adaptability via a flexible inpainting strategy. For example, WVD can estimate XYZ frames from ground-truth RGB or generate novel RGB frames using XYZ projections along a specified camera trajectory. In doing so, WVD unifies tasks like single-image-to-3D generation, multi-view stereo, and camera-controlled video generation. Our approach demonstrates competitive performance across multiple benchmarks, providing a scalable solution for 3D-consistent video and image generation with a single pretrained model.

Key facts. Cited by 62 papers in the map (data through 7 Oct 2026). Placed in 3D, point cloud and scene generation, one of 14 directions in Diffusion models. Builds on 61 of its 73 references that are in the map. First posted on arXiv 2 Dec 2024.

Builds on

61 of 73 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Controlling Space and Time with Diffusion Models10 Jul
  2. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Models16 May
  3. 2023ImageDream: Image-Prompt Multi-view Diffusion for 3D Generation2 Dec
  4. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Image7 Sep
  5. 2023MVDream: Multi-view Diffusion for 3D Generation31 Aug
  6. 2023Control3Diff: Learning Controllable 3D Diffusion Models from Single-view Images13 Apr
  7. 2023Zero-1-to-3: Zero-shot One Image to 3D Object20 Mar
  8. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Models24 Jan

Other references · 53

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2022Scalable Diffusion Models with Transformerscited by 5,931
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  9. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  10. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  11. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  12. 2022Video Diffusion Modelscited by 2,280
38 more
  1. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  2. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  3. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  4. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
  5. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  6. 2019Root Mean Square Layer Normalizationcited by 1,425
  7. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  8. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  9. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  10. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 974
  11. 2024Grounding Image Matching in 3D with MASt3Rcited by 960
  12. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  13. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  14. 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
  15. 2018MVSNet: Depth Inference for Unstructured Multi-view Stereocited by 780
  16. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
  17. 2021Depth-supervised NeRF: Fewer Views and Faster Training for Freecited by 732
  18. 2021Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstructioncited by 646
  19. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  20. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  21. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  22. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Modelscited by 349
  23. 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 330
  24. 2023Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Drivingcited by 309
  25. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  26. 2023Visual Geometry Grounded Deep Structure From Motioncited by 225
  27. 2023MVImgNet: A Large-scale Dataset of Multi-view Imagescited by 219
  28. 2018Stereo Magnification: Learning View Synthesis using Multiplane Imagescited by 206
  29. 20203D Photography using Context-aware Layered Depth Inpaintingcited by 206
  30. 2016DeMoN: Depth and Motion Network for Learning Monocular Stereocited by 200
  31. 2018DeepV2D: Video to Depth with Differentiable Structure from Motioncited by 185
  32. 2019ReFusion: 3D Reconstruction in Dynamic Environments for RGB-D Cameras Exploiting Residualscited by 180
  33. 2023NerfDiff: Single-image View Synthesis with NeRF-guided Distillation from 3D-aware Diffusioncited by 179
  34. 2024CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generationcited by 163
  35. 2021GNeRF: GAN-based Neural Radiance Field without Posed Cameracited by 153
  36. 20213D-aware Image Synthesis via Learning Structural and Textural Representationscited by 109
  37. 2023ConsistNet: Enforcing 3D Consistency for Multi-view Images Diffusioncited by 65
  38. 2024ViewDiff: 3D-Consistent Image Generation with Text-to-Image Modelscited by 57

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 30; 2026: 32.

62 papers.

Sort:

By area:

  1. 2025FLARE: Feed-forward Geometry, Appearance and Camera Estimation from Uncalibrated Sparse Viewscited by 176
  2. 2025Representation Entanglement for Generation: Training Diffusion Transformers Is Much Easier Than You Thinkcited by 95
  3. 2025RELIC: Interactive Video World Model with Long-Horizon Memorycited by 89
  4. 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modelingcited by 86
  5. 2025Geo4D: Leveraging Video Generators for Geometric 4D Scene Reconstructioncited by 66
  6. 2025Bolt3D: Generating 3D Scenes in Secondscited by 57
  7. 2025Boosting Generative Image Modeling via Joint Image-Feature Synthesiscited by 42
  8. 2025Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrationscited by 33
  9. 2025FantasyWorld: Geometry-Consistent World Modeling via Unified Video and 3D Predictioncited by 33
  10. 20254DNeX: Feed-Forward 4D Generative Modeling Made Easycited by 31
  11. 2025Emergent Temporal Correspondences from Video Diffusion Transformerscited by 30
  12. 2026Gen3R: 3D Scene Generation Meets Feed-Forward Reconstructioncited by 24
38 more
  1. 2025Epipolar Geometry Improves Video Generation Modelscited by 22
  2. 2026VGGRPO: Towards World-Consistent Video Generation with 4D Latent Rewardcited by 19
  3. 2025VicaSplat: A Single Run is All You Need for 3D Gaussian Splatting and Camera Estimation from Unposed Video Framescited by 15
  4. 2026WorldStereo: Bridging Camera-Guided Video Generation and Scene Reconstruction via 3D Geometric Memoriescited by 15
  5. 2025SPATIALGEN: Layout-guided 3D Indoor Scene Generationcited by 14
  6. 2025GeoVideo: Introducing Geometric Regularization into Video Generation Modelcited by 14
  7. 2026AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memoriescited by 13
  8. 2025BulletTime: Decoupled Control of Time and Camera Pose for Video Generationcited by 12
  9. 2026VGGT-World: Transforming VGGT into an Autoregressive Geometry World Modelcited by 11
  10. 2025Can Video Diffusion Model Reconstruct 4D Geometry?cited by 10
  11. 2025How Much 3D Do Video Foundation Models Encode?cited by 9
  12. 2025One4D: Unified 4D Generation and Reconstruction via Decoupled LoRA Controlcited by 8
  13. 2025RoboTransfer: Controllable Geometry-Consistent Video Diffusion for Manipulation Policy Transfercited by 7
  14. 2025SeqTex: Generate Mesh Textures in Video Sequencecited by 7
  15. 2026Geometry-Aware Implicit Memory for Video World Modelscited by 7
  16. 2026CineScene: Implicit 3D as Effective Scene Representation for Cinematic Video Generationcited by 6
  17. 2026World-R1: Reinforcing 3D Constraints for Text-to-Video Generationcited by 6
  18. 2026VideoWeave: Unlocking Geometric Consistency in Video Generation via Joint Geometry-Video Modelingcited by 6
  19. 2025Model as a Game: On Numerical and Spatial Consistency for Generative Gamescited by 5
  20. 20253D Scene Understanding Through Local Random Access Sequence Modelingcited by 5
  21. 2026NOVA3R: Non-pixel-aligned Visual Transformer for Amodal 3D Reconstructioncited by 5
  22. 2026GeoStream: Toward Precise Camera Controlled Streaming Video Generationcited by 4
  23. 2025View-Consistent Diffusion Representations for 3D-Consistent Video Generationcited by 3
  24. 2025WorldReel: 4D Video Generation with Consistent Geometry and Motion Modelingcited by 3
  25. 2026MotionCrafter: Dense Geometry and Motion Reconstruction with a 4D VAEcited by 3
  26. 2026CRePE: Curved Ray Expectation Positional Encoding for Unified-Camera-Controlled Video Generationcited by 3
  27. 2025Towards Depth Foundation Model: Recent Trends in Vision-Based Depth Estimationcited by 2
  28. 2025Avi: Action from Volumetric Inferencecited by 2
  29. 2026RnG: A Unified Transformer for Complete 3D Modeling from Partial Observationscited by 2
  30. 2026Unified 3D Scene Understanding Through Physical World Modelingcited by 2
  31. 2026E3C: Video Generation with 3D Environmental Memory and Ego-Exo Human Pose Controlcited by 2
  32. 2026PointAction: 3D Points as Universal Action Representations for Robot Controlcited by 2
  33. 2026JointHOI: Jointly Generating Contact Maps Enhances Hand Object Interaction Generationcited by 2
  34. 2025Geometry-Aware Single-Image 4D Synthesis via Dense Trajectory Generationcited by 1
  35. 2025Grab-3D: Detecting AI-Generated Videos from 3D Geometric Temporal Consistencycited by 1
  36. 2025GeCo: Evaluating Geometric Consistency for Video Generation via Motion and Structurecited by 1
  37. 2026Retrieve What's Missing: Coverage-Maximizing Retrieval for Consistent Long Video Generationcited by 1
  38. 2026JointEdit3D: Feed-Forward 3D Scene Editing in a Unified Latent Spacecited by 1

50 most cited of 62 All 62 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Voyager: Long-Range and World-Consistent Video Diffusion for Explorable 3D Scene Generation
  2. 2024Diffusion4D: Fast Spatial-temporal Consistent 4D Generation via Video Diffusion Models
  3. 2025VideoFrom3D: 3D Scene Video Generation via Complementary Image and Video Diffusion Models
  4. 2024V3D: Video Diffusion Models are Effective 3D Generators
  5. 20244Diffusion: Multi-view Video Diffusion Model for 4D Generation
  6. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control
  7. 2026RefAny3D: 3D Asset-Referenced Diffusion Models for Image Generation
  8. 2024Diffusion2: Dynamic 3D Content Generation via Score Composition of Video and Multi-view Diffusion Models