mlsanityData to 7 Oct 2026

VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control

arXiv 2407.12781 ↗First posted 17 Jul 2024cs.CVcited by 156

On the map: Diffusion models → Video generation

Abstract (arXiv)

Modern text-to-video synthesis models demonstrate coherent, photorealistic generation of complex videos from a text description. However, most existing models lack fine-grained control over camera movement, which is critical for downstream applications related to content creation, visual effects, and 3D vision. Recently, new methods demonstrate the ability to generate videos with controllable camera poses these techniques leverage pre-trained U-Net-based diffusion models that explicitly disentangle spatial and temporal generation. Still, no existing approach enables camera control for new, transformer-based video diffusion models that process spatial and temporal information jointly. Here, we propose to tame video transformers for 3D camera control using a ControlNet-like conditioning mechanism that incorporates spatiotemporal camera embeddings based on Pl"ucker coordinates. The approach demonstrates state-of-the-art performance for controllable video generation after fine-tuning on the RealEstate10K dataset. To the best of our knowledge, our work is the first to enable camera control for transformer-based video diffusion models.

Key facts. Cited by 156 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 145 of its 161 references that are in the map. First posted on arXiv 17 Jul 2024.

Builds on

145 of 161 references.

Closest ancestors · 18

Same direction, newest first

  1. 2024Training-free Camera Control for Video Generation14 Jun
  2. 2024Vivid-ZOO: Multi-View Video Generation with Diffusion Model12 Jun
  3. 2024CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation4 Jun
  4. 2024Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control27 May
  5. 2024Video Diffusion Models are Training-free Motion Interpreter and Controller23 May
  6. 2024MotionMaster: Training-free Camera Motion Transfer For Video Generation24 Apr
  7. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generation2 Apr
  8. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motion5 Feb
  9. 2024Latte: Latent Diffusion Transformer for Video Generation5 Jan
  10. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  11. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generation30 Oct
  12. 2023LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation16 Oct
6 more
  1. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  2. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  3. 2022Scalable Adaptive Computation for Iterative Generation22 Dec
  4. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  5. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  6. 2022Video Diffusion Models7 Apr

Other references · 127

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  6. 2022Scalable Diffusion Models with Transformerscited by 5,931
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  10. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  11. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  12. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
38 more
  1. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  2. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  3. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  4. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  5. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  6. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  7. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  8. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  9. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 730
  10. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
  11. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
  12. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  13. 2023pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstructioncited by 668
  14. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 638
  15. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  16. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  17. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 555
  18. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  19. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  20. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 425
  21. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 395
  22. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 376
  23. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 369
  24. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusioncited by 364
  25. 2023Splatter Image: Ultra-Fast Single-View 3D Reconstructioncited by 359
  26. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Priorcited by 341
  27. 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 330
  28. 2022Novel View Synthesis with Diffusion Modelscited by 299
  29. 2024GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generationcited by 298
  30. 2023LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matchingcited by 290
  31. 2021StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generationcited by 289
  32. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Modelscited by 273
  33. 2021GODIVA: Generating Open-DomaIn Videos from nAtural Descriptionscited by 254
  34. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  35. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  36. 2024TripoSR: Fast 3D Object Reconstruction from a Single Imagecited by 227
  37. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  38. 2023DMV3D: Denoising Multi-View Diffusion using 3D Large Reconstruction Modelcited by 213

50 most cited of 127 All 127 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 25; 2025: 75; 2026: 56.

156 papers.

Sort:

By area:

  1. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 392
  2. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 278
  3. 2024CAT4D: Create Anything in 4D with Multi-View Video Diffusion Modelscited by 160
  4. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  5. 2024DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusioncited by 139
  6. 2025Video World Models with Long-term Spatial Memorycited by 124
  7. 2025TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Modelscited by 120
  8. 2024Wonderland: Navigating 3D Scenes from a Single Imagecited by 98
  9. 2025CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Modelscited by 97
  10. 2025Controllable Video Generation: A Surveycited by 93
  11. 2024SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpointscited by 92
  12. 2025Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls for Video Generationcited by 75
38 more
  1. 20243DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generationcited by 71
  2. 2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 71
  3. 2024SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generationcited by 68
  4. 2024ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuningcited by 68
  5. 2024You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scalecited by 65
  6. 2025Towards Understanding Camera Motions in Any Videocited by 62
  7. 2025Follow-Your-Creation: Empowering 4D Creation through Video Inpaintingcited by 61
  8. 2025RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Controlcited by 54
  9. 2024InfiniCube: Unbounded and Controllable Dynamic 3D Driving Scene Generation with World-Guided Video Modelscited by 53
  10. 2025MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generationcited by 50
  11. 2024StreetCrafter: Street View Synthesis with Controllable Video Diffusion Modelscited by 49
  12. 2025FullDiT: Multi-Task Video Generative Foundation Model with Full Attentioncited by 45
  13. 2025CamCloneMaster: Enabling Reference-based Camera Control for Video Generationcited by 44
  14. 2025FloVD: Optical Flow Meets Video Diffusion Model for Enhanced Camera-Controlled Video Synthesiscited by 37
  15. 2025FlexWorld: Progressively Expanding 3D Scenes for Flexiable-View Synthesiscited by 37
  16. 2025Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillationcited by 37
  17. 2026Lyra 2.0: Explorable Generative 3D Worldscited by 36
  18. 2024AnimateAnything: Consistent and Controllable Animation for Video Generationcited by 34
  19. 2025Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Controlcited by 33
  20. 2025Matrix-3D: Omnidirectional Explorable 3D World Generationcited by 33
  21. 2025OmniNWM: Omniscient Driving Navigation World Modelscited by 32
  22. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  23. 20244Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusioncited by 27
  24. 2024Generative Photography: Scene-Consistent Camera Control for Realistic Text-to-Image Synthesiscited by 26
  25. 2025WorldExplorer: Towards Generating Fully Navigable 3D Scenescited by 26
  26. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Controlcited by 26
  27. 2025VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generationcited by 25
  28. 2024Cavia: Camera-controllable Multi-view Video Diffusion with View-Integrated Attentioncited by 24
  29. 2024Framer: Interactive Frame Interpolationcited by 24
  30. 2025Generative Physical AI in Vision: A Surveycited by 24
  31. 2025Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Surveycited by 23
  32. 2026MosaicMem: Hybrid Spatial Memory for Controllable Video World Modelscited by 23
  33. 2024Boosting Camera Motion Control for Video Diffusion Transformerscited by 20
  34. 2025Pippo: High-Resolution Multi-View Humans from a Single Imagecited by 20
  35. 2025EPiC: Efficient Video Camera Control Learning with Precise Anchor-Video Guidancecited by 20
  36. 2026Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Modelscited by 20
  37. 2026INSPATIO-WORLD: A Real-Time 4D World Simulator via Spatiotemporal Autoregressive Modelingcited by 18
  38. 2024AKiRa: Augmentation Kit on Rays for optical video generationcited by 17

50 most cited of 156 All 156 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Lighting-grounded Video Generation with Renderer-based Agent Reasoning
  2. 2023Photorealistic Video Generation with Diffusion Models
  3. 2026PE-Field 4D: Video Generation Models as Canvas
  4. 2026QWERTY: Training-Free Motion Control via Query-Warped Video Diffusion Transformers
  5. 2025DiTraj: training-free trajectory control for video diffusion transformer
  6. 2024DiTCtrl: Exploring Attention Control in Multi-Modal Diffusion Transformer for Tuning-Free Multi-Prompt Longer Video Generation
  7. 2023Generative Rendering: Controllable 4D-Guided Video Generation with 2D Diffusion Models
  8. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control