mlsanityData to 7 Oct 2026

AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformers

arXiv 2411.18673 ↗First posted 27 Nov 2024cs.CVcited by 151

On the map: Diffusion models → Video generation

Abstract (arXiv)

Numerous works have recently integrated 3D camera control into foundational text-to-video models, but the resulting camera control is often imprecise, and video generation quality suffers. In this work, we analyze camera motion from a first principles perspective, uncovering insights that enable precise 3D camera manipulation without compromising synthesis quality. First, we determine that motion induced by camera movements in videos is low-frequency in nature. This motivates us to adjust train and test pose conditioning schedules, accelerating training convergence while improving visual and motion quality. Then, by probing the representations of an unconditional video diffusion transformer, we observe that they implicitly perform camera pose estimation under the hood, and only a sub-portion of their layers contain the camera information. This suggested us to limit the injection of camera conditioning to a subset of the architecture to prevent interference with other video features, leading to a 4x reduction of training parameters, improved training speed, and 10% higher visual quality. Finally, we complement the typical dataset for camera control learning with a curated dataset of 20K diverse, dynamic videos with stationary cameras. This helps the model distinguish between camera and scene motion and improves the dynamics of generated pose-conditioned videos. We compound these findings to design the Advanced 3D Camera Control (AC3D) architecture, the new state-of-the-art model for generative video modeling with camera control.

Key facts. Cited by 151 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 179 of its 204 references that are in the map. First posted on arXiv 27 Nov 2024.

Builds on

179 of 204 references.

Closest ancestors · 25

Same direction, newest first

  1. 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Model21 Oct
  2. 2024Boosting Camera Motion Control for Video Diffusion Transformers14 Oct
  3. 2024Cavia: Camera-controllable Multi-view Video Diffusion with View-Integrated Attention14 Oct
  4. 2024AvatarGO: Zero-shot 4D Human-Object Interaction Generation and Animation9 Oct
  5. 2024Phy124: Fast Physics-Driven 4D Content Generation from a Single Image11 Sep
  6. 2024Compositional 3D-aware Video Generation with LLM Director31 Aug
  7. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  8. 2024Tora: Trajectory-oriented Diffusion Transformer for Video Generation31 Jul
  9. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control17 Jul
  10. 2024FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Models24 Jun
  11. 2024Training-free Camera Control for Video Generation14 Jun
  12. 2024Vivid-ZOO: Multi-View Video Generation with Diffusion Model12 Jun
13 more
  1. 2024CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation4 Jun
  2. 2024Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control27 May
  3. 2024Video Diffusion Models are Training-free Motion Interpreter and Controller23 May
  4. 2024MotionMaster: Training-free Camera Motion Transfer For Video Generation24 Apr
  5. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generation2 Apr
  6. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motion5 Feb
  7. 2024Latte: Latent Diffusion Transformer for Video Generation5 Jan
  8. 2023TrailBlazer: Trajectory Control for Diffusion-Based Video Generation31 Dec
  9. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  10. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generation30 Oct
  11. 2023LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation16 Oct
  12. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  13. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr

Other references · 154

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  9. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
  10. 2022Scalable Diffusion Models with Transformerscited by 5,931
  11. 2022Classifier-Free Diffusion Guidancecited by 5,717
  12. 2016Layer Normalizationcited by 5,579
38 more
  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  5. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  8. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  9. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  10. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  11. 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
  12. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  13. 2019Root Mean Square Layer Normalizationcited by 1,425
  14. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  15. 2023ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillationcited by 1,207
  16. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  17. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  18. 2023MVDream: Multi-view Diffusion for 3D Generationcited by 928
  19. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  20. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 834
  21. 2024LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creationcited by 730
  22. 2023Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creationcited by 684
  23. 2023Wonder3D: Single Image to 3D using Cross-Domain Diffusioncited by 682
  24. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 638
  25. 2024Movie Gen: A Cast of Media Foundation Modelscited by 614
  26. 2022Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generationcited by 597
  27. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  28. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  29. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  30. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 425
  31. 2023Analyzing and Improving the Training Dynamics of Diffusion Modelscited by 396
  32. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 395
  33. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 392
  34. 2023Instant3D: Fast Text-to-3D with Sparse-View Generation and Large Reconstruction Modelcited by 376
  35. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 369
  36. 2024SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusioncited by 364
  37. 2023Splatter Image: Ultra-Fast Single-View 3D Reconstructioncited by 359
  38. 2023Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Imagecited by 357

50 most cited of 154 All 154 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 70; 2026: 80.

151 papers.

Sort:

By area:

  1. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 278
  2. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Controlcited by 184
  3. 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modelingcited by 152
  4. 2025Video World Models with Long-term Spatial Memorycited by 124
  5. 2024Wonderland: Navigating 3D Scenes from a Single Imagecited by 98
  6. 2025CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Modelscited by 97
  7. 2025Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls for Video Generationcited by 75
  8. 2025CineMaster: A 3D-Aware and Controllable Framework for Cinematic Text-to-Video Generationcited by 73
  9. 2025Towards Understanding Camera Motions in Any Videocited by 62
  10. 2025Follow-Your-Creation: Empowering 4D Creation through Video Inpaintingcited by 61
  11. 2025OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modelingcited by 59
  12. 2025RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Controlcited by 54
38 more
  1. 2025MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generationcited by 50
  2. 2025VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Priorcited by 49
  3. 2025CamCloneMaster: Enabling Reference-based Camera Control for Video Generationcited by 44
  4. 2025FlexWorld: Progressively Expanding 3D Scenes for Flexiable-View Synthesiscited by 37
  5. 2025Lyra: Generative 3D Scene Reconstruction via Video Diffusion Model Self-Distillationcited by 37
  6. 2026Lyra 2.0: Explorable Generative 3D Worldscited by 36
  7. 2025Reangle-A-Video: 4D Video Generation as Video-to-Video Translationcited by 35
  8. 2025Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Controlcited by 33
  9. 2025Matrix-3D: Omnidirectional Explorable 3D World Generationcited by 33
  10. 2026DreamX-World 1.0: A General-Purpose Interactive World Modelcited by 33
  11. 2025EX-4D: EXtreme Viewpoint 4D Video Synthesis via Depth Watertight Meshcited by 32
  12. 2025WorldExplorer: Towards Generating Fully Navigable 3D Scenescited by 26
  13. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Controlcited by 26
  14. 2025VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generationcited by 25
  15. 2025Generative Physical AI in Vision: A Surveycited by 24
  16. 2025Advances in Feed-Forward 3D Reconstruction and View Synthesis: A Surveycited by 23
  17. 2025EPiC: Efficient Video Camera Control Learning with Precise Anchor-Video Guidancecited by 20
  18. 2025AnyTalker: Scaling Multi-Person Talking Video Generation with Interactivity Refinementcited by 19
  19. 2026INSPATIO-WORLD: A Real-Time 4D World Simulator via Spatiotemporal Autoregressive Modelingcited by 18
  20. 2025SpatialCrafter: Unleashing the Imagination of Video Diffusion Models for Scene Reconstruction from Limited Observationscited by 16
  21. 2025Zero4D: Training-Free 4D Video Generation From Single Video Using Off-the-Shelf Video Diffusioncited by 15
  22. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  23. 2026WorldStereo: Bridging Camera-Guided Video Generation and Scene Reconstruction via 3D Geometric Memoriescited by 15
  24. 2025Survey of Video Diffusion Models: Foundations, Implementations, and Applicationscited by 14
  25. 2025Taming Video Models for 3D and 4D Generation via Zero-Shot Camera Controlcited by 14
  26. 2026iWorld-Bench: A Benchmark for Interactive World Models with a Unified Action Generation Frameworkcited by 14
  27. 2025SteerX: Creating Any Camera-Free 3D and 4D Scenes with Geometric Steeringcited by 13
  28. 20253D Scene Prompting for Scene-Consistent Camera-Controllable Video Generationcited by 13
  29. 2026AnchorWeave: World-Consistent Video Generation with Retrieved Local Spatial Memoriescited by 13
  30. 2025Exo2EgoSyn: Unlocking Foundation Video Generation Models for Exocentric-to-Egocentric Video Synthesiscited by 12
  31. 2025BulletTime: Decoupled Control of Time and Camera Pose for Video Generationcited by 12
  32. 2026AnyView: Synthesizing Any Novel View in Dynamic Scenescited by 12
  33. 2026ReRoPE: Repurposing RoPE for Relative Camera Controlcited by 12
  34. 2026When and How Much to Imagine: Adaptive Test-Time Scaling with World Models for Visual Spatial Reasoningcited by 12
  35. 2025DualCamCtrl: Dual-Branch Diffusion Model for Geometry-Aware Camera-Controlled Video Generationcited by 11
  36. 2025Can Video Diffusion Model Reconstruct 4D Geometry?cited by 10
  37. 2025AniCrafter: Customizing Realistic Human-Centric Animation via Avatar-Background Conditioning in Video Diffusion Modelscited by 10
  38. 2025ReDirector: Creating Any-Length Video Retakes with Rotary Camera Encodingcited by 10

50 most cited of 151 All 151 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025CamCtrl3D: Single-Image Scene Exploration with Precise 3D Camera Control
  2. 2026QWERTY: Training-Free Motion Control via Query-Warped Video Diffusion Transformers
  3. 2024CPA: Camera-pose-awareness Diffusion Transformer for Video Generation
  4. 2026PartiCam: Camera Controlled Video Generation with Reward Guidance
  5. 2025DT-NVS: Diffusion Transformers for Novel View Synthesis
  6. 2026Tri-Prompting: Video Diffusion with Unified Control over Scene, Subject, and Motion
  7. 2026Video Models as Native 4D Renderers: World-Grounded Conditioning from Animated Mesh
  8. 2025LMP: Leveraging Motion Prior in Zero-Shot Video Generation with Diffusion Transformer