mlsanityData to 7 Oct 2026

CameraCtrl: Enabling Camera Control for Text-to-Video Generation

arXiv 2404.02101 ↗First posted 2 Apr 2024cs.CVcited by 448

On the map: Diffusion models → Video generation

Abstract (arXiv)

Controllability plays a crucial role in video generation, as it allows users to create and edit content more precisely. Existing models, however, lack control of camera pose that serves as a cinematic language to express deeper narrative nuances. To alleviate this issue, we introduce CameraCtrl, enabling accurate camera pose control for video diffusion models. Our approach explores effective camera trajectory parameterization along with a plug-and-play camera pose control module that is trained on top of a video diffusion model, leaving other modules of the base model untouched. Moreover, a comprehensive study on the effect of various training datasets is conducted, suggesting that videos with diverse camera distributions and similar appearance to the base model indeed enhance controllability and generalization. Experimental results demonstrate the effectiveness of CameraCtrl in achieving precise camera control with different video generation models, marking a step forward in the pursuit of dynamic and customized video storytelling from textual and camera pose inputs.

Key facts. Cited by 448 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 53 of its 72 references that are in the map. First posted on arXiv 2 Apr 2024.

Builds on

53 of 72 references.

Closest ancestors · 18

Same direction, newest first

  1. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motion5 Feb
  2. 2024Lumiere: A Space-Time Diffusion Model for Video Generation23 Jan
  3. 2024Latte: Latent Diffusion Transformer for Video Generation5 Jan
  4. 2023Photorealistic Video Generation with Diffusion Models11 Dec
  5. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation28 Nov
  6. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Models28 Nov
  7. 2023MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model27 Nov
  8. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  9. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models7 Nov
  10. 2023SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction31 Oct
  11. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
  12. 2023ControlVideo: Training-free Controllable Text-to-Video Generation22 May
6 more
  1. 2023Motion-Conditioned Diffusion Model for Controllable Video Synthesis27 Apr
  2. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  3. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  4. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  5. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  6. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct

Other references · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2022Scalable Diffusion Models with Transformerscited by 5,931
  7. 2022Classifier-Free Diffusion Guidancecited by 5,717
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  10. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  11. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  12. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
23 more
  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  2. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  3. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  4. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  5. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  6. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  7. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  8. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  9. 2024Grounded SAM: Assembling Open-World Models for Diverse Visual Taskscited by 914
  10. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  11. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  12. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  13. 2021Light Field Networks: Neural Scene Representations with Single-Evaluation Renderingcited by 330
  14. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
  15. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 260
  16. 2020Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Imagecited by 239
  17. 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generationcited by 225
  18. 2023MVImgNet: A Large-scale Dataset of Multi-view Imagescited by 219
  19. 2024Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modelingcited by 209
  20. 2023Make-Your-Video: Customized Video Generation Using Textual and Structural Guidancecited by 129
  21. 2019Learning the Depths of Moving People by Watching Frozen Peoplecited by 120
  22. 2024Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesiscited by 105
  23. 2023Compositional 3D Scene Generation using Locally Conditioned Diffusioncited by 101

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 64; 2025: 198; 2026: 186.

448 papers.

Sort:

By area:

  1. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  2. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 392
  3. 2025GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Controlcited by 325
  4. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 278
  5. 2024WonderWorld: Interactive 3D Scene Generation from a Single Imagecited by 197
  6. 2025Stable Virtual Camera: Generative View Synthesis with Diffusion Modelscited by 179
  7. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 176
  8. 2025Matrix-game 2.0: An open-source real-time and streaming interactive world modelcited by 174
  9. 2024CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generationcited by 163
  10. 2024CAT4D: Create Anything in 4D with Multi-View Video Diffusion Modelscited by 160
  11. 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrievalcited by 157
  12. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
38 more
  1. 2025WorldMem: Long-term Consistent World Simulation with Memorycited by 152
  2. 2025WorldPlay: Towards Long-Term Geometric Consistency for Real-Time Interactive World Modelingcited by 152
  3. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  4. 2025WorldScore: A Unified Evaluation Benchmark for World Generationcited by 143
  5. 2024ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Modelcited by 139
  6. 2024DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusioncited by 139
  7. 2026Cosmos 3: Omnimodal World Models for Physical AIcited by 137
  8. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 124
  9. 2025TrajectoryCrafter: Redirecting Camera Trajectory for Monocular Videos via Diffusion Modelscited by 120
  10. 2024Training-free Camera Control for Video Generationcited by 119
  11. 2024Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Controlcited by 117
  12. 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Modelcited by 115
  13. 2025Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noisecited by 114
  14. 2025Long Context Tuning for Video Generationcited by 110
  15. 2025VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed View Memorycited by 102
  16. 2024Wonderland: Navigating 3D Scenes from a Single Imagecited by 98
  17. 2025CameraCtrl II: Dynamic Scene Exploration via Camera-controlled Video Diffusion Modelscited by 97
  18. 2025Controllable Video Generation: A Surveycited by 93
  19. 2024MotionBooth: Motion-Aware Customized Text-to-Video Generationcited by 92
  20. 2024SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpointscited by 92
  21. 2025Hunyuan-GameCraft: High-dynamic Interactive Game Video Generation with Hybrid History Conditioncited by 92
  22. 2025Geometry Forcing: Marrying Video Diffusion and 3D Representation for Consistent World Modelingcited by 86
  23. 2025Autoregressive Adversarial Post-Training for Real-Time Interactive Video Generationcited by 84
  24. 2025Scaling Instruction-Based Video Editing with a High-Quality Synthetic Datasetcited by 76
  25. 2025Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls for Video Generationcited by 75
  26. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 74
  27. 2025SpatialVID: A Large-Scale Video Dataset with Spatial Annotationscited by 72
  28. 20243DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generationcited by 71
  29. 2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 71
  30. 2024SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generationcited by 68
  31. 2024ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuningcited by 68
  32. 2024Controlling Space and Time with Diffusion Modelscited by 65
  33. 2024You See it, You Got it: Learning 3D Creation on Pose-Free Videos at Scalecited by 65
  34. 2024HumanVid: Demystifying Training Data for Camera-controllable Human Image Animationcited by 63
  35. 2024Image Conductor: Precision Control for Interactive Video Synthesiscited by 62
  36. 2024World-consistent Video Diffusion with Explicit 3D Modelingcited by 62
  37. 2025Towards Understanding Camera Motions in Any Videocited by 62
  38. 2025Follow-Your-Creation: Empowering 4D Creation through Video Inpaintingcited by 61

50 most cited of 448 All 448 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023TrailBlazer: Trajectory Control for Diffusion-Based Video Generation
  2. 2025OmniCam: Unified Multimodal Video Generation via Camera Control
  3. 2025CETCAM: Camera-Controllable Video Generation via Consistent and Extensible Tokenization
  4. 2026ActCam: Zero-Shot Joint Camera and 3D Motion Control for Video Generation
  5. 2025CineMaster: A 3D-Aware and Controllable Framework for Cinematic Text-to-Video Generation
  6. 2026Auteur: Language-Driven Cinematographic Framing for Human-Centric Video Generation
  7. 2026CineWeaver: Training-Free Reference-Controllable Multi-Shot Long Video Generation for Cinematic Storytelling
  8. 2026PartiCam: Camera Controlled Video Generation with Reward Guidance