mlsanityData to 7 Oct 2026

Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motion

arXiv 2402.03162 ↗First posted 5 Feb 2024cs.CVcited by 167

On the map: Diffusion models → Video generation

Abstract (arXiv)

Recent text-to-video diffusion models have achieved impressive progress. In practice, users often desire the ability to control object motion and camera movement independently for customized video creation. However, current methods lack the focus on separately controlling object motion and camera movement in a decoupled manner, which limits the controllability and flexibility of text-to-video models. In this paper, we introduce Direct-a-Video, a system that allows users to independently specify motions for multiple objects as well as camera's pan and zoom movements, as if directing a video. We propose a simple yet effective strategy for the decoupled control of object motion and camera movement. Object motion is controlled through spatial cross-attention modulation using the model's inherent priors, requiring no additional optimization. For camera movement, we introduce new temporal cross-attention layers to interpret quantitative camera movement parameters. We further employ an augmentation-based approach to train these layers in a self-supervised manner on a small-scale dataset, eliminating the need for explicit motion annotation. Both components operate independently, allowing individual or combined control, and can generalize to open-domain scenarios. Extensive experiments demonstrate the superiority and effectiveness of our method. Project page and code are available at https://direct-a-video.github.io/.

Key facts. Cited by 167 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 61 of its 75 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

61 of 75 references.

Closest ancestors · 22

Same direction, newest first

  1. 2023DreaMoving: A Human Video Generation Framework based on Diffusion Models8 Dec
  2. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motion7 Dec
  3. 2023VideoSwap: Customized Video Subject Swapping with Interactive Semantic Point Correspondence4 Dec
  4. 2023VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models1 Dec
  5. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation28 Nov
  6. 2023MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model27 Nov
  7. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  8. 2023LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation16 Oct
  9. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Models12 Oct
  10. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editing18 Aug
  11. 2023ModelScope Text-to-Video Technical Report12 Aug
  12. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
10 more
  1. 2023Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation13 Jun
  2. 2023Motion-Conditioned Diffusion Model for Controllable Video Synthesis27 Apr
  3. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  4. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models30 Mar
  5. 2023Pix2Video: Video Editing using Image Diffusion22 Mar
  6. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editing16 Mar
  7. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  8. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  9. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  10. 2022Video Diffusion Models7 Apr

Other references · 39

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  7. 2022Classifier-Free Diffusion Guidancecited by 5,717
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  10. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  11. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  12. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
27 more
  1. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  2. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  3. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  4. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  5. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  6. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  7. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  8. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
  9. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  10. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  11. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  12. 2023Emergent Correspondence from Image Diffusioncited by 437
  13. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  14. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  15. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 260
  16. 2023Dense Text-to-Image Generation with Attention Modulationcited by 162
  17. 2020Towards Photo-Realistic Virtual Try-On by Adaptively GeneratingleftrightarrowPreserving Image Contentcited by 148
  18. 2023DisCo: Disentangled Control for Realistic Human Dance Generationcited by 141
  19. 2021Layered Neural Atlases for Consistent Video Editingcited by 137
  20. 2023CoDeF: Content Deformation Fields for Temporally Consistent Video Processingcited by 108
  21. 2022Animal Kingdom: A Large and Diverse Dataset for Animal Behavior Understandingcited by 102
  22. 2023VideoFlow: Exploiting Temporal Cues for Multi-frame Optical Flow Estimationcited by 82
  23. 2023PEEKABOO: Interactive Video Generation via Masked-Diffusioncited by 76
  24. 2023Directed Diffusion: Direct Control of Object Placement through Attention Guidancecited by 69
  25. 2023Uni-paint: A Unified Framework for Multimodal Image Inpainting with Pretrained Diffusion Modelcited by 60
  26. 2020A Unified Framework for Shot Type Classification Based on Subject Centric Lenscited by 44
  27. 2023DragVideo: Interactive Drag-style Video Editingcited by 40

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 45; 2025: 78; 2026: 44.

167 papers.

Sort:

By area:

  1. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  2. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 337
  3. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 278
  4. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Controlcited by 184
  5. 2025Matrix-game 2.0: An open-source real-time and streaming interactive world modelcited by 174
  6. 2024CAT4D: Create Anything in 4D with Multi-View Video Diffusion Modelscited by 160
  7. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  8. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  9. 2024DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusioncited by 139
  10. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 124
  11. 2024Training-free Camera Control for Video Generationcited by 119
  12. 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Modelcited by 115
38 more
  1. 2025Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noisecited by 114
  2. 2025Controllable Video Generation: A Surveycited by 93
  3. 2024MotionBooth: Motion-Aware Customized Text-to-Video Generationcited by 92
  4. 2024DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmarkcited by 85
  5. 2024TC4D: Trajectory-Conditioned Text-to-4D Generationcited by 76
  6. 2025Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls for Video Generationcited by 75
  7. 2025CineMaster: A 3D-Aware and Controllable Framework for Cinematic Text-to-Video Generationcited by 73
  8. 2025Yume: An Interactive World Generation Modelcited by 73
  9. 20243DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generationcited by 71
  10. 2025MotionStream: Real-Time Video Generation with Interactive Motion Controlscited by 71
  11. 2024SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generationcited by 68
  12. 2025Towards Understanding Camera Motions in Any Videocited by 62
  13. 2024Human4DiT: 360-degree Human Video Generation with 4D Diffusion Transformercited by 61
  14. 2025MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidancecited by 60
  15. 2024FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Modelscited by 55
  16. 2025RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Controlcited by 54
  17. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  18. 2024Still-Moving: Customized Video Generation without Customized Video Datacited by 49
  19. 2024PhysMotion: Physics-Grounded Dynamics From a Single Imagecited by 45
  20. 2024DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise Motion Controlcited by 42
  21. 2025ATI: Any Trajectory Instruction for Controllable Video Generationcited by 42
  22. 2024Vivid-ZOO: Multi-View Video Generation with Diffusion Modelcited by 40
  23. 2025PhysAnimator: Physics-Guided Generative Cartoon Animationcited by 40
  24. 2024EgoVid-5M: A Large-Scale Video-Action Dataset for Egocentric Video Generationcited by 38
  25. 2025FloVD: Optical Flow Meets Video Diffusion Model for Enhanced Camera-Controlled Video Synthesiscited by 37
  26. 2025Reangle-A-Video: 4D Video Generation as Video-to-Video Translationcited by 35
  27. 2025Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Controlcited by 33
  28. 2025Spatia: Video Generation with Updatable Spatial Memorycited by 33
  29. 2025OmniNWM: Omniscient Driving Navigation World Modelscited by 32
  30. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  31. 2025JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimizationcited by 30
  32. 2025MotionPro: A Precise Motion Controller for Image-to-Video Generationcited by 29
  33. 20244Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusioncited by 27
  34. 2024Generative Photography: Scene-Consistent Camera Control for Realistic Text-to-Image Synthesiscited by 26
  35. 2026VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Controlcited by 26
  36. 2025VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generationcited by 25
  37. 2025TransAnimate: Taming Layer Diffusion to Generate RGBA Videocited by 24
  38. 2024InfinityDrive: Breaking Time Limits in Driving World Modelscited by 21

50 most cited of 167 All 167 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MOVi: Training-free Text-conditioned Multi-Object Video Generation
  2. 2024Animate Your Motion: Turning Still Images into Dynamic Videos
  3. 2024Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling
  4. 2026QWERTY: Training-Free Motion Control via Query-Warped Video Diffusion Transformers
  5. 2026DynaTokens: Teaching Dynamics to Camera-Controlled Video Models at Test Time
  6. 2024MotionMaster: Training-free Camera Motion Transfer For Video Generation
  7. 2024Motion Prompting: Controlling Video Generation with Motion Trajectories
  8. 2025Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoising