mlsanityData to 7 Oct 2026

VideoComposer: Compositional Video Synthesis with Motion Controllability

arXiv 2306.02018 ↗First posted 3 Jun 2023cs.CVcited by 461

Not on the map. Why?

Cited by 217 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The pursuit of controllability as a higher standard of visual content creation has yielded remarkable progress in customizable image synthesis. However, achieving controllable video synthesis remains challenging due to the large variation of temporal dynamics and the requirement of cross-frame temporal consistency. Based on the paradigm of compositional generation, this work presents VideoComposer that allows users to flexibly compose a video with textual conditions, spatial conditions, and more importantly temporal conditions. Specifically, considering the characteristic of video data, we introduce the motion vector from compressed videos as an explicit control signal to provide guidance regarding temporal dynamics. In addition, we develop a Spatio-Temporal Condition encoder (STC-encoder) that serves as a unified interface to effectively incorporate the spatial and temporal relations of sequential inputs, with which the model could make better use of temporal conditions and hence achieve higher inter-frame consistency. Extensive experimental results suggest that VideoComposer is able to control the spatial and temporal patterns simultaneously within a synthesized video in various forms, such as text description, sketch sequence, reference video, or even simply hand-crafted motions. The code and models will be publicly available at https://videocomposer.github.io.

Key facts. Cited by 461 papers in the map (data through 7 Oct 2026). Builds on 76 of its 84 references that are in the map. First posted on arXiv 3 Jun 2023.

Builds on

76 of 84 references.

References · 76

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  10. 2022Classifier-Free Diffusion Guidancecited by 5,717
  11. 2022Generative Adversarial Networkscited by 5,603
  12. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
38 more
  1. 2013Auto-Encoding Variational Bayescited by 3,980
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  4. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  6. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  7. 2020RAFT: Recurrent All-Pairs Field Transforms for Optical Flowcited by 2,385
  8. 2022Video Diffusion Modelscited by 2,280
  9. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  10. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  11. 2021ViViT: A Video Vision Transformercited by 1,610
  12. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  13. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  14. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  15. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  16. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  17. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  18. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  19. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  20. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  21. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  22. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  23. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 899
  24. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  25. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 643
  26. 2021Score-based Generative Modeling in Latent Spacecited by 634
  27. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  28. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  29. 2023Zero-shot Image-to-Image Translationcited by 503
  30. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  31. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  32. 2017Learning Spatio-Temporal Representation with Pseudo-3D Residual Networkscited by 386
  33. 2023Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Modelscited by 358
  34. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 330
  35. 2021GMFlow: Learning Optical Flow via Global Matchingcited by 324
  36. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  37. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 303
  38. 2022Flexible Diffusion Modeling of Long Videoscited by 294

50 most cited of 76 All 76 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 67; 2024: 169; 2025: 149; 2026: 76.

461 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  2. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 683
  3. 2023ModelScope Text-to-Video Technical Reportcited by 651
  4. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 568
  5. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  6. 2025VACE: All-in-One Video Creation and Editingcited by 467
  7. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  8. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  9. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  10. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 343
  11. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 260
  12. 2023DriveDreamer: Towards Real-world-driven World Models for Autonomous Drivingcited by 258
38 more
  1. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  2. 2024StreamingT2V: Consistent, Dynamic, and Extendable Long Video Generation from Textcited by 242
  3. 2024Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modelingcited by 209
  4. 2023A Survey on Video Diffusion Modelscited by 203
  5. 2024Hallo: Hierarchical Audio-Driven Visual Synthesis for Portrait Image Animationcited by 198
  6. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 186
  7. 2024DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video Generationcited by 186
  8. 20234D-fy: Text-to-4D Generation Using Hybrid Score Distillation Samplingcited by 184
  9. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Controlcited by 184
  10. 2024Motion Prompting: Controlling Video Generation with Motion Trajectoriescited by 176
  11. 2023FreeNoise: Tuning-Free Longer Video Diffusion via Noise Reschedulingcited by 173
  12. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Modelscited by 172
  13. 2023SparseCtrl: Adding Sparse Controls to Text-to-Video Diffusion Modelscited by 172
  14. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 169
  15. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
  16. 2024Tora: Trajectory-oriented Diffusion Transformer for Video Generationcited by 161
  17. 2024VD3D: Taming Large Video Diffusion Transformers for 3D Camera Controlcited by 156
  18. 2024AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Taskscited by 151
  19. 2024AC3D: Analyzing and Improving 3D Camera Control in Video Diffusion Transformerscited by 151
  20. 2024MiraData: A Large-Scale Video Dataset with Long Durations and Structured Captionscited by 144
  21. 2023Make Pixels Dance: High-Dynamic Video Generationcited by 140
  22. 2023Upscale-A-Video: Temporal-Consistent Diffusion Model for Real-World Video Super-Resolutioncited by 131
  23. 2024ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generationcited by 130
  24. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  25. 2023Vision Language Models in Autonomous Driving: A Survey and Outlookcited by 124
  26. 2023Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfercited by 124
  27. 2024Hallo3: Highly Dynamic and Realistic Portrait Image Animation with Video Diffusion Transformercited by 121
  28. 2024Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Controlcited by 117
  29. 2024CamI2V: Camera-Controlled Image-to-Video Diffusion Modelcited by 115
  30. 2025Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noisecited by 114
  31. 2023Animate124: Animating One Image to 4D Dynamic Scenecited by 113
  32. 2024Boximator: Generating Rich and Controllable Motions for Video Synthesiscited by 107
  33. 2024MotionClone: Training-Free Motion Cloning for Controllable Video Generationcited by 104
  34. 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
  35. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animationcited by 101
  36. 2024Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenizationcited by 98
  37. 2024Hallo2: Long-Duration and High-Resolution Audio-Driven Portrait Image Animationcited by 95
  38. 2023Generative Image Dynamicscited by 93

50 most cited of 461 All 461 (JSON)