mlsanityData to 14 Oct 2026

Controllable Video Generation: A Survey

arXiv 2507.16869 ↗First posted 22 Jul 2025cs.GR cs.CVcited by 98

Not on the map. Why?

Cited by 23 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

With the rapid development of AI-generated content (AIGC), video generation has emerged as one of its most dynamic and impactful subfields. In particular, the advancement of video generation foundation models has led to growing demand for controllable video generation methods that can more accurately reflect user intent. Most existing foundation models are designed for text-to-video generation, where text prompts alone are often insufficient to express complex, multi-modal, and fine-grained user requirements. This limitation makes it challenging for users to generate videos with precise control using current models. To address this issue, recent research has explored the integration of additional non-textual conditions, such as camera motion, depth maps, and human pose, to extend pretrained video generation models and enable more controllable video synthesis. These approaches aim to enhance the flexibility and practical applicability of AIGC-driven video generation systems. In this survey, we provide a systematic review of controllable video generation, covering both theoretical foundations and recent advances in the field. We begin by introducing the key concepts and commonly used open-source video generation models. We then focus on control mechanisms in video diffusion models, analyzing how different types of conditions can be incorporated into the denoising process to guide generation. Finally, we categorize existing methods based on the types of control signals they leverage, including single-condition generation, multi-condition generation, and universal controllable generation. For a complete list of the literature on controllable video generation reviewed, please visit our curated repository at https://github.com/mayuelala/Awesome-Controllable-Video-Generation.

Key facts. Cited by 98 papers in the map (data through 14 Oct 2026). Builds on 226 of its 305 references that are in the map. First posted on arXiv 22 Jul 2025.

Builds on

226 of 305 references.

References · 226

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,523
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  5. 2018Neural Ordinary Differential Equationscited by 3,779
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
  7. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  8. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  9. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  10. 2016Density estimation using Real NVPcited by 2,042
  11. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  12. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
38 more
  1. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  2. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  3. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  4. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  5. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  6. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,162
  7. 2022MaskGIT: Masked Generative Image Transformercited by 1,099
  8. 2014NICE: Non-linear Independent Components Estimationcited by 1,058
  9. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  10. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  11. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
  12. 2025Cosmos World Foundation Model Platform for Physical AIcited by 864
  13. 2016Conditional Image Generation with PixelCNN Decoderscited by 862
  14. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  15. 2023Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animationcited by 688
  16. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  17. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
  18. 2018DensePose: Dense Human Pose Estimation In The Wildcited by 575
  19. 2023DL3DV-10K: A Large-Scale Scene Dataset for Deep Learning-based 3D Visioncited by 537
  20. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 535
  21. 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
  22. 2025VACE: All-in-One Video Creation and Editingcited by 479
  23. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 471
  24. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  25. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 460
  26. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 455
  27. 2024Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robotscited by 394
  28. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 393
  29. 2022Latent Video Diffusion Models for High-Fidelity Long Video Generationcited by 388
  30. 2024Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllabilitycited by 347
  31. 2025GEN3C: 3D-Informed World-Consistent Video Generation with Precise Camera Controlcited by 341
  32. 2024OminiControl: Minimal and Universal Control for Diffusion Transformercited by 340
  33. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 314
  34. 2024Open-Sora Plan: Open-Source Large Video Generation Modelcited by 296
  35. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 287
  36. 2025MAGI-1: Autoregressive Video Generation at Scalecited by 276
  37. 2024Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidancecited by 272
  38. 2021Motion Representations for Articulated Animationcited by 255

50 most cited of 226 All 226 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 37; 2026: 61.

98 papers.

Sort:

By area:

  1. 2025Follow-Your-Emoji-Faster: Towards Efficient, Fine-Controllable, and Expressive Freestyle Portrait Animationcited by 43
  2. 2025MultiShotMaster: A Controllable Multi-Shot Video Generation Frameworkcited by 36
  3. 2025In-Context Learning with Unpaired Clips for Instruction-based Video Editingcited by 35
  4. 2025JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimizationcited by 31
  5. 2025Aurora: Towards Universal Generative Multimodal Time Series Forecastingcited by 29
  6. 2025Follow-Your-Shape: Shape-Aware Image Editing via Trajectory-Guided Region Controlcited by 28
  7. 2025ContextFlow: Training-Free Video Object Editing via Adaptive Context Enrichmentcited by 27
  8. 2025What about gravity in video generation? Post-Training Newton's Laws with Verifiable Rewardscited by 25
  9. 2025Paper2Video: Automatic Video Generation from Scientific Paperscited by 24
  10. 2025WorldLens: Full-Spectrum Evaluations of Driving World Models in Real Worldcited by 23
  11. 2026FastVMT: Eliminating Redundancy in Video Motion Transfercited by 23
  12. 2025A Survey on Cache Methods in Diffusion Models: Toward Efficient Multi-Modal Generationcited by 16
38 more
  1. 2026Past- and Future-Informed KV Cache Policy with Salience Estimation in Autoregressive Video Diffusioncited by 16
  2. 2026Manifold-Aware Exploration for Reinforcement Learning in Video Generationcited by 15
  3. 2026LiveEdit: Towards Real-Time Diffusion-Based Streaming Video Editingcited by 15
  4. 2026GCGNet: Graph-Consistent Generative Network for Time Series Forecasting with Exogenous Variablescited by 13
  5. 2025VFXMaster: Unlocking Dynamic Visual Effect Generation via In-Context Learningcited by 12
  6. 2025UltraViCo: Breaking Extrapolation Limits in Video Diffusion Transformerscited by 12
  7. 2026VQQA: An Agentic Approach for Video Evaluation and Quality Improvementcited by 12
  8. 2025ReDirector: Creating Any-Length Video Retakes with Rotary Camera Encodingcited by 11
  9. 2026Group Editing: Edit Multiple Images in One Gocited by 11
  10. 2025Follow-Your-Instruction: A Comprehensive MLLM Agent for World Data Synthesiscited by 10
  11. 2025SMRABooth: Subject and Motion Representation Alignment for Customized Video Generationcited by 10
  12. 2025MIND-V: Hierarchical World Model for Long-Horizon Robotic Manipulation with RL-based Physical Alignmentcited by 9
  13. 2025ASTGI: Adaptive Spatio-Temporal Graph Interactions for Irregular Multivariate Time Series Forecastingcited by 8
  14. 2025UniPR-3D: Towards Universal Visual Place Recognition with Visual Geometry Grounded Transformercited by 8
  15. 2025CML-Bench: A Framework for Evaluating and Enhancing LLM-Powered Movie Scripts Generationcited by 7
  16. 2026VII: Visual Instruction Injection for Jailbreaking Image-to-Video Generation Modelscited by 7
  17. 2026CoRA: Boosting Time Series Foundation Models for Multivariate Forecasting through Correlation-aware Adaptercited by 7
  18. 2026LiveLight: Real-time Streaming Video Relighting with Interactive Controlcited by 7
  19. 2026UniFusion: A Unified Image Fusion Framework with Robust Representation and Source-Aware Preservationcited by 6
  20. 2026A2RD: Agentic Autoregressive Diffusion for Long Video Consistencycited by 6
  21. 2025OmniRefiner: Reinforcement-Guided Local Diffusion Refinementcited by 5
  22. 2026Embedding-perturbed Exploration Preference Optimization for Flow Modelscited by 5
  23. 2025NumeriKontrol: Adding Numeric Control to Diffusion Transformers for Instruction-based Image Editingcited by 4
  24. 2026GlyphBanana: Advancing Precise Text Rendering Through Agentic Workflowscited by 4
  25. 2026Immune2V: Image Immunization Against Dual-Stream Image-to-Video Generationcited by 4
  26. 2025FLAME: Flow Enhanced Legendre Memory Models for General Time Series Forecastingcited by 3
  27. 2025Active Intelligence in Video Avatars via Closed-loop World Modelingcited by 3
  28. 2026Towards Multimodal Time Series Anomaly Detection with Semantic Alignment and Condensed Interactioncited by 3
  29. 2026RealCam: Real-Time Novel-View Video Generation with Interactive Camera Controlcited by 3
  30. 2026Image-to-Video Diffusion: From Foundations to Open Frontierscited by 3
  31. 2026Smart-Insertion-V: Photorealistic Video Insertion via a Closed-Loop Feedback Dual-Stream Frameworkcited by 3
  32. 2025ChronosObserver: Taming 4D World with Hyperspace Diffusion Samplingcited by 2
  33. 2025AMap: Distilling Future Priors for Ahead-Aware Online HD Map Constructioncited by 2
  34. 2026AgriWorld:A World Tools Protocol Framework for Verifiable Agricultural Reasoning with Code-Executing LLM Agentscited by 2
  35. 2026ResWM: Residual-Action World Model for Visual RLcited by 2
  36. 2026ViFeEdit: A Video-Free Tuner of Your Video Diffusion Transformercited by 2
  37. 2026GTA: Advancing Image-to-3D World Generation via Geometry Then Appearance Video Diffusioncited by 2
  38. 2026Focused Forcing: Content-Aware Per-Frame KV Selection for Efficient Autoregressive Video Diffusioncited by 2

50 most cited of 98 All 98 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023ControlVideo: Training-free Controllable Text-to-Video Generation
  2. 2025Generating Human Motion Videos using a Cascaded Text-to-Video Framework
  3. 2023TrailBlazer: Trajectory Control for Diffusion-Based Video Generation
  4. 2024Video Diffusion Transformers are In-Context Learners
  5. 2025CtrlVDiff: Controllable Video Generation via Unified Multimodal Video Diffusion
  6. 2024Video Diffusion Models: A Survey
  7. 2025Resource-Efficient Motion Control for Video Generation via Dynamic Mask Guidance
  8. 2024EasyControl: Transfer ControlNet to Video Diffusion for Controllable Generation and Interpolation