mlsanityData to 7 Oct 2026

Make-Your-Video: Customized Video Generation Using Textual and Structural Guidance

arXiv 2306.00943 ↗First posted 1 Jun 2023cs.CVcited by 129

Not on the map. Why?

Cited by 58 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Creating a vivid video from the event or scenario in our imagination is a truly fascinating experience. Recent advancements in text-to-video synthesis have unveiled the potential to achieve this with prompts only. While text is convenient in conveying the overall scene context, it may be insufficient to control precisely. In this paper, we explore customized video generation by utilizing text as context description and motion structure (e.g. frame-wise depth) as concrete guidance. Our method, dubbed Make-Your-Video, involves joint-conditional video generation using a Latent Diffusion Model that is pre-trained for still image synthesis and then promoted for video generation with the introduction of temporal modules. This two-stage learning scheme not only reduces the computing resources required, but also improves the performance by transferring the rich concepts available in image datasets solely into video generation. Moreover, we use a simple yet effective causal attention mask strategy to enable longer video synthesis, which mitigates the potential quality degradation effectively. Experimental results show the superiority of our method over existing baselines, particularly in terms of temporal coherence and fidelity to users' guidance. In addition, our model enables several intriguing applications that demonstrate potential for practical usage.

Key facts. Cited by 129 papers in the map (data through 7 Oct 2026). Builds on 47 of its 56 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

47 of 56 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2022Classifier-Free Diffusion Guidancecited by 5,717
35 more
  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  3. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  4. 2022Video Diffusion Modelscited by 2,280
  5. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  6. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  7. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  8. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  9. 2019Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfercited by 1,484
  10. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  11. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  12. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  13. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  14. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  15. 2018Self-Attention with Relative Position Representationscited by 1,112
  16. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  17. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  18. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  19. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  20. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  21. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  22. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  23. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  24. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 643
  25. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 589
  26. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  27. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 462
  28. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  29. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  30. 2023Follow Your Pose: Pose-Guided Text-to-Video Generation using Pose-Free Videoscited by 312
  31. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  32. 2023Dreamix: Video Diffusion Models are General Video Editorscited by 189
  33. 2022ReCo: Region-Controlled Text-to-Image Generationcited by 177
  34. 2021Layered Neural Atlases for Consistent Video Editingcited by 137
  35. 2022SceneComposer: Any-Level Semantic Image Synthesiscited by 43

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 20; 2024: 40; 2025: 45; 2026: 24.

129 papers.

Sort:

By area:

  1. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  2. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 530
  3. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  4. 2023DynamiCrafter: Animating Open-domain Images with Video Diffusion Priorscited by 463
  5. 2024CameraCtrl: Enabling Camera Control for Text-to-Video Generationcited by 448
  6. 2024ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesiscited by 392
  7. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Modelscited by 343
  8. 2025ReCamMaster: Camera-Controlled Generative Rendering from A Single Videocited by 278
  9. 2023A Survey on Video Diffusion Modelscited by 203
  10. 2024Follow-Your-Emoji: Fine-Controllable and Expressive Freestyle Portrait Animationcited by 172
  11. 2024ControlNeXt: Powerful and Efficient Control for Image and Video Generationcited by 158
  12. 2023ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Modelscited by 127
38 more
  1. 2024MotionClone: Training-Free Motion Cloning for Controllable Video Generationcited by 104
  2. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animationcited by 101
  3. 2025ACE++: Instruction-Based Image Creation and Editing via Context-Aware Content Fillingcited by 95
  4. 2025Controllable Video Generation: A Surveycited by 93
  5. 2023Animate-A-Story: Storytelling with Retrieval-Augmented Video Generationcited by 89
  6. 2024MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Modelcited by 87
  7. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 85
  8. 2025Follow-Your-Motion: Video Motion Transfer via Efficient Spatial-Temporal Decoupled Finetuningcited by 81
  9. 2023ARTboldsymbol·V: Auto-Regressive Text-to-Video Generation with Diffusion Modelscited by 74
  10. 2025DiffuEraser: A Diffusion Model for Video Inpaintingcited by 69
  11. 2024ToonCrafter: Generative Cartoon Interpolationcited by 68
  12. 2024Time-MMD: Multi-Domain Multimodal Dataset for Time Series Analysiscited by 65
  13. 2023MagicStick: Controllable Video Editing via Control Handle Transformationscited by 62
  14. 2025Follow-Your-Creation: Empowering 4D Creation through Video Inpaintingcited by 61
  15. 2024Magic-Me: Identity-Specific Video Customized Diffusioncited by 58
  16. 2024360DVD: Controllable Panorama Video Generation with 360-Degree Video Diffusion Modelcited by 56
  17. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  18. 2024Follow-Your-Click: Open-domain Regional Image Animation via Short Promptscited by 52
  19. 2023A Recipe for Scaling up Text-to-Video Generation with Text-free Videoscited by 51
  20. 2025MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video Generationcited by 50
  21. 2024VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesiscited by 48
  22. 2023RenAIssance: A Survey into AI Text-to-Image Generation in the Era of Large Modelcited by 47
  23. 2024UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editingcited by 47
  24. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  25. 2025Follow-Your-Emoji-Faster: Towards Efficient, Fine-Controllable, and Expressive Freestyle Portrait Animationcited by 41
  26. 2025InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbingcited by 40
  27. 2024Video Motion Transfer with Diffusion Transformerscited by 39
  28. 2024LeviTor: 3D Trajectory Oriented Image-to-Video Synthesiscited by 39
  29. 2024StereoCrafter: Diffusion-based Generation of Long and High-fidelity Stereoscopic 3D from Monocular Videoscited by 36
  30. 2024Generative Video Propagationcited by 36
  31. 2024Make-Your-3D: Fast and Consistent Subject-Driven 3D Content Generationcited by 32
  32. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  33. 2024Video Diffusion Models: A Surveycited by 28
  34. 2023MicroCinema: A Divide-and-Conquer Approach for Text-to-Video Generationcited by 26
  35. 2025Code2Video: A Code-centric Paradigm for Educational Video Generationcited by 23
  36. 2026FastVMT: Eliminating Redundancy in Video Motion Transfercited by 23
  37. 2025OmniVDiff: Omni Controllable Video Diffusion for Generation and Understandingcited by 22
  38. 2024DisenStudio: Customized Multi-subject Text-to-Video Generation with Disentangled Spatial Controlcited by 20

50 most cited of 129 All 129 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023MEVG: Multi-event Video Generation with Text-to-Video Models
  2. 2023VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation
  3. 2025Anchored Video Generation: Decoupling Scene Construction and Temporal Synthesis in Text-to-Video Diffusion Models
  4. 2023DreamVideo: High-Fidelity Image-to-Video Generation with Image Retention and Text Guidance
  5. 2024Animate Your Motion: Turning Still Images into Dynamic Videos
  6. 2024Video Diffusion Transformers are In-Context Learners
  7. 2023VideoBooth: Diffusion-based Video Generation with Image Prompts
  8. 2026Image-to-Video Diffusion: From Foundations to Open Frontiers