mlsanityData to 7 Oct 2026

DreamVideo: Composing Your Dream Videos with Customized Subject and Motion

arXiv 2312.04433 ↗First posted 7 Dec 2023cs.CVcited by 186

On the map: Diffusion models → Video generation

Abstract (arXiv)

Customized generation using diffusion models has made impressive progress in image generation, but remains unsatisfactory in the challenging video generation task, as it requires the controllability of both subjects and motions. To that end, we present DreamVideo, a novel approach to generating personalized videos from a few static images of the desired subject and a few videos of target motion. DreamVideo decouples this task into two stages, subject learning and motion learning, by leveraging a pre-trained video diffusion model. The subject learning aims to accurately capture the fine appearance of the subject from provided images, which is achieved by combining textual inversion and fine-tuning of our carefully designed identity adapter. In motion learning, we architect a motion adapter and fine-tune it on the given videos to effectively model the target motion pattern. Combining these two lightweight and efficient adapters allows for flexible customization of any subject with any motion. Extensive experimental results demonstrate the superior performance of our DreamVideo over the state-of-the-art methods for customized video generation. Our project page is at https://dreamvideo-t2v.github.io.

Key facts. Cited by 186 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 80 of its 96 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

80 of 96 references.

Closest ancestors · 28

Same direction, newest first

  1. 2023I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models7 Nov
  2. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generation30 Oct
  3. 2023LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation16 Oct
  4. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Models12 Oct
  5. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation27 Sep
  6. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models26 Sep
  7. 2023VideoGen: A Reference-Guided Latent Diffusion Approach for High Definition Text-to-Video Generation1 Sep
  8. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editing18 Aug
  9. 2023ModelScope Text-to-Video Technical Report12 Aug
  10. 2023DiffSynth: Latent In-Iteration Deflickering for Realistic Video Synthesis7 Aug
  11. 2023AnyDoor: Zero-shot Object-level Image Customization18 Jul
  12. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
16 more
  1. 2023ControlVideo: Training-free Controllable Text-to-Video Generation22 May
  2. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models17 May
  3. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  4. 2023Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation17 Apr
  5. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  6. 2023Pix2Video: Video Editing using Image Diffusion22 Mar
  7. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editing16 Mar
  8. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation15 Mar
  9. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  10. 2023Dreamix: Video Diffusion Models are General Video Editors2 Feb
  11. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  12. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  13. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  14. 2022Flexible Diffusion Modeling of Long Videos23 May
  15. 2022Video Diffusion Models7 Apr
  16. 2022Diffusion Probabilistic Modeling for Video Generation16 Mar

Other references · 52

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2020Denoising Diffusion Implicit Modelscited by 9,655
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2022Classifier-Free Diffusion Guidancecited by 5,717
  8. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  10. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  11. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  12. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
38 more
  1. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
  2. 2023VBench: Comprehensive Benchmark Suite for Video Generative Modelscited by 1,755
  3. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  4. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  5. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  6. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 973
  7. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  8. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  9. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  10. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 523
  11. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  12. 2023ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generationcited by 414
  13. 2016Generating Videos with Scene Dynamicscited by 376
  14. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 343
  15. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  16. 2022Analytic-DPM: an Analytic Estimate of the Optimal Reverse Variance in Diffusion Probabilistic Modelscited by 331
  17. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attentioncited by 313
  18. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  19. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  20. 2021StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2cited by 300
  21. 2022Exploring Visual Prompts for Adapting Large-Scale Modelscited by 271
  22. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
  23. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  24. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 230
  25. 2023HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Modelscited by 224
  26. 2023Subject-driven Text-to-Image Generation via Apprenticeship Learningcited by 210
  27. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 192
  28. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuningcited by 191
  29. 2021A Good Image Generator Is What You Need for High-Resolution Video Synthesiscited by 162
  30. 2023Cones: Concept Neurons in Diffusion Models for Customized Generationcited by 150
  31. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 145
  32. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 138
  33. 2020Few-shot Image Generation with Elastic Weight Consolidationcited by 122
  34. 2023Cones 2: Customizable Image Synthesis with Multiple Subjectscited by 101
  35. 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
  36. 2023Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animatorcited by 74
  37. 2021CCVS: Context-aware Controllable Video Synthesiscited by 68
  38. 2023DirecT2V: Large Language Models are Frame-Level Directors for Zero-Shot Text-to-Video Generationcited by 47

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 50; 2025: 89; 2026: 43.

186 papers.

Sort:

By area:

  1. 2024Timestep Embedding Tells: It's Time to Cache for Video Diffusion Modelcited by 245
  2. 2024PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generationcited by 169
  3. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
  4. 2024Identity-Preserving Text-to-Video Generation by Frequency Decompositioncited by 155
  5. 2024DimensionX: Create Any 3D and 4D Scenes from a Single Image with Controllable Video Diffusioncited by 139
  6. 2024UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animationcited by 101
  7. 2024MotionBooth: Motion-Aware Customized Text-to-Video Generationcited by 92
  8. 2023VideoLCM: Video Latent Consistency Modelcited by 90
  9. 2023InstructVideo: Instructing Video Diffusion Models with Human Feedbackcited by 85
  10. 2024DeMamba: AI-Generated Video Detection on Million-Scale GenVideo Benchmarkcited by 85
  11. 2024Controllable Generation with Text-to-Image Diffusion Models: A Surveycited by 82
  12. 2025Automated Movie Generation via Multi-Agent CoT Planningcited by 82
38 more
  1. 2024CustomVideo: Customizing Text-to-Video Generation with Multiple Subjectscited by 76
  2. 2024ReVideo: Remake a Video with Motion and Content Controlcited by 74
  3. 2025OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generationcited by 73
  4. 2025SkyReels-A2: Compose Anything in Video Diffusion Transformerscited by 72
  5. 2025ConceptMaster: Multi-Concept Video Customization on Diffusion Transformer Models Without Test-Time Tuningcited by 67
  6. 2025Multi-subject Open-set Personalization in Video Generationcited by 65
  7. 2024Customize-A-Video: One-Shot Motion Customization of Text-to-Video Diffusion Modelscited by 55
  8. 2024FreeTraj: Tuning-Free Trajectory Control in Video Diffusion Modelscited by 55
  9. 2023A Recipe for Scaling up Text-to-Video Generation with Text-free Videoscited by 51
  10. 2024Still-Moving: Customized Video Generation without Customized Video Datacited by 49
  11. 2024PhysMotion: Physics-Grounded Dynamics From a Single Imagecited by 45
  12. 2025EEdit: Rethinking the Spatial and Temporal Redundancy for Efficient Image Editingcited by 45
  13. 2025FullDiT: Multi-Task Video Generative Foundation Model with Full Attentioncited by 45
  14. 2024DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise Motion Controlcited by 42
  15. 2025VideoAnydoor: High-fidelity Video Object Insertion with Precise Motion Controlcited by 42
  16. 2025PhysAnimator: Physics-Guided Generative Cartoon Animationcited by 40
  17. 2024I2VControl-Camera: Precise Video Camera Control with Adjustable Motion Strengthcited by 38
  18. 2025MagCache: Fast Video Generation with Magnitude-Aware Cachecited by 36
  19. 2024MovieBench: A Hierarchical Movie Level Dataset for Long Video Generationcited by 35
  20. 2026DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Modelscited by 35
  21. 2024Make-Your-3D: Fast and Consistent Subject-Driven 3D Content Generationcited by 32
  22. 2025D3: Training-Free AI-Generated Video Detection Using Second-Order Featurescited by 32
  23. 2024A Survey on Personalized Content Synthesis with Diffusion Modelscited by 29
  24. 2024PersonalVideo: High ID-Fidelity Video Customization without Dynamic and Semantic Degradationcited by 29
  25. 2024FreeScale: Unleashing the Resolution of Diffusion Models via Tuning-Free Scale Fusioncited by 29
  26. 2024DreamMotion: Space-Time Self-Similar Score Distillation for Zero-Shot Video Editingcited by 28
  27. 2024StoryAgent: Customized Storytelling Video Generation via Multi-Agent Collaborationcited by 28
  28. 2024Is Your World Simulator a Good Story Presenter? A Consecutive Events-Based Benchmark for Future Long Video Generationcited by 28
  29. 2025Movie Weaver: Tuning-Free Multi-Concept Video Personalization with Anchored Promptscited by 28
  30. 2025Stand-In: A Lightweight and Plug-and-Play Identity Control for Video Generationcited by 27
  31. 2024From Sora What We Can See: A Survey of Text-to-Video Generationcited by 26
  32. 2025Generative Physical AI in Vision: A Surveycited by 24
  33. 2025CAPability: A Comprehensive Visual Caption Benchmark for Evaluating Both Correctness and Thoroughnesscited by 23
  34. 2025DreamRelation: Relation-Centric Video Customizationcited by 23
  35. 2025Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidancecited by 23
  36. 2026VideoMemory: Toward Consistent Video Generation via Memory Integrationcited by 22
  37. 2024Spectral Motion Alignment for Video Motion Transfer using Diffusion Modelscited by 20
  38. 2024DisenStudio: Customized Multi-subject Text-to-Video Generation with Disentangled Spatial Controlcited by 20

50 most cited of 186 All 186 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023DreamVideo: High-Fidelity Image-to-Video Generation with Image Retention and Text Guidance
  2. 2023DreaMoving: A Human Video Generation Framework based on Diffusion Models
  3. 2024CustomCrafter: Customized Video Generation with Preserving Motion and Concept Composition Abilities
  4. 2025DreamInsert: Zero-Shot Image-to-Video Object Insertion from A Single Image
  5. 2023VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models
  6. 2025LMP: Leveraging Motion Prior in Zero-Shot Video Generation with Diffusion Transformer
  7. 2024Optical-Flow Guided Prompt Optimization for Coherent Video Generation
  8. 2023Make Pixels Dance: High-Dynamic Video Generation