mlsanityData to 7 Oct 2026

VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models

arXiv 2312.00845 ↗First posted 1 Dec 2023cs.CV cs.AI cs.LGcited by 86

On the map: Diffusion models → Video generation

Abstract (arXiv)

Text-to-video diffusion models have advanced video generation significantly. However, customizing these models to generate videos with tailored motions presents a substantial challenge. In specific, they encounter hurdles in (a) accurately reproducing motion from a target video, and (b) creating diverse visual variations. For example, straightforward extensions of static image customization methods to video often lead to intricate entanglements of appearance and motion data. To tackle this, here we present the Video Motion Customization (VMC) framework, a novel one-shot tuning approach crafted to adapt temporal attention layers within video diffusion models. Our approach introduces a novel motion distillation objective using residual vectors between consecutive frames as a motion reference. The diffusion process then preserves low-frequency motion trajectories while mitigating high-frequency motion-unrelated noise in image space. We validate our method against state-of-the-art video generative models across diverse real-world motions and contexts. Our codes, data and the project demo can be found at https://video-motion-customization.github.io

Key facts. Cited by 86 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 35 of its 45 references that are in the map. First posted on arXiv 1 Dec 2023.

Builds on

35 of 45 references.

Closest ancestors · 13

Same direction, newest first

  1. 2023LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation16 Oct
  2. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Models12 Oct
  3. 2023Ground-A-Video: Zero-shot Grounded Video Editing using Text-to-image Diffusion Models2 Oct
  4. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation27 Sep
  5. 2023LAVIE: High-Quality Video Generation with Cascaded Latent Diffusion Models26 Sep
  6. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models17 May
  7. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  8. 2023Pix2Video: Video Editing using Image Diffusion22 Mar
  9. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  10. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  11. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Models20 Nov
  12. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
1 more
  1. 2022Video Diffusion Models7 Apr

Other references · 22

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  10. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  11. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  12. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
10 more
  1. 2017Mixed Precision Trainingcited by 1,135
  2. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 942
  3. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 461
  4. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 440
  5. 2023ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generationcited by 414
  6. 2023InstantBooth: Personalized Text-to-Image Generation without Test-Time Finetuningcited by 343
  7. 2023SVDiff: Compact Parameter Space for Diffusion Fine-Tuningcited by 338
  8. 2023Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Modelscited by 261
  9. 2023HyperDreamBooth: HyperNetworks for Fast Personalization of Text-to-Image Modelscited by 224
  10. 2023Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuningcited by 191

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 31; 2025: 37; 2026: 17.

86 papers.

Sort:

By area:

  1. 2024Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motioncited by 167
  2. 2024Tora: Trajectory-oriented Diffusion Transformer for Video Generationcited by 161
  3. 2024MotionClone: Training-Free Motion Cloning for Controllable Video Generationcited by 104
  4. 2025Follow-Your-Motion: Video Motion Transfer via Efficient Spatial-Temporal Decoupled Finetuningcited by 81
  5. 20243DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generationcited by 71
  6. 2024Customize-A-Video: One-Shot Motion Customization of Text-to-Video Diffusion Modelscited by 55
  7. 2024I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Modelscited by 55
  8. 2024UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editingcited by 47
  9. 2024PhysMotion: Physics-Grounded Dynamics From a Single Imagecited by 45
  10. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  11. 2024DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise Motion Controlcited by 42
  12. 2025PhysAnimator: Physics-Guided Generative Cartoon Animationcited by 40
38 more
  1. 2024Video Motion Transfer with Diffusion Transformerscited by 39
  2. 2024Generative Video Propagationcited by 36
  3. 2025Reangle-A-Video: 4D Video Generation as Video-to-Video Translationcited by 35
  4. 2025Exploring the Evolution of Physics Cognition in Video Generation: A Surveycited by 31
  5. 2025FlowAlign: Trajectory-Regularized, Inversion-Free Flow-based Image Editingcited by 31
  6. 2024Motion Inversion for Video Customizationcited by 30
  7. 2024A Survey on Personalized Content Synthesis with Diffusion Modelscited by 29
  8. 2024DreamMotion: Space-Time Self-Similar Score Distillation for Zero-Shot Video Editingcited by 28
  9. 2025ContextFlow: Training-Free Video Object Editing via Adaptive Context Enrichmentcited by 27
  10. 2025DreamRelation: Relation-Centric Video Customizationcited by 23
  11. 2026FastVMT: Eliminating Redundancy in Video Motion Transfercited by 23
  12. 2025Generative Video Motion Editing with 3D Point Trackscited by 21
  13. 2024Spectral Motion Alignment for Video Motion Transfer using Diffusion Modelscited by 20
  14. 2023NewMove: Customizing text-to-video models with novel motionscited by 18
  15. 2025VideoMage: Multi-Subject and Motion Customization of Text-to-Video Diffusion Modelscited by 17
  16. 2025Free-Form Motion Control: Controlling the 6D Poses of Camera and Objects in Video Generationcited by 15
  17. 2025VFX Creator: Animated Visual Effect Generation with Controllable Diffusion Transformercited by 15
  18. 2024TweedieMix: Improving Multi-Concept Fusion for Diffusion-based Image/Video Generationcited by 14
  19. 2025VideoHandles: Editing 3D Object Compositions in Videos Using Video Generative Priorscited by 14
  20. 2025Decouple and Track: Benchmarking and Improving Video Diffusion Transformers for Motion Transfercited by 14
  21. 2025RoPECraft: Training-Free Motion Transfer with Trajectory-Guided RoPE Optimization on Diffusion Transformerscited by 14
  22. 2025Video-As-Prompt: Unified Semantic Control for Video Generationcited by 13
  23. 2025Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoisingcited by 13
  24. 2025ConMo: Controllable Motion Disentanglement and Recomposition for Zero-Shot Motion Transfercited by 12
  25. 2025MagicTryOn: Harnessing Diffusion Transformer for Garment-Preserving Video Virtual Try-oncited by 12
  26. 2024Edit-Your-Motion: Space-Time Diffusion Decoupling Learning for Video Motion Editingcited by 10
  27. 2025DisMo: Disentangled Motion Representations for Open-World Motion Transfercited by 9
  28. 2024ContextualStory: Consistent Visual Storytelling with Spatially-Enhanced and Storyline Contextcited by 8
  29. 2024ByTheWay: Boost Your Text-to-Video Generation Model to Higher Quality in a Training-free Waycited by 7
  30. 2024DreamRunner: Fine-Grained Compositional Story-to-Video Generation with Retrieval-Augmented Motion Adaptationcited by 7
  31. 2025Articulate That Object Part (ATOP): 3D Part Articulation via Text and Motion Personalizationcited by 7
  32. 2025SynMotion: Semantic-Visual Adaptation for Motion Customized Video Generationcited by 7
  33. 2025MotionRAG: Motion Retrieval-Augmented Image-to-Video Generationcited by 7
  34. 2025Real-Time Motion-Controllable Autoregressive Video Diffusioncited by 7
  35. 2024Reenact Anything: Semantic Video Motion Transfer Using Motion-Textual Inversioncited by 6
  36. 2026OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfercited by 6
  37. 2024AudioScenic: Audio-Driven Video Scene Editingcited by 5
  38. 2024Unified Editing of Panorama, 3D Scenes, and Videos Through Disentangled Self-Attention Injectioncited by 4

50 most cited of 86 All 86 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Separate Motion from Appearance: Customizing Motion via Customizing Text-to-Video Diffusion Models
  2. 2023MotionCrafter: One-Shot Motion Customization of Diffusion Models
  3. 2024CustomCrafter: Customized Video Generation with Preserving Motion and Concept Composition Abilities
  4. 2023Dreamix: Video Diffusion Models are General Video Editors
  5. 2024CustomTTT: Motion and Appearance Customized Video Generation via Test-Time Training
  6. 2023Highly Detailed and Temporal Consistent Video Stylization via Synchronized Multi-Frame Diffusion
  7. 2024Animate Your Motion: Turning Still Images into Dynamic Videos
  8. 2024Video Diffusion Models are Training-free Motion Interpreter and Controller