mlsanityData to 14 Oct 2026

Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer

arXiv 2311.17009 ↗First posted 28 Nov 2023cs.CVcited by 124

Not on the map. Why?

Cited by 59 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present a new method for text-driven motion transfer - synthesizing a video that complies with an input text prompt describing the target objects and scene while maintaining an input video's motion and scene layout. Prior methods are confined to transferring motion across two subjects within the same or closely related object categories and are applicable for limited domains (e.g., humans). In this work, we consider a significantly more challenging setting in which the target and source objects differ drastically in shape and fine-grained motion characteristics (e.g., translating a jumping dog into a dolphin). To this end, we leverage a pre-trained and fixed text-to-video diffusion model, which provides us with generative and motion priors. The pillar of our method is a new space-time feature loss derived directly from the model. This loss guides the generation process to preserve the overall motion of the input video while complying with the target object in terms of shape and fine-grained motion traits.

Key facts. Cited by 124 papers in the map (data through 14 Oct 2026). Builds on 65 of its 70 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

65 of 70 references.

References · 65

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  9. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  10. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
  11. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  12. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
38 more
  1. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  2. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  3. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,064
  4. 2022Diffusion Models in Vision: A Surveycited by 1,015
  5. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  6. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 961
  7. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
  8. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  9. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 727
  10. 2020First Order Motion Model for Image Animationcited by 704
  11. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 702
  12. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 670
  13. 2023ModelScope Text-to-Video Technical Reportcited by 658
  14. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
  15. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 536
  16. 2023CoTracker: It is Better to Track Togethercited by 522
  17. 2023Zero-shot Image-to-Image Translationcited by 505
  18. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  19. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 466
  20. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  21. 2023Emergent Correspondence from Image Diffusioncited by 446
  22. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 442
  23. 2014Understanding Deep Image Representations by Inverting Themcited by 403
  24. 2023Pix2Video: Video Editing using Image Diffusioncited by 320
  25. 2018Everybody Dance Nowcited by 315
  26. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  27. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  28. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 285
  29. 2018Deep Video Portraitscited by 267
  30. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  31. 2022Splicing ViT Features for Semantic Appearance Transfercited by 209
  32. 2018Animating Arbitrary Objects via Deep Motion Transfercited by 208
  33. 2023DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Modelscited by 195
  34. 2023Dreamix: Video Diffusion Models are General Video Editorscited by 191
  35. 2018X2Face: A network for controlling face generation by using images, audio, and pose codescited by 189
  36. 2023StableVideo: Text-driven Consistency-aware Diffusion Video Editingcited by 183
  37. 2017Video Generation From Textcited by 154
  38. 2022Improving Sample Quality of Diffusion Models Using Self-Attention Guidancecited by 154

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 33; 2025: 50; 2026: 40.

124 papers.

Sort:

By area:

  1. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Controlcited by 185
  2. 2025Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noisecited by 117
  3. 2025AdaWorld: Learning Adaptable World Models with Latent Actionscited by 114
  4. 2024Adaptive Caching for Faster Video Generation with Diffusion Transformerscited by 108
  5. 2025Follow-Your-Motion: Video Motion Transfer via Efficient Spatial-Temporal Decoupled Finetuningcited by 81
  6. 2024ReVideo: Remake a Video with Motion and Content Controlcited by 74
  7. 2024Video Diffusion Models are Training-free Motion Interpreter and Controllercited by 61
  8. 2024I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Modelscited by 55
  9. 2024Diffusion Models and Representation Learning: A Surveycited by 54
  10. 2025EditVerse: Unifying Image and Video Editing and Generation with In-Context Learningcited by 54
  11. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  12. 2025VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editingcited by 50
38 more
  1. 2024PhysMotion: Physics-Grounded Dynamics From a Single Imagecited by 45
  2. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  3. 2024DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise Motion Controlcited by 43
  4. 2024Video Editing via Factorized Diffusion Distillationcited by 41
  5. 2024Video Motion Transfer with Diffusion Transformerscited by 39
  6. 2024Generative Video Propagationcited by 36
  7. 2025Reangle-A-Video: 4D Video Generation as Video-to-Video Translationcited by 35
  8. 2024Monkey See, Monkey Do: Harnessing Self-attention in Motion Diffusion for Zero-shot Motion Transfercited by 31
  9. 2024Motion Inversion for Video Customizationcited by 30
  10. 2024DreamMotion: Space-Time Self-Similar Score Distillation for Zero-Shot Video Editingcited by 28
  11. 2025IVEBench: Modern Benchmark Suite for Instruction-Guided Video Editing Assessmentcited by 28
  12. 2024Videoshop: Localized Semantic Video Editing with Noise-Extrapolated Diffusion Inversioncited by 27
  13. 2024DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cutcited by 26
  14. 2024StableV2V: Stablizing Shape Consistency in Video-to-Video Editingcited by 26
  15. 2025FlexiAct: Towards Flexible Action Control in Heterogeneous Scenarioscited by 25
  16. 2025DreamRelation: Relation-Centric Video Customizationcited by 23
  17. 2025Generative Video Motion Editing with 3D Point Trackscited by 23
  18. 2026FastVMT: Eliminating Redundancy in Video Motion Transfercited by 23
  19. 2024HOI-Swap: Swapping Objects in Videos with Hand-Object Interaction Awarenesscited by 21
  20. 2024Spectral Motion Alignment for Video Motion Transfer using Diffusion Modelscited by 20
  21. 2024Towards Precise Scaling Laws for Video Diffusion Transformerscited by 20
  22. 2025Re-HOLD: Video Hand Object Interaction Reenactment via adaptive Layout-instructed Diffusion Modelcited by 19
  23. 2026Omni-Video 2: Scaling MLLM-Conditioned Diffusion for Unified Video Generation and Editingcited by 19
  24. 2023NewMove: Customizing text-to-video models with novel motionscited by 18
  25. 2024MagicPose4D: Crafting Articulated Models with Appearance and Motion Controlcited by 18
  26. 2025FiVE: A Fine-grained Video Editing Benchmark for Evaluating Emerging Diffusion and Rectified Flow Modelscited by 16
  27. 2025SketchVideo: Sketch-based Video Generation and Editingcited by 16
  28. 2024OnlyFlow: Optical Flow based Motion Conditioning for Video Diffusion Modelscited by 15
  29. 2025Free-Form Motion Control: Controlling the 6D Poses of Camera and Objects in Video Generationcited by 15
  30. 2025Decouple and Track: Benchmarking and Improving Video Diffusion Transformers for Motion Transfercited by 14
  31. 2025RoPECraft: Training-Free Motion Transfer with Trajectory-Guided RoPE Optimization on Diffusion Transformerscited by 14
  32. 2025Video-As-Prompt: Unified Semantic Control for Video Generationcited by 14
  33. 2025BulletTime: Decoupled Control of Time and Camera Pose for Video Generationcited by 14
  34. 2025Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoisingcited by 13
  35. 2025EasyV2V: A High-quality Instruction-based Video Editing Frameworkcited by 13
  36. 2024Sync4D: Video Guided Controllable Dynamics for Physics-Based 4D Generationcited by 12
  37. 2024Video Diffusion Transformers are In-Context Learnerscited by 12
  38. 2024InterDyn: Controllable Interactive Dynamics with Video Diffusion Modelscited by 12

50 most cited of 124 All 124 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LMP: Leveraging Motion Prior in Zero-Shot Video Generation with Diffusion Transformer
  2. 2024Animate Your Motion: Turning Still Images into Dynamic Videos
  3. 2024Towards motion from video diffusion models
  4. 2024MoTrans: Customized Motion Transfer with Text-driven Video Diffusion Models
  5. 2023DiffusionPhase: Motion Diffusion in Frequency Domain
  6. 2024Searching Priors Makes Text-to-Video Synthesis Better
  7. 2025PackDiT: Joint Human Motion and Text Generation via Mutual Prompting
  8. 2023Priority-Centric Human Motion Generation in Discrete Latent Space