# Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfer

arXiv: https://arxiv.org/abs/2311.17009 · HTML: https://mlsanity.com/paper/2311.17009/

## Key facts

- Cited by 124 papers in the map (data through 14 Oct 2026).
- Builds on 65 of its 70 references that are in the map.
- First posted on arXiv 28 Nov 2023.

## Builds on (most cited first, 30 of 65)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translation](https://mlsanity.com/paper/2211.12572/) (2022)
- [MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing](https://mlsanity.com/paper/2304.08465/) (2023)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)
- [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generation](https://mlsanity.com/paper/2302.08113/) (2023)
- [CoTracker: It is Better to Track Together](https://mlsanity.com/paper/2307.07635/) (2023)
- [Zero-shot Image-to-Image Translation](https://mlsanity.com/paper/2302.03027/) (2023)
- [Phenaki: Variable Length Video Generation From Open Domain Textual Description](https://mlsanity.com/paper/2210.02399/) (2022)

## Built on by (most cited first, 30 of 124)

- [Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control](https://mlsanity.com/paper/2501.03847/) (2025)
- [Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using Real-Time Warped Noise](https://mlsanity.com/paper/2501.08331/) (2025)
- [AdaWorld: Learning Adaptable World Models with Latent Actions](https://mlsanity.com/paper/2503.18938/) (2025)
- [Adaptive Caching for Faster Video Generation with Diffusion Transformers](https://mlsanity.com/paper/2411.02397/) (2024)
- [Follow-Your-Motion: Video Motion Transfer via Efficient Spatial-Temporal Decoupled Finetuning](https://mlsanity.com/paper/2506.05207/) (2025)
- [ReVideo: Remake a Video with Motion and Content Control](https://mlsanity.com/paper/2405.13865/) (2024)
- [Video Diffusion Models are Training-free Motion Interpreter and Controller](https://mlsanity.com/paper/2405.14864/) (2024)
- [I2VEdit: First-Frame-Guided Video Editing via Image-to-Video Diffusion Models](https://mlsanity.com/paper/2405.16537/) (2024)
- [Diffusion Models and Representation Learning: A Survey](https://mlsanity.com/paper/2407.00783/) (2024)
- [EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning](https://mlsanity.com/paper/2509.20360/) (2025)
- [Sora as a World Model? A Complete Survey on Text-to-Video Generation](https://mlsanity.com/paper/2403.05131/) (2024)
- [VideoGrain: Modulating Space-Time Attention for Multi-grained Video Editing](https://mlsanity.com/paper/2502.17258/) (2025)
- [PhysMotion: Physics-Grounded Dynamics From a Single Image](https://mlsanity.com/paper/2411.17189/) (2024)
- [Diffusion Model-Based Video Editing: A Survey](https://mlsanity.com/paper/2407.07111/) (2024)
- [DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise Motion Control](https://mlsanity.com/paper/2410.13830/) (2024)
- [Video Editing via Factorized Diffusion Distillation](https://mlsanity.com/paper/2403.09334/) (2024)
- [Video Motion Transfer with Diffusion Transformers](https://mlsanity.com/paper/2412.07776/) (2024)
- [Generative Video Propagation](https://mlsanity.com/paper/2412.19761/) (2024)
- [Reangle-A-Video: 4D Video Generation as Video-to-Video Translation](https://mlsanity.com/paper/2503.09151/) (2025)
- [Monkey See, Monkey Do: Harnessing Self-attention in Motion Diffusion for Zero-shot Motion Transfer](https://mlsanity.com/paper/2406.06508/) (2024)
- [Motion Inversion for Video Customization](https://mlsanity.com/paper/2403.20193/) (2024)
- [DreamMotion: Space-Time Self-Similar Score Distillation for Zero-Shot Video Editing](https://mlsanity.com/paper/2403.12002/) (2024)
- [IVEBench: Modern Benchmark Suite for Instruction-Guided Video Editing Assessment](https://mlsanity.com/paper/2510.11647/) (2025)
- [Videoshop: Localized Semantic Video Editing with Noise-Extrapolated Diffusion Inversion](https://mlsanity.com/paper/2403.14617/) (2024)
- [DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cut](https://mlsanity.com/paper/2406.02842/) (2024)
- [StableV2V: Stablizing Shape Consistency in Video-to-Video Editing](https://mlsanity.com/paper/2411.11045/) (2024)
- [FlexiAct: Towards Flexible Action Control in Heterogeneous Scenarios](https://mlsanity.com/paper/2505.03730/) (2025)
- [DreamRelation: Relation-Centric Video Customization](https://mlsanity.com/paper/2503.07602/) (2025)
- [Generative Video Motion Editing with 3D Point Tracks](https://mlsanity.com/paper/2512.02015/) (2025)
- [FastVMT: Eliminating Redundancy in Video Motion Transfer](https://mlsanity.com/paper/2602.05551/) (2026)

## Abstract (arXiv)

We present a new method for text-driven motion transfer - synthesizing a video that complies with an input text prompt describing the target objects and scene while maintaining an input video's motion and scene layout. Prior methods are confined to transferring motion across two subjects within the same or closely related object categories and are applicable for limited domains (e.g., humans). In this work, we consider a significantly more challenging setting in which the target and source objects differ drastically in shape and fine-grained motion characteristics (e.g., translating a jumping dog into a dolphin). To this end, we leverage a pre-trained and fixed text-to-video diffusion model, which provides us with generative and motion priors. The pillar of our method is a new space-time feature loss derived directly from the model. This loss guides the generation process to preserve the overall motion of the input video while complying with the target object in terms of shape and fine-grained motion traits.