# Video Motion Transfer with Diffusion Transformers

arXiv: https://arxiv.org/abs/2412.07776 · HTML: https://mlsanity.com/paper/2412.07776/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 58 of its 66 references that are in the map.
- First posted on arXiv 10 Dec 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 58)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [Generative Modeling by Estimating Gradients of the Data Distribution](https://mlsanity.com/paper/1907.05600/) (2019)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Prompt-to-Prompt Image Editing with Cross Attention Control](https://mlsanity.com/paper/2208.01626/) (2022)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [The 2017 DAVIS Challenge on Video Object Segmentation](https://mlsanity.com/paper/1704.00675/) (2017)
- [PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesis](https://mlsanity.com/paper/2310.00426/) (2023)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)
- [Movie Gen: A Cast of Media Foundation Models](https://mlsanity.com/paper/2410.13720/) (2024)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [MotionCtrl: A Unified and Flexible Motion Controller for Video Generation](https://mlsanity.com/paper/2312.03641/) (2023)

## Built on by (most cited first, 30 of 39)

- [Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Control](https://mlsanity.com/paper/2501.03847/) (2025)
- [Follow-Your-Motion: Video Motion Transfer via Efficient Spatial-Temporal Decoupled Finetuning](https://mlsanity.com/paper/2506.05207/) (2025)
- [FastVMT: Eliminating Redundancy in Video Motion Transfer](https://mlsanity.com/paper/2602.05551/) (2026)
- [Decouple and Track: Benchmarking and Improving Video Diffusion Transformers for Motion Transfer](https://mlsanity.com/paper/2503.17350/) (2025)
- [RoPECraft: Training-Free Motion Transfer with Trajectory-Guided RoPE Optimization on Diffusion Transformers](https://mlsanity.com/paper/2505.13344/) (2025)
- [Video-As-Prompt: Unified Semantic Control for Video Generation](https://mlsanity.com/paper/2510.20888/) (2025)
- [Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoising](https://mlsanity.com/paper/2511.08633/) (2025)
- [Video2Act: A Dual-System Video Diffusion Policy with Robotic Spatio-Motional Modeling](https://mlsanity.com/paper/2512.03044/) (2025)
- [DualCamCtrl: Dual-Branch Diffusion Model for Geometry-Aware Camera-Controlled Video Generation](https://mlsanity.com/paper/2511.23127/) (2025)
- [FlexTraj: Image-to-Video Generation with Flexible Point Trajectory Control](https://mlsanity.com/paper/2510.08527/) (2025)
- [OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfer](https://mlsanity.com/paper/2601.14250/) (2026)
- [Proprio: Latent Self-Scoring and Inference-Time Refinement for Physically Plausible Video Generation](https://mlsanity.com/paper/2605.28230/) (2026)
- [TempoControl: Temporal Attention Guidance for Text-to-Video Models](https://mlsanity.com/paper/2510.02226/) (2025)
- [ActionParty: Multi-Subject Action Binding in Generative Video Games](https://mlsanity.com/paper/2604.02330/) (2026)
- [Versatile Editing of Video Content, Actions, and Dynamics without Training](https://mlsanity.com/paper/2603.17989/) (2026)
- [LMP: Leveraging Motion Prior in Zero-Shot Video Generation with Diffusion Transformer](https://mlsanity.com/paper/2505.14167/) (2025)
- [SemanticMoments: Training-Free Motion Similarity via Third Moment Features](https://mlsanity.com/paper/2602.09146/) (2026)
- [Let Your Image Move with Your Motion! -- Implicit Multi-Object Multi-Motion Transfer](https://mlsanity.com/paper/2603.01000/) (2026)
- [Towards Efficient Real-Time Video Motion Transfer via Generative Time Series Modeling](https://mlsanity.com/paper/2504.05537/) (2025)
- [When Distillation Breaks Motion Control: Restoring Generative Trajectories for Fast Video Generators](https://mlsanity.com/paper/2506.19348/) (2025)
- [Learning to Generate Rigid Body Interactions with Video Diffusion Models](https://mlsanity.com/paper/2510.02284/) (2025)
- [Phase-Aligned RoPE for Mixed-Resolution Diffusion Transformer](https://mlsanity.com/paper/2511.19778/) (2025)
- [Moaw: Unleashing Motion Awareness for Video Diffusion Models](https://mlsanity.com/paper/2601.12761/) (2026)
- [Interpretable Motion-Attentive Maps: Spatio-Temporally Localizing Concepts in Video Diffusion Transformers](https://mlsanity.com/paper/2603.02919/) (2026)
- [FlowMotion: Training-Free Flow Guidance for Video Motion Transfer](https://mlsanity.com/paper/2603.06289/) (2026)
- [Fast 4D Mesh Generation by Spatio-Temporal Attention Chains](https://mlsanity.com/paper/2605.19786/) (2026)
- [MotionAdapter: Video Motion Transfer via Content-Aware Attention Customization](https://mlsanity.com/paper/2601.01955/) (2026)
- [Video2LoRA: Unified Semantic-Controlled Video Generation via Per-Reference-Video LoRA](https://mlsanity.com/paper/2603.08210/) (2026)
- [MotionGrounder: Grounded Multi-Object Motion Transfer via Diffusion Transformer](https://mlsanity.com/paper/2604.00853/) (2026)
- [ActCam: Zero-Shot Joint Camera and 3D Motion Control for Video Generation](https://mlsanity.com/paper/2605.06667/) (2026)

## Abstract (arXiv)

We propose DiTFlow, a method for transferring the motion of a reference video to a newly synthesized one, designed specifically for Diffusion Transformers (DiT). We first process the reference video with a pre-trained DiT to analyze cross-frame attention maps and extract a patch-wise motion signal called the Attention Motion Flow (AMF). We guide the latent denoising process in an optimization-based, training-free, manner by optimizing latents with our AMF loss to generate videos reproducing the motion of the reference one. We also apply our optimization strategy to transformer positional embeddings, granting us a boost in zero-shot motion transfer capabilities. We evaluate DiTFlow against recently published methods, outperforming all across multiple metrics and human evaluation.