mlsanityData to 14 Oct 2026

Video Motion Transfer with Diffusion Transformers

arXiv 2412.07776 ↗First posted 10 Dec 2024cs.CV cs.AI cs.LGcited by 39

On the map: Diffusion models → Video generation

Abstract (arXiv)

We propose DiTFlow, a method for transferring the motion of a reference video to a newly synthesized one, designed specifically for Diffusion Transformers (DiT). We first process the reference video with a pre-trained DiT to analyze cross-frame attention maps and extract a patch-wise motion signal called the Attention Motion Flow (AMF). We guide the latent denoising process in an optimization-based, training-free, manner by optimizing latents with our AMF loss to generate videos reproducing the motion of the reference one. We also apply our optimization strategy to transformer positional embeddings, granting us a boost in zero-shot motion transfer capabilities. We evaluate DiTFlow against recently published methods, outperforming all across multiple metrics and human evaluation.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 58 of its 66 references that are in the map. First posted on arXiv 10 Dec 2024.

Builds on

58 of 66 references.

Closest ancestors · 20

Same direction, newest first

  1. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer12 Aug
  2. 2024Tora: Trajectory-oriented Diffusion Transformer for Video Generation31 Jul
  3. 2024Video Diffusion Models are Training-free Motion Interpreter and Controller23 May
  4. 2024MotionMaster: Training-free Camera Motion Transfer For Video Generation24 Apr
  5. 2024Investigating the Effectiveness of Cross-Attention to Unlock Zero-Shot Editing of Text-to-Video Diffusion Models8 Apr
  6. 2024VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models17 Jan
  7. 2024Latte: Latent Diffusion Transformer for Video Generation5 Jan
  8. 2023Photorealistic Video Generation with Diffusion Models11 Dec
  9. 2023VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models1 Dec
  10. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets25 Nov
  11. 2023MotionDirector: Motion Customization of Text-to-Video Diffusion Models12 Oct
  12. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning10 Jul
8 more
  1. 2023VDT: General-purpose Video Diffusion Transformers via Mask Modeling22 May
  2. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models18 Apr
  3. 2023Zero-Shot Video Editing Using Off-The-Shelf Image Diffusion Models30 Mar
  4. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators23 Mar
  5. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editing16 Mar
  6. 2023Structure and Content-Guided Video Synthesis with Diffusion Models6 Feb
  7. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation22 Dec
  8. 2022Video Diffusion Models7 Apr

Other references · 38

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  8. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  9. 2022Scalable Diffusion Models with Transformerscited by 6,054
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  12. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
26 more
  1. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  2. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  3. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  4. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,245
  5. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,179
  6. 2018Self-Attention with Relative Position Representationscited by 1,120
  7. 2017The 2017 DAVIS Challenge on Video Object Segmentationcited by 957
  8. 2023PixArt-α: Fast Training of Diffusion Transformer for Photorealistic Text-to-Image Synthesiscited by 864
  9. 2024Movie Gen: A Cast of Media Foundation Modelscited by 644
  10. 2023MotionCtrl: A Unified and Flexible Motion Controller for Video Generationcited by 580
  11. 2023CoTracker: It is Better to Track Togethercited by 522
  12. 2023VideoComposer: Compositional Video Synthesis with Motion Controllabilitycited by 464
  13. 2023TokenFlow: Consistent Diffusion Features for Consistent Video Editingcited by 442
  14. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 314
  15. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  16. 2023DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectorycited by 264
  17. 2024AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Taskscited by 154
  18. 2023Make-Your-Video: Customized Video Generation Using Textual and Structural Guidancecited by 129
  19. 2023Space-Time Diffusion Features for Zero-Shot Text-Driven Motion Transfercited by 124
  20. 2024Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesiscited by 105
  21. 2024MotionClone: Training-Free Motion Cloning for Controllable Video Generationcited by 105
  22. 2023GenTron: Diffusion Transformers for Image and Video Generationcited by 82
  23. 2023AnimateAnything: Fine-Grained Open Domain Image Animation with Motion Guidancecited by 66
  24. 2024UniEdit: A Unified Tuning-Free Framework for Video Motion and Appearance Editingcited by 48
  25. 2024Reenact Anything: Semantic Video Motion Transfer Using Motion-Textual Inversioncited by 6
  26. 2024Translation-based Video-to-Video Synthesiscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 24.

39 papers.

Sort:

By area:

  1. 2025Diffusion as Shader: 3D-aware Video Diffusion for Versatile Video Generation Controlcited by 185
  2. 2025Follow-Your-Motion: Video Motion Transfer via Efficient Spatial-Temporal Decoupled Finetuningcited by 81
  3. 2026FastVMT: Eliminating Redundancy in Video Motion Transfercited by 23
  4. 2025Decouple and Track: Benchmarking and Improving Video Diffusion Transformers for Motion Transfercited by 14
  5. 2025RoPECraft: Training-Free Motion Transfer with Trajectory-Guided RoPE Optimization on Diffusion Transformerscited by 14
  6. 2025Video-As-Prompt: Unified Semantic Control for Video Generationcited by 14
  7. 2025Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoisingcited by 13
  8. 2025Video2Act: A Dual-System Video Diffusion Policy with Robotic Spatio-Motional Modelingcited by 13
  9. 2025DualCamCtrl: Dual-Branch Diffusion Model for Geometry-Aware Camera-Controlled Video Generationcited by 11
  10. 2025FlexTraj: Image-to-Video Generation with Flexible Point Trajectory Controlcited by 6
  11. 2026OmniTransfer: All-in-one Framework for Spatio-temporal Video Transfercited by 6
  12. 2026Proprio: Latent Self-Scoring and Inference-Time Refinement for Physically Plausible Video Generationcited by 6
27 more
  1. 2025TempoControl: Temporal Attention Guidance for Text-to-Video Modelscited by 5
  2. 2026ActionParty: Multi-Subject Action Binding in Generative Video Gamescited by 5
  3. 2026Versatile Editing of Video Content, Actions, and Dynamics without Trainingcited by 4
  4. 2025LMP: Leveraging Motion Prior in Zero-Shot Video Generation with Diffusion Transformercited by 3
  5. 2026SemanticMoments: Training-Free Motion Similarity via Third Moment Featurescited by 3
  6. 2026Let Your Image Move with Your Motion! -- Implicit Multi-Object Multi-Motion Transfercited by 3
  7. 2025Towards Efficient Real-Time Video Motion Transfer via Generative Time Series Modelingcited by 2
  8. 2025When Distillation Breaks Motion Control: Restoring Generative Trajectories for Fast Video Generatorscited by 2
  9. 2025Learning to Generate Rigid Body Interactions with Video Diffusion Modelscited by 2
  10. 2025Phase-Aligned RoPE for Mixed-Resolution Diffusion Transformercited by 2
  11. 2026Moaw: Unleashing Motion Awareness for Video Diffusion Modelscited by 1
  12. 2026Interpretable Motion-Attentive Maps: Spatio-Temporally Localizing Concepts in Video Diffusion Transformerscited by 1
  13. 2026FlowMotion: Training-Free Flow Guidance for Video Motion Transfercited by 1
  14. 2026Fast 4D Mesh Generation by Spatio-Temporal Attention Chainscited by 1
  15. 2026MotionAdapter: Video Motion Transfer via Content-Aware Attention Customizationcited by 0
  16. 2026Video2LoRA: Unified Semantic-Controlled Video Generation via Per-Reference-Video LoRAcited by 0
  17. 2026MotionGrounder: Grounded Multi-Object Motion Transfer via Diffusion Transformercited by 0
  18. 2026ActCam: Zero-Shot Joint Camera and 3D Motion Control for Video Generationcited by 0
  19. 2026TeDiO: Temporal Diagonal Optimization for Training-Free Coherent Video Diffusioncited by 0
  20. 2026Sound Sparks Motion: Audio and Text Tuning for Video Editingcited by 0
  21. 2026Phi-Noise: Training-Free Temporal Video Conditioning via Phase-Based Noise Manipulationcited by 0
  22. 2026RoPEMover: Depth-Aware Object Relocation via Positional Embeddingscited by 0
  23. 2026QWERTY: Training-Free Motion Control via Query-Warped Video Diffusion Transformerscited by 0
  24. 2026ProxyUp: Training-Free Proxy-Conditioned Video Generation for Controllable Dynamicscited by 0
  25. 2026Controlling Motion Transfer in Diffusion Transformers via Attention Headscited by 0
  26. 2026Motion Beyond Morphology: Bootstrapping Cross-Category Motion Transfer from Abstract Motion Representationscited by 0
  27. 2026ReaDiT Guidance: Control for Image and Video Generation using Diffusion Transformer Featurescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MultiMotion: Multi Subject Video Motion Transfer via Video Diffusion Transformer
  2. 2025Emergent Temporal Correspondences from Video Diffusion Transformers
  3. 2024MotionFlow: Attention-Driven Motion Transfer in Video Diffusion Models
  4. 2025EDEN: Enhanced Diffusion for High-quality Large-motion Video Frame Interpolation
  5. 2026Not All Tokens Need 40 Steps: Heterogeneous Step Allocation in Diffusion Transformers for Efficient Video Generation
  6. 2026FrameDiT: Diffusion Transformer with Matrix Attention for Efficient Video Generation
  7. 2026AcrossVAM2.0: Conflict-Aware Point Supervision for Parameter-Efficient Video Diffusion Adaptation
  8. 2025VGDFR: Diffusion-based Video Generation with Dynamic Latent Frame Rate