# MoTrans: Customized Motion Transfer with Text-driven Video Diffusion Models

arXiv: https://arxiv.org/abs/2412.01343 · HTML: https://mlsanity.com/paper/2412.01343/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 31 of its 51 references that are in the map.
- First posted on arXiv 2 Dec 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 31)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023)
- [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models](https://mlsanity.com/paper/2401.09047/) (2024)
- [DensePose: Dense Human Pose Estimation In The Wild](https://mlsanity.com/paper/1802.00434/) (2018)
- [Paint by Example: Exemplar-based Image Editing with Diffusion Models](https://mlsanity.com/paper/2211.13227/) (2022)
- [MagicVideo: Efficient Video Generation With Latent Diffusion Models](https://mlsanity.com/paper/2211.11018/) (2022)
- [AnyDoor: Zero-shot Object-level Image Customization](https://mlsanity.com/paper/2307.09481/) (2023)
- [Photorealistic Video Generation with Diffusion Models](https://mlsanity.com/paper/2312.06662/) (2023)
- [Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning](https://mlsanity.com/paper/2311.10709/) (2023)
- [Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs](https://mlsanity.com/paper/2401.11708/) (2024)
- [Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Models](https://mlsanity.com/paper/2302.12228/) (2023)
- [MotionDirector: Motion Customization of Text-to-Video Diffusion Models](https://mlsanity.com/paper/2310.08465/) (2023)
- [Make Pixels Dance: High-Dynamic Video Generation](https://mlsanity.com/paper/2311.10982/) (2023)
- [Cones 2: Customizable Image Synthesis with Multiple Subjects](https://mlsanity.com/paper/2305.19327/) (2023)
- [Guided Image Synthesis via Initial Image Editing in Diffusion Model](https://mlsanity.com/paper/2305.03382/) (2023)
- [Uni-paint: A Unified Framework for Multimodal Image Inpainting with Pretrained Diffusion Model](https://mlsanity.com/paper/2310.07222/) (2023)
- [LAMP: Learn A Motion Pattern for Few-Shot-Based Video Generation](https://mlsanity.com/paper/2310.10769/) (2023)
- [StableIdentity: Inserting Anybody into Anywhere at First Sight](https://mlsanity.com/paper/2401.15975/) (2024)

## Built on by (most cited first, 11 of 11)

- [Personalized Image Generation with Deep Generative Models: A Decade Survey](https://mlsanity.com/paper/2502.13081/) (2025)
- [ReNeg: Learning Negative Embedding with Reward Guidance](https://mlsanity.com/paper/2412.19637/) (2024)
- [Evolution of Video Generative Foundations](https://mlsanity.com/paper/2604.06339/) (2026)
- [JointTuner: Appearance-Motion Adaptive Joint Training for Customized Video Generation](https://mlsanity.com/paper/2503.23951/) (2025)
- [AIComposer: Any Style and Content Image Composition via Feature Integration](https://mlsanity.com/paper/2507.20721/) (2025)
- [DreamVideo-Omni: Omni-Motion Controlled Multi-Subject Video Customization with Latent Identity Reinforcement Learning](https://mlsanity.com/paper/2603.12257/) (2026)
- [Learning to Animate Images from A Few Videos to Portray Delicate Human Actions](https://mlsanity.com/paper/2503.00276/) (2025)
- [Moaw: Unleashing Motion Awareness for Video Diffusion Models](https://mlsanity.com/paper/2601.12761/) (2026)
- [IM-Animation: An Implicit Motion Representation for Identity-decoupled Character Animation](https://mlsanity.com/paper/2602.07498/) (2026)
- [Crafting Your Evolving Dreams: Concept-Incremental Versatile Customization](https://mlsanity.com/paper/2606.04797/) (2026)
- [Inverting the Streaming-Diffusion Bottleneck: Video-Rate MLLM-Conditioned Edit Diffusion on a Consumer GPU](https://mlsanity.com/paper/2606.05981/) (2026)

## Abstract (arXiv)

Existing pretrained text-to-video (T2V) models have demonstrated impressive abilities in generating realistic videos with basic motion or camera movement. However, these models exhibit significant limitations when generating intricate, human-centric motions. Current efforts primarily focus on fine-tuning models on a small set of videos containing a specific motion. They often fail to effectively decouple motion and the appearance in the limited reference videos, thereby weakening the modeling capability of motion patterns. To this end, we propose MoTrans, a customized motion transfer method enabling video generation of similar motion in new context. Specifically, we introduce a multimodal large language model (MLLM)-based recaptioner to expand the initial prompt to focus more on appearance and an appearance injection module to adapt appearance prior from video frames to the motion modeling process. These complementary multimodal representations from recaptioned prompt and video frames promote the modeling of appearance and facilitate the decoupling of appearance and motion. In addition, we devise a motion-specific embedding for further enhancing the modeling of the specific motion. Experimental results demonstrate that our method effectively learns specific motion pattern from singular or multiple reference videos, performing favorably against existing methods in customized video generation.