# Separate Motion from Appearance: Customizing Motion via Customizing Text-to-Video Diffusion Models arXiv: https://arxiv.org/abs/2501.16714 ยท HTML: https://mlsanity.com/paper/2501.16714/ ## Key facts - Cited by 4 papers in the map (data through 14 Oct 2026). - Placed in Video generation, one of 14 directions in Diffusion models. - Builds on 25 of its 45 references that are in the map. - First posted on arXiv 28 Jan 2025. ## Placed in - Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/) ## Builds on (most cited first, 25 of 25) - [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023) - [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022) - [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023) - [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022) - [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022) - [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023) - [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023) - [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022) - [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023) - [VideoGPT: Video Generation using VQ-VAE and Transformers](https://mlsanity.com/paper/2104.10157/) (2021) - [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023) - [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023) - [MagicVideo: Efficient Video Generation With Latent Diffusion Models](https://mlsanity.com/paper/2211.11018/) (2022) - [I2VGen-XL: High-Quality Image-to-Video Synthesis via Cascaded Diffusion Models](https://mlsanity.com/paper/2311.04145/) (2023) - [ControlVideo: Training-free Controllable Text-to-Video Generation](https://mlsanity.com/paper/2305.13077/) (2023) - [Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation](https://mlsanity.com/paper/2306.07954/) (2023) - [Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer](https://mlsanity.com/paper/2204.03638/) (2022) - [FreeU: Free Lunch in Diffusion U-Net](https://mlsanity.com/paper/2309.11497/) (2023) - [DreamVideo: Composing Your Dream Videos with Customized Subject and Motion](https://mlsanity.com/paper/2312.04433/) (2023) - [A Good Image Generator Is What You Need for High-Resolution Video Synthesis](https://mlsanity.com/paper/2104.15069/) (2021) - [Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generation](https://mlsanity.com/paper/2304.08477/) (2023) - [Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animator](https://mlsanity.com/paper/2309.14494/) (2023) - [CCVS: Context-aware Controllable Video Synthesis](https://mlsanity.com/paper/2107.08037/) (2021) - [CVPR 2023 Text Guided Video Editing Competition](https://mlsanity.com/paper/2310.16003/) (2023) - [MoStGAN-V: Video Generation with Temporal Motion Styles](https://mlsanity.com/paper/2304.02777/) (2023) ## Built on by (most cited first, 4 of 4) - [Group Relative Attention Guidance for Image Editing](https://mlsanity.com/paper/2510.24657/) (2025) - [GenHSI: Controllable Generation of Human-Scene Interaction Videos](https://mlsanity.com/paper/2506.19840/) (2025) - [JointTuner: Appearance-Motion Adaptive Joint Training for Customized Video Generation](https://mlsanity.com/paper/2503.23951/) (2025) - [A Style is Worth One Code: Unlocking Code-to-Style Image Generation with Discrete Style Space](https://mlsanity.com/paper/2511.10555/) (2025) ## Abstract (arXiv) Motion customization aims to adapt the diffusion model (DM) to generate videos with the motion specified by a set of video clips with the same motion concept. To realize this goal, the adaptation of DM should be possible to model the specified motion concept, without compromising the ability to generate diverse appearances. Thus, the key to solving this problem lies in how to separate the motion concept from the appearance in the adaptation process of DM. Typical previous works explore different ways to represent and insert a motion concept into large-scale pretrained text-to-video diffusion models, e.g., learning a motion LoRA, using latent noise residuals, etc. While those methods can encode the motion concept, they also inevitably encode the appearance in the reference videos, resulting in weakened appearance generation capability. In this paper, we follow the typical way to learn a motion LoRA to encode the motion concept, but propose two novel strategies to enhance motion-appearance separation, including temporal attention purification (TAP) and appearance highway (AH). Specifically, we assume that in the temporal attention module, the pretrained Value embeddings are sufficient to serve as basic components needed by producing a new motion. Thus, in TAP, we choose only to reshape the temporal attention with motion LoRAs so that Value embeddings can be reorganized to produce a new motion. Further, in AH, we alter the starting point of each skip connection in U-Net from the output of each temporal attention module to the output of each spatial attention module. Extensive experiments demonstrate that compared to previous works, our method can generate videos with appearance more aligned with the text descriptions and motion more consistent with the reference videos.