# MotionFollower: Editing Video Motion via Lightweight Score-Guided Diffusion

arXiv: https://arxiv.org/abs/2405.20325 · HTML: https://mlsanity.com/paper/2405.20325/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 46 of its 52 references that are in the map.
- First posted on arXiv 30 May 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 46)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Cascaded Diffusion Models for High Fidelity Image Generation](https://mlsanity.com/paper/2106.15282/) (2021)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023)
- [Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators](https://mlsanity.com/paper/2303.13439/) (2023)
- [Structure and Content-Guided Video Synthesis with Diffusion Models](https://mlsanity.com/paper/2302.03011/) (2023)
- [FateZero: Fusing Attentions for Zero-shot Text-based Video Editing](https://mlsanity.com/paper/2303.09535/) (2023)
- [Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.04803/) (2023)
- [MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model](https://mlsanity.com/paper/2311.16498/) (2023)
- [ControlVideo: Training-free Controllable Text-to-Video Generation](https://mlsanity.com/paper/2305.13077/) (2023)
- [Pix2Video: Video Editing using Image Diffusion](https://mlsanity.com/paper/2303.12688/) (2023)

## Built on by (most cited first, 29 of 29)

- [StableAnimator: High-Quality Identity-Preserving Human Image Animation](https://mlsanity.com/paper/2411.17697/) (2024)
- [EchoMimicV2: Towards Striking, Simplified, and Semi-Body Human Animation](https://mlsanity.com/paper/2411.10061/) (2024)
- [MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidance](https://mlsanity.com/paper/2503.16421/) (2025)
- [StableAvatar: Infinite-Length Audio-Driven Avatar Video Generation](https://mlsanity.com/paper/2508.08248/) (2025)
- [OmniHuman-1.5: Instilling an Active Mind in Avatars via Cognitive Simulation](https://mlsanity.com/paper/2508.19209/) (2025)
- [A Comprehensive Survey on Human Video Generation: Challenges, Methods, and Insights](https://mlsanity.com/paper/2407.08428/) (2024)
- [Human Motion Video Generation: A Survey](https://mlsanity.com/paper/2509.03883/) (2025)
- [DreamRelation: Relation-Centric Video Customization](https://mlsanity.com/paper/2503.07602/) (2025)
- [AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Prediction](https://mlsanity.com/paper/2406.06465/) (2024)
- [EDEN: Enhanced Diffusion for High-quality Large-motion Video Frame Interpolation](https://mlsanity.com/paper/2503.15831/) (2025)
- [DreamDance: Animating Human Images by Enriching 3D Geometry Cues from 2D Poses](https://mlsanity.com/paper/2412.00397/) (2024)
- [REDUCIO! Generating 1K Video within 16 Seconds using Extremely Compressed Motion Latents](https://mlsanity.com/paper/2411.13552/) (2024)
- [StableAnimator++: Overcoming Pose Misalignment and Face Distortion for Human Image Animation](https://mlsanity.com/paper/2507.15064/) (2025)
- [FlashPortrait: 6x Faster Infinite Portrait Animation with Adaptive Latent Prediction](https://mlsanity.com/paper/2512.16900/) (2025)
- [VA-Judger: Reward Modeling from Human Preference Feedback for Joint Video-Audio Generation](https://mlsanity.com/paper/2608.18607/) (2026)
- [Replace Anyone in Videos](https://mlsanity.com/paper/2409.19911/) (2024)
- [AnimateAnyMesh++: A Flexible Feed-Forward Framework for High-Fidelity Text-Driven Mesh Animation](https://mlsanity.com/paper/2604.26917/) (2026)
- [Baton: Explicit Semantic Blueprints for Joint Video-Audio Generation](https://mlsanity.com/paper/2605.25195/) (2026)
- [CFSynthesis: Controllable and Free-view 3D Human Video Synthesis](https://mlsanity.com/paper/2412.11067/) (2024)
- [MotionWeaver: Holistic 4D-Anchored Framework for Multi-Humanoid Image Animation](https://mlsanity.com/paper/2602.13326/) (2026)
- [POPoS: Improving Efficient and Robust Facial Landmark Detection with Parallel Optimal Position Search](https://mlsanity.com/paper/2410.09583/) (2024)
- [Preference Score Distillation: Leveraging 2D Rewards to Align Text-to-3D Generation with Human Preference](https://mlsanity.com/paper/2603.01594/) (2026)
- [SPEED: One-Step Pixel Diffusion for High-quality Video Frame Interpolation](https://mlsanity.com/paper/2607.15585/) (2026)
- [AdaptiveFusion: Adaptive Multi-Modal Multi-View Fusion for 3D Human Body Reconstruction](https://mlsanity.com/paper/2409.04851/) (2024)
- [ExpertEdit: Learning Skill-Aware Motion Editing from Expert Videos](https://mlsanity.com/paper/2604.10466/) (2026)
- [Agentic Visual Generation: From Generative Models to Agentic Control](https://mlsanity.com/paper/2609.06758/) (2026)
- [PersonaAnimator: Personalized Motion Transfer from Unconstrained Videos](https://mlsanity.com/paper/2508.19895/) (2025)
- [TeleMorpher: Toward Robust Simultaneous Motion-Location Editing](https://mlsanity.com/paper/2606.19676/) (2026)
- [3D Scene-Adaptive Trajectory-Controllable Human Image Animation with Camera Movement](https://mlsanity.com/paper/2606.30514/) (2026)

## Abstract (arXiv)

Despite impressive advancements in diffusion-based video editing models in altering video attributes, there has been limited exploration into modifying motion information while preserving the original protagonist's appearance and background. In this paper, we propose MotionFollower, a lightweight score-guided diffusion model for video motion editing. To introduce conditional controls to the denoising process, MotionFollower leverages two of our proposed lightweight signal controllers, one for poses and the other for appearances, both of which consist of convolution blocks without involving heavy attention calculations. Further, we design a score guidance principle based on a two-branch architecture, including the reconstruction and editing branches, which significantly enhance the modeling capability of texture details and complicated backgrounds. Concretely, we enforce several consistency regularizers and losses during the score estimation. The resulting gradients thus inject appropriate guidance to the intermediate latents, forcing the model to preserve the original background details and protagonists' appearances without interfering with the motion modification. Experiments demonstrate the competitive motion editing ability of MotionFollower qualitatively and quantitatively. Compared with MotionEditor, the most advanced motion editing model, MotionFollower achieves an approximately 80% reduction in GPU memory while delivering superior motion editing performance and exclusively supporting large camera movements and actions.