# Motion-Zero: Zero-Shot Moving Object Control Framework for Diffusion-Based Video Generation

arXiv: https://arxiv.org/abs/2401.10150 · HTML: https://mlsanity.com/paper/2401.10150/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 25 of its 28 references that are in the map.
- First posted on arXiv 18 Jan 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 25 of 25)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Imagen Video: High Definition Video Generation with Diffusion Models](https://mlsanity.com/paper/2210.02303/) (2022)
- [Make-A-Video: Text-to-Video Generation without Text-Video Data](https://mlsanity.com/paper/2209.14792/) (2022)
- [AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning](https://mlsanity.com/paper/2307.04725/) (2023)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation](https://mlsanity.com/paper/2305.01569/) (2023)
- [Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2212.11565/) (2022)
- [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023)
- [MotionCtrl: A Unified and Flexible Motion Controller for Video Generation](https://mlsanity.com/paper/2312.03641/) (2023)
- [VideoCrafter1: Open Diffusion Models for High-Quality Video Generation](https://mlsanity.com/paper/2310.19512/) (2023)
- [MagicVideo: Efficient Video Generation With Latent Diffusion Models](https://mlsanity.com/paper/2211.11018/) (2022)
- [VideoComposer: Compositional Video Synthesis with Motion Controllability](https://mlsanity.com/paper/2306.02018/) (2023)
- [Diffusion Self-Guidance for Controllable Image Generation](https://mlsanity.com/paper/2306.00986/) (2023)
- [Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generation](https://mlsanity.com/paper/2309.15818/) (2023)
- [BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion](https://mlsanity.com/paper/2307.10816/) (2023)
- [DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory](https://mlsanity.com/paper/2308.08089/) (2023)
- [Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions](https://mlsanity.com/paper/2111.10337/) (2021)
- [TrailBlazer: Trajectory Control for Diffusion-Based Video Generation](https://mlsanity.com/paper/2401.00896/) (2023)

## Built on by (most cited first, 25 of 25)

- [Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling](https://mlsanity.com/paper/2401.15977/) (2024)
- [CamI2V: Camera-Controlled Image-to-Video Diffusion Model](https://mlsanity.com/paper/2410.15957/) (2024)
- [MotionBooth: Motion-Aware Customized Text-to-Video Generation](https://mlsanity.com/paper/2406.17758/) (2024)
- [4Real: Towards Photorealistic 4D Scene Generation via Video Diffusion Models](https://mlsanity.com/paper/2406.07472/) (2024)
- [RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Control](https://mlsanity.com/paper/2502.10059/) (2025)
- [DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise Motion Control](https://mlsanity.com/paper/2410.13830/) (2024)
- [Free-Form Motion Control: Controlling the 6D Poses of Camera and Objects in Video Generation](https://mlsanity.com/paper/2501.01425/) (2025)
- [EasyControl: Transfer ControlNet to Video Diffusion for Controllable Generation and Interpolation](https://mlsanity.com/paper/2408.13005/) (2024)
- [Animate Your Motion: Turning Still Images into Dynamic Videos](https://mlsanity.com/paper/2403.10179/) (2024)
- [AniCrafter: Customizing Realistic Human-Centric Animation via Avatar-Background Conditioning in Video Diffusion Models](https://mlsanity.com/paper/2505.20255/) (2025)
- [SmartSight: Mitigating Hallucination in Video-LLMs Without Compromising Video Understanding via Temporal Attention Collapse](https://mlsanity.com/paper/2512.18671/) (2025)
- [CinePreGen: Camera Controllable Video Previsualization via Engine-powered Diffusion](https://mlsanity.com/paper/2408.17424/) (2024)
- [3D Object Manipulation in a Single Image using Generative Models](https://mlsanity.com/paper/2501.12935/) (2025)
- [LiON-LoRA: Rethinking LoRA Fusion to Unify Controllable Spatial and Temporal Generation for Video Diffusion](https://mlsanity.com/paper/2507.05678/) (2025)
- [Evolution of Video Generative Foundations](https://mlsanity.com/paper/2604.06339/) (2026)
- [Reenact Anything: Semantic Video Motion Transfer Using Motion-Textual Inversion](https://mlsanity.com/paper/2408.00458/) (2024)
- [DiTraj: training-free trajectory control for video diffusion transformer](https://mlsanity.com/paper/2509.21839/) (2025)
- [DreamVideo-Omni: Omni-Motion Controlled Multi-Subject Video Customization with Latent Identity Reinforcement Learning](https://mlsanity.com/paper/2603.12257/) (2026)
- [StarVid: Enhancing Semantic Alignment in Video Diffusion Models via Spatial and SynTactic Guided Attention Refocusing](https://mlsanity.com/paper/2409.15259/) (2024)
- [What Are You Doing? A Closer Look at Controllable Human Video Generation](https://mlsanity.com/paper/2503.04666/) (2025)
- [LMP: Leveraging Motion Prior in Zero-Shot Video Generation with Diffusion Transformer](https://mlsanity.com/paper/2505.14167/) (2025)
- [Interactive Video Generation via Domain Adaptation](https://mlsanity.com/paper/2505.24253/) (2025)
- [When to Lock Attention: Training-Free KV Control in Video Diffusion](https://mlsanity.com/paper/2603.09657/) (2026)
- [Making Video Models Adhere to User Intent with Minor Adjustments](https://mlsanity.com/paper/2603.19672/) (2026)
- [TrajLoc: Trajectory-Attention Localization for Multi-Object Motion Control](https://mlsanity.com/paper/2607.00861/) (2026)

## Abstract (arXiv)

Recent large-scale pre-trained diffusion models have demonstrated a powerful generative ability to produce high-quality videos from detailed text descriptions. However, exerting control over the motion of objects in videos generated by any video diffusion model is a challenging problem. In this paper, we propose a novel zero-shot moving object trajectory control framework, Motion-Zero, to enable a bounding-box-trajectories-controlled text-to-video diffusion model. To this end, an initial noise prior module is designed to provide a position-based prior to improve the stability of the appearance of the moving object and the accuracy of position. In addition, based on the attention map of the U-net, spatial constraints are directly applied to the denoising process of diffusion models, which further ensures the positional and spatial consistency of moving objects during the inference. Furthermore, temporal consistency is guaranteed with a proposed shift temporal attention mechanism. Our method can be flexibly applied to various state-of-the-art video diffusion models without any training process. Extensive experiments demonstrate our proposed method can control the motion trajectories of objects and generate high-quality videos. Our project page is https://vpx-ecnu.github.io/MotionZero-website/