# ATI: Any Trajectory Instruction for Controllable Video Generation

arXiv: https://arxiv.org/abs/2505.22944 · HTML: https://mlsanity.com/paper/2505.22944/

## Key facts

- Cited by 43 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 42 references that are in the map.
- First posted on arXiv 28 May 2025.

## Builds on (most cited first, 30 of 41)

- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [HunyuanVideo: A Systematic Framework For Large Video Generative Models](https://mlsanity.com/paper/2412.03603/) (2024)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers](https://mlsanity.com/paper/2205.15868/) (2022)
- [Open-Sora: Democratizing Efficient Video Production for All](https://mlsanity.com/paper/2412.20404/) (2024)
- [MotionCtrl: A Unified and Flexible Motion Controller for Video Generation](https://mlsanity.com/paper/2312.03641/) (2023)
- [VideoCrafter2: Overcoming Data Limitations for High-Quality Video Diffusion Models](https://mlsanity.com/paper/2401.09047/) (2024)
- [CameraCtrl: Enabling Camera Control for Text-to-Video Generation](https://mlsanity.com/paper/2404.02101/) (2024)
- [Lumiere: A Space-Time Diffusion Model for Video Generation](https://mlsanity.com/paper/2401.12945/) (2024)
- [ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis](https://mlsanity.com/paper/2409.02048/) (2024)
- [Open-Sora Plan: Open-Source Large Video Generation Model](https://mlsanity.com/paper/2412.00131/) (2024)
- [DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory](https://mlsanity.com/paper/2308.08089/) (2023)
- [TAP-Vid: A Benchmark for Tracking Any Point in a Video](https://mlsanity.com/paper/2211.03726/) (2022)
- [Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling](https://mlsanity.com/paper/2401.15977/) (2024)
- [Motion Prompting: Controlling Video Generation with Motion Trajectories](https://mlsanity.com/paper/2412.02700/) (2024)
- [MotionDirector: Motion Customization of Text-to-Video Diffusion Models](https://mlsanity.com/paper/2310.08465/) (2023)
- [Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model](https://mlsanity.com/paper/2502.10248/) (2025)
- [Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motion](https://mlsanity.com/paper/2402.03162/) (2024)
- [CamCo: Camera-Controllable 3D-Consistent Image-to-Video Generation](https://mlsanity.com/paper/2406.02509/) (2024)
- [DragAnything: Motion Control for Anything using Entity Representation](https://mlsanity.com/paper/2403.07420/) (2024)
- [Training-free Camera Control for Video Generation](https://mlsanity.com/paper/2406.10126/) (2024)
- [CamI2V: Camera-Controlled Image-to-Video Diffusion Model](https://mlsanity.com/paper/2410.15957/) (2024)
- [Collaborative Video Diffusion: Consistent Multi-video Generation with Camera Control](https://mlsanity.com/paper/2405.17414/) (2024)
- [Boximator: Generating Rich and Controllable Motions for Video Synthesis](https://mlsanity.com/paper/2402.01566/) (2024)
- [MotionBooth: Motion-Aware Customized Text-to-Video Generation](https://mlsanity.com/paper/2406.17758/) (2024)
- [TrailBlazer: Trajectory Control for Diffusion-Based Video Generation](https://mlsanity.com/paper/2401.00896/) (2023)
- [MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Model](https://mlsanity.com/paper/2405.20222/) (2024)

## Built on by (most cited first, 30 of 43)

- [MotionStream: Real-Time Video Generation with Interactive Motion Controls](https://mlsanity.com/paper/2511.01266/) (2025)
- [LoL: Longer than Longer, Scaling Video Generation to Hour](https://mlsanity.com/paper/2601.16914/) (2026)
- [Generative Video Motion Editing with 3D Point Tracks](https://mlsanity.com/paper/2512.02015/) (2025)
- [WorldStereo: Bridging Camera-Guided Video Generation and Scene Reconstruction via 3D Geometric Memories](https://mlsanity.com/paper/2603.02049/) (2026)
- [MotionV2V: Editing Motion in a Video](https://mlsanity.com/paper/2511.20640/) (2025)
- [Time-to-Move: Training-Free Motion Controlled Video Generation via Dual-Clock Denoising](https://mlsanity.com/paper/2511.08633/) (2025)
- [SymphoMotion: Joint Control of Camera Motion and Object Dynamics for Coherent Video Generation](https://mlsanity.com/paper/2604.03723/) (2026)
- [The World is Your Canvas: Painting Promptable Events with Reference Images, Trajectories, and Text](https://mlsanity.com/paper/2512.16924/) (2025)
- [CausalSpatial: A Benchmark for Object-Centric Causal Spatial Reasoning](https://mlsanity.com/paper/2601.13304/) (2026)
- [Planning with Sketch-Guided Verification for Physics-Aware Video Generation](https://mlsanity.com/paper/2511.17450/) (2025)
- [Objects in Generated Videos Are Slower Than They Appear: Models Suffer Sub-Earth Gravity and Don't Know Galileo's Principle...for now](https://mlsanity.com/paper/2512.02016/) (2025)
- [PoseAnything: Universal Pose-guided Video Generation with Part-aware Temporal Coherence](https://mlsanity.com/paper/2512.13465/) (2025)
- [AVControl: Efficient Framework for Training Audio-Visual Controls](https://mlsanity.com/paper/2603.24793/) (2026)
- [PhyCo: Learning Controllable Physical Priors for Generative Motion](https://mlsanity.com/paper/2604.28169/) (2026)
- [VHOI: Controllable Video Generation of Human-Object Interactions from Sparse Trajectories via Motion Densification](https://mlsanity.com/paper/2512.09646/) (2025)
- [Controllable Egocentric Video Generation via Occlusion-Aware Sparse 3D Hand Joints](https://mlsanity.com/paper/2603.11755/) (2026)
- [TGT: Text-Grounded Trajectories for Locally Controlled Video Generation](https://mlsanity.com/paper/2510.15104/) (2025)
- [MoRight: Motion Control Done Right](https://mlsanity.com/paper/2604.07348/) (2026)
- [OptiWorld: Optimal Control for Video World Generation under Physical Constraints](https://mlsanity.com/paper/2606.00499/) (2026)
- [CineLOG: A Training Free Approach for Cinematic Long Video Generation](https://mlsanity.com/paper/2512.12209/) (2025)
- [TETO: Tracking Events with Teacher Observation for Motion Estimation and Frame Interpolation](https://mlsanity.com/paper/2603.23487/) (2026)
- [TrajectoryMover: Generative Movement of Object Trajectories in Videos](https://mlsanity.com/paper/2603.29092/) (2026)
- [Seeing Fast and Slow: Learning the Flow of Time in Videos](https://mlsanity.com/paper/2604.21931/) (2026)
- [Preserve, Reveal, Expand: Faithful 4D Video Editing with Region-Aware Conditioning](https://mlsanity.com/paper/2605.20961/) (2026)
- [SVI-Bench: A Dynamic Microworld for Strategic Video Intelligence](https://mlsanity.com/paper/2605.31529/) (2026)
- [Go-with-the-Track: Video Compositing and Motion Control with Point Tracking](https://mlsanity.com/paper/2606.20891/) (2026)
- [EO-WM: A Physically Informed World Model for Probabilistic Earth Observation Forecasting](https://mlsanity.com/paper/2606.27277/) (2026)
- [WorldCast: Distributed Multiplayer World Models](https://mlsanity.com/paper/2610.12412/) (2026)
- [HECTOR: Hybrid Editable Compositional Object References for Video Generation](https://mlsanity.com/paper/2603.08850/) (2026)
- [Eulerian Motion Guidance: Robust Image Animation via Bidirectional Geometric Consistency](https://mlsanity.com/paper/2605.06280/) (2026)

## Abstract (arXiv)

We propose a unified framework for motion control in video generation that seamlessly integrates camera movement, object-level translation, and fine-grained local motion using trajectory-based inputs. In contrast to prior methods that address these motion types through separate modules or task-specific designs, our approach offers a cohesive solution by projecting user-defined trajectories into the latent space of pre-trained image-to-video generation models via a lightweight motion injector. Users can specify keypoints and their motion paths to control localized deformations, entire object motion, virtual camera dynamics, or combinations of these. The injected trajectory signals guide the generative process to produce temporally consistent and semantically aligned motion sequences. Our framework demonstrates superior performance across multiple video motion control tasks, including stylized motion effects (e.g., motion brushes), dynamic viewpoint changes, and precise local motion manipulation. Experiments show that our method provides significantly better controllability and visual quality compared to prior approaches and commercial solutions, while remaining broadly compatible with various state-of-the-art video generation backbones. Project page: https://anytraj.github.io/.