# Motion Modes: What Could Happen Next?

arXiv: https://arxiv.org/abs/2412.00148 · HTML: https://mlsanity.com/paper/2412.00148/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 36 references that are in the map.
- First posted on arXiv 29 Nov 2024.

## Builds on (most cited first, 30 of 30)

- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Movie Gen: A Cast of Media Foundation Models](https://mlsanity.com/paper/2410.13720/) (2024)
- [GANSpace: Discovering Interpretable GAN Controls](https://mlsanity.com/paper/2004.02546/) (2020)
- [VideoComposer: Compositional Video Synthesis with Motion Controllability](https://mlsanity.com/paper/2306.02018/) (2023)
- [Diffusion Self-Guidance for Controllable Image Generation](https://mlsanity.com/paper/2306.00986/) (2023)
- [DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing](https://mlsanity.com/paper/2306.14435/) (2023)
- [Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold](https://mlsanity.com/paper/2305.10973/) (2023)
- [Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling](https://mlsanity.com/paper/2401.15977/) (2024)
- [DragonDiffusion: Enabling Drag-style Manipulation on Diffusion Models](https://mlsanity.com/paper/2307.02421/) (2023)
- [DragAnything: Motion Control for Anything using Entity Representation](https://mlsanity.com/paper/2403.07420/) (2024)
- [Boximator: Generating Rich and Controllable Motions for Video Synthesis](https://mlsanity.com/paper/2402.01566/) (2024)
- [Motion-Conditioned Diffusion Model for Controllable Video Synthesis](https://mlsanity.com/paper/2304.14404/) (2023)
- [Generative Image Dynamics](https://mlsanity.com/paper/2309.07906/) (2023)
- [MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Model](https://mlsanity.com/paper/2405.20222/) (2024)
- [PEEKABOO: Interactive Video Generation via Masked-Diffusion](https://mlsanity.com/paper/2312.07509/) (2023)
- [Particle Guidance: non-I.I.D. Diverse Sampling with Diffusion Models](https://mlsanity.com/paper/2310.13102/) (2023)
- [AnimateAnything: Fine-Grained Open Domain Image Animation with Motion Guidance](https://mlsanity.com/paper/2311.12886/) (2023)
- [Video Diffusion Models are Training-free Motion Interpreter and Controller](https://mlsanity.com/paper/2405.14864/) (2024)
- [Diffusion Handles: Enabling 3D Edits for Diffusion Models by Lifting Activations to 3D](https://mlsanity.com/paper/2312.02190/) (2023)
- [iPOKE: Poking a Still Image for Controlled Stochastic Video Synthesis](https://mlsanity.com/paper/2107.02790/) (2021)
- [TrackGo: A Flexible and Efficient Method for Controllable Video Generation](https://mlsanity.com/paper/2408.11475/) (2024)
- [AnimateAnything: Consistent and Controllable Animation for Video Generation](https://mlsanity.com/paper/2411.10836/) (2024)
- [DragAPart: Learning a Part-Level Motion Prior for Articulated Objects](https://mlsanity.com/paper/2403.15382/) (2024)
- [Puppet-Master: Scaling Interactive Video Generation as a Motion Prior for Part-Level Dynamics](https://mlsanity.com/paper/2408.04631/) (2024)
- [DiffUHaul: A Training-Free Method for Object Dragging in Images](https://mlsanity.com/paper/2406.01594/) (2024)
- [LightningDrag: Lightning Fast and Accurate Drag-based Image Editing Emerging from Videos](https://mlsanity.com/paper/2405.13722/) (2024)
- [Blowing in the Wind: CycleNet for Human Cinemagraphs from Still Images](https://mlsanity.com/paper/2303.08639/) (2023)
- [Learn the Force We Can: Enabling Sparse Motion Control in Multi-Object Video Generation](https://mlsanity.com/paper/2306.03988/) (2023)

## Built on by (most cited first, 12 of 12)

- [NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamics](https://mlsanity.com/paper/2509.21309/) (2025)
- [Exploring the Evolution of Physics Cognition in Video Generation: A Survey](https://mlsanity.com/paper/2503.21765/) (2025)
- [VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control](https://mlsanity.com/paper/2601.05138/) (2026)
- [VidCRAFT3: Camera, Object, and Lighting Control for Image-to-Video Generation](https://mlsanity.com/paper/2502.07531/) (2025)
- [What Happens Next? Anticipating Future Motion by Generating Point Trajectories](https://mlsanity.com/paper/2509.21592/) (2025)
- [PhysAlign: Physics-Coherent Image-to-Video Generation through Feature and 3D Representation Alignment](https://mlsanity.com/paper/2603.13770/) (2026)
- [ViPS: Video-informed Pose Spaces for Auto-Rigged Meshes](https://mlsanity.com/paper/2604.17623/) (2026)
- [NEXUS: Neural Energy Fields for Physically Consistent Contact-Rich 3D Object Dynamics](https://mlsanity.com/paper/2606.15015/) (2026)
- [Wildfire Spread Scenarios: Increasing Sample Diversity of Segmentation Diffusion Models with Training-Free Methods](https://mlsanity.com/paper/2603.20188/) (2026)
- [DynaVid: Learning to Generate Highly Dynamic Videos using Synthetic Motion Data](https://mlsanity.com/paper/2604.01666/) (2026)
- [FlexComposer: Unified Video Compositing from Images to Dynamic Footage with Flexible Trajectory Control](https://mlsanity.com/paper/2607.29627/) (2026)
- [JtT: Rethinking Point Trajectory Diffusion for Diverse and Rigid Motion from an Image](https://mlsanity.com/paper/2610.16591/) (2026)

## Abstract (arXiv)

Predicting diverse object motions from a single static image remains challenging, as current video generation models often entangle object movement with camera motion and other scene changes. While recent methods can predict specific motions from motion arrow input, they rely on synthetic data and predefined motions, limiting their application to complex scenes. We introduce Motion Modes, a training-free approach that explores a pre-trained image-to-video generator's latent distribution to discover various distinct and plausible motions focused on selected objects in static images. We achieve this by employing a flow generator guided by energy functions designed to disentangle object and camera motion. Additionally, we use an energy inspired by particle guidance to diversify the generated motions, without requiring explicit training data. Experimental results demonstrate that Motion Modes generates realistic and varied object animations, surpassing previous methods and even human predictions regarding plausibility and diversity. Project Webpage: https://motionmodes.github.io/