# Click to Move: Controlling Video Generation with Sparse Motion

arXiv: https://arxiv.org/abs/2108.08815 · HTML: https://mlsanity.com/paper/2108.08815/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 53 references that are in the map.
- First posted on arXiv 19 Aug 2021.

## Builds on (most cited first, 30 of 40)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [YOLOv3: An Incremental Improvement](https://mlsanity.com/paper/1804.02767/) (2018)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [StarGAN v2: Diverse Image Synthesis for Multiple Domains](https://mlsanity.com/paper/1912.01865/) (2019)
- [Least Squares Generative Adversarial Networks](https://mlsanity.com/paper/1611.04076/) (2016)
- [FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks](https://mlsanity.com/paper/1612.01925/) (2016)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [Interpreting the Latent Space of GANs for Semantic Face Editing](https://mlsanity.com/paper/1907.10786/) (2019)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [Panoptic-DeepLab](https://mlsanity.com/paper/1910.04751/) (2019)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [MakeItTalk: Speaker-Aware Talking-Head Animation](https://mlsanity.com/paper/2004.12992/) (2020)
- [Deep multi-scale video prediction beyond mean square error](https://mlsanity.com/paper/1511.05440/) (2015)
- [Unsupervised Learning for Physical Interaction through Video Prediction](https://mlsanity.com/paper/1605.07157/) (2016)
- [Hierarchical Cross-Modal Talking Face Generationwith Dynamic Pixel-Wise Loss](https://mlsanity.com/paper/1905.03820/) (2019)
- [Animating Arbitrary Objects via Deep Motion Transfer](https://mlsanity.com/paper/1812.08861/) (2018)
- [X2Face: A network for controlling face generation by using images, audio, and pose codes](https://mlsanity.com/paper/1807.10550/) (2018)
- [Stochastic Adversarial Video Prediction](https://mlsanity.com/paper/1804.01523/) (2018)
- [EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning](https://mlsanity.com/paper/1901.00212/) (2019)
- [Video Generation From Text](https://mlsanity.com/paper/1710.00421/) (2017)
- [Stochastic Latent Residual Video Prediction](https://mlsanity.com/paper/2002.09219/) (2020)
- [Learning to Generate Long-term Future via Hierarchical Prediction](https://mlsanity.com/paper/1704.05831/) (2017)
- [VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation](https://mlsanity.com/paper/1903.01434/) (2019)
- [Dual Motion GAN for Future-Flow Embedded Video Prediction](https://mlsanity.com/paper/1708.00284/) (2017)
- [Future Video Synthesis with Object Motion Prediction](https://mlsanity.com/paper/2004.00542/) (2020)
- [Flow-Grounded Spatial-Temporal Video Prediction from Still Images](https://mlsanity.com/paper/1807.09755/) (2018)

## Built on by (most cited first, 11 of 11)

- [DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory](https://mlsanity.com/paper/2308.08089/) (2023)
- [Motion Prompting: Controlling Video Generation with Motion Trajectories](https://mlsanity.com/paper/2412.02700/) (2024)
- [DragAnything: Motion Control for Anything using Entity Representation](https://mlsanity.com/paper/2403.07420/) (2024)
- [Motion-Conditioned Diffusion Model for Controllable Video Synthesis](https://mlsanity.com/paper/2304.14404/) (2023)
- [LeviTor: 3D Trajectory Oriented Image-to-Video Synthesis](https://mlsanity.com/paper/2412.15214/) (2024)
- [DragEntity: Trajectory Guided Video Generation using Entity and Positional Relationships](https://mlsanity.com/paper/2410.10751/) (2024)
- [Consistent Video Editing as Flow-Driven Image-to-Video Generation](https://mlsanity.com/paper/2506.07713/) (2025)
- [Planning with Sketch-Guided Verification for Physics-Aware Video Generation](https://mlsanity.com/paper/2511.17450/) (2025)
- [Point Prompting: Counterfactual Tracking with Video Diffusion Models](https://mlsanity.com/paper/2510.11715/) (2025)
- [Motion Marionette: Rethinking Rigid Motion Transfer via Prior Guidance](https://mlsanity.com/paper/2511.19909/) (2025)
- [Inference-based GAN Video Generation](https://mlsanity.com/paper/2512.21776/) (2025)

## Abstract (arXiv)

This paper introduces Click to Move (C2M), a novel framework for video generation where the user can control the motion of the synthesized video through mouse clicks specifying simple object trajectories of the key objects in the scene. Our model receives as input an initial frame, its corresponding segmentation map and the sparse motion vectors encoding the input provided by the user. It outputs a plausible video sequence starting from the given frame and with a motion that is consistent with user input. Notably, our proposed deep architecture incorporates a Graph Convolution Network (GCN) modelling the movements of all the objects in the scene in a holistic manner and effectively combining the sparse user motion information and image features. Experimental results show that C2M outperforms existing methods on two publicly available datasets, thus demonstrating the effectiveness of our GCN framework at modelling object interactions. The source code is publicly available at https://github.com/PierfrancescoArdino/C2M.