# Unsupervised Keypoint Learning for Guiding Class-Conditional Video Prediction

arXiv: https://arxiv.org/abs/1910.02027 · HTML: https://mlsanity.com/paper/1910.02027/

## Key facts

- Cited by 36 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 4 Oct 2019.

## Built on by (most cited first, 30 of 36)

- [Motion Representations for Articulated Animation](https://mlsanity.com/paper/2104.11280/) (2021)
- [Conditional Image-to-Video Generation with Latent Flow Diffusion Models](https://mlsanity.com/paper/2303.13744/) (2023)
- [Playable Video Generation](https://mlsanity.com/paper/2101.12195/) (2021)
- [Video Prediction Recalling Long-term Motion Context via Memory Alignment Learning](https://mlsanity.com/paper/2104.00924/) (2021)
- [Efficient Video Prediction via Sparsely Conditioned Flow Matching](https://mlsanity.com/paper/2211.14575/) (2022)
- [Moonshot: Towards Controllable Video Generation and Editing with Multimodal Conditions](https://mlsanity.com/paper/2401.01827/) (2024)
- [Unsupervised Learning of Visual 3D Keypoints for Control](https://mlsanity.com/paper/2106.07643/) (2021)
- [Revisiting Hierarchical Approach for Persistent Long-Term Video Prediction](https://mlsanity.com/paper/2104.06697/) (2021)
- [Self-Supervised Keypoint Discovery in Behavioral Videos](https://mlsanity.com/paper/2112.05121/) (2021)
- [GANSeg: Learning to Segment by Unsupervised Hierarchical Image Generation](https://mlsanity.com/paper/2112.01036/) (2021)
- [DDLP: Unsupervised Object-Centric Video Prediction with Deep Dynamic Latent Particles](https://mlsanity.com/paper/2306.05957/) (2023)
- [AutoLink: Self-supervised Learning of Human Skeletons and Object Outlines by Linking Keypoints](https://mlsanity.com/paper/2205.10636/) (2022)
- [MIMO Is All You Need : A Strong Multi-In-Multi-Out Baseline for Video Prediction](https://mlsanity.com/paper/2212.04655/) (2022)
- [Action2video: Generating Videos of Human 3D Actions](https://mlsanity.com/paper/2111.06925/) (2021)
- [Dance Any Beat: Blending Beats with Visuals in Dance Video Generation](https://mlsanity.com/paper/2405.09266/) (2024)
- [Accurate Grid Keypoint Learning for Efficient Video Prediction](https://mlsanity.com/paper/2107.13170/) (2021)
- [Flow Guided Transformable Bottleneck Networks for Motion Retargeting](https://mlsanity.com/paper/2106.07771/) (2021)
- [Future Frame Prediction for Robot-assisted Surgery](https://mlsanity.com/paper/2103.10308/) (2021)
- [Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modeling](https://mlsanity.com/paper/2603.04553/) (2026)
- [Conditional Temporal Variational AutoEncoder for Action Video Prediction](https://mlsanity.com/paper/2108.05658/) (2021)
- [Layered Controllable Video Generation](https://mlsanity.com/paper/2111.12747/) (2021)
- [Towards End-to-End Generative Modeling of Long Videos with Memory-Efficient Bidirectional Transformers](https://mlsanity.com/paper/2303.11251/) (2023)
- [Controllable Video Generation through Global and Local Motion Dynamics](https://mlsanity.com/paper/2204.06558/) (2022)
- [Few-shot Geometry-Aware Keypoint Localization](https://mlsanity.com/paper/2303.17216/) (2023)
- [GaussiGAN: Controllable Image Synthesis with 3D Gaussians from Unposed Silhouettes](https://mlsanity.com/paper/2106.13215/) (2021)
- [Patch-based Object-centric Transformers for Efficient Video Generation](https://mlsanity.com/paper/2206.04003/) (2022)
- [Unsupervised Part Discovery via Dual Representation Alignment](https://mlsanity.com/paper/2408.08108/) (2024)
- [Pose-guided Generative Adversarial Net for Novel View Action Synthesis](https://mlsanity.com/paper/2110.07993/) (2021)
- [Continuous-Time Video Generation via Learning Motion Dynamics with Neural ODE](https://mlsanity.com/paper/2112.10960/) (2021)
- [FlexLip: A Controllable Text-to-Lip System](https://mlsanity.com/paper/2206.03206/) (2022)

## Abstract (arXiv)

We propose a deep video prediction model conditioned on a single image and an action class. To generate future frames, we first detect keypoints of a moving object and predict future motion as a sequence of keypoints. The input image is then translated following the predicted keypoints sequence to compose future frames. Detecting the keypoints is central to our algorithm, and our method is trained to detect the keypoints of arbitrary objects in an unsupervised manner. Moreover, the detected keypoints of the original videos are used as pseudo-labels to learn the motion of objects. Experimental results show that our method is successfully applied to various datasets without the cost of labeling keypoints in videos. The detected keypoints are similar to human-annotated labels, and prediction results are more realistic compared to the previous methods.