# Deep Video Generation, Prediction and Completion of Human Action Sequences

arXiv: https://arxiv.org/abs/1711.08682 · HTML: https://mlsanity.com/paper/1711.08682/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Nov 2017.

## Built on by (most cited first, 30 of 52)

- [Action-Conditioned 3D Human Motion Synthesis with Transformer VAE](https://mlsanity.com/paper/2104.05670/) (2021)
- [MoMask: Generative Masked Modeling of 3D Human Motions](https://mlsanity.com/paper/2312.00063/) (2023)
- [Towards Diverse and Natural Scene-aware 3D Human Motion Synthesis](https://mlsanity.com/paper/2205.13001/) (2022)
- [Scene-aware Generative Network for Human Motion Synthesis](https://mlsanity.com/paper/2105.14804/) (2021)
- [Continuous 3D Multi-Channel Sign Language Production via Progressive Transformers and Mixture Density Networks](https://mlsanity.com/paper/2103.06982/) (2021)
- [On Evaluation Metrics for Graph Generative Models](https://mlsanity.com/paper/2201.09871/) (2022)
- [MultiAct: Long-Term 3D Human Motion Generation from Multiple Action Labels](https://mlsanity.com/paper/2212.05897/) (2022)
- [Diffusion Motion: Generate Text-Guided 3D Human Motion by Diffusion Model](https://mlsanity.com/paper/2210.12315/) (2022)
- [MoGenTS: Motion Generation based on Spatial-Temporal Joint Modeling](https://mlsanity.com/paper/2409.17686/) (2024)
- [A Comprehensive Survey on Human Video Generation: Challenges, Methods, and Insights](https://mlsanity.com/paper/2407.08428/) (2024)
- [Generative Adversarial Graph Convolutional Networks for Human Action Synthesis](https://mlsanity.com/paper/2110.11191/) (2021)
- [FLAG3D: A 3D Fitness Activity Dataset with Language Instruction](https://mlsanity.com/paper/2212.04638/) (2022)
- [Learning Semantic-Aware Dynamics for Video Prediction](https://mlsanity.com/paper/2104.09762/) (2021)
- [AVD2: Accident Video Diffusion for Accident Video Description](https://mlsanity.com/paper/2502.14801/) (2025)
- [Prior Image-Constrained Reconstruction using Style-Based Generative Models](https://mlsanity.com/paper/2102.12525/) (2021)
- [Action-conditioned On-demand Motion Generation](https://mlsanity.com/paper/2207.08164/) (2022)
- [Diverse Video Generation using a Gaussian Process Trigger](https://mlsanity.com/paper/2107.04619/) (2021)
- [DiffusionPhase: Motion Diffusion in Frequency Domain](https://mlsanity.com/paper/2312.04036/) (2023)
- [Action2video: Generating Videos of Human 3D Actions](https://mlsanity.com/paper/2111.06925/) (2021)
- [The Image Local Autoregressive Transformer](https://mlsanity.com/paper/2106.02514/) (2021)
- [DanceTogether! Identity-Preserving Multi-Person Interactive Video Generation](https://mlsanity.com/paper/2505.18078/) (2025)
- [Video Prediction by Efficient Transformers](https://mlsanity.com/paper/2212.06026/) (2022)
- [T2LM: Long-Term 3D Human Motion Generation from Multiple Sentences](https://mlsanity.com/paper/2406.00636/) (2024)
- [Conditional Temporal Variational AutoEncoder for Action Video Prediction](https://mlsanity.com/paper/2108.05658/) (2021)
- [A Survey on Future Frame Synthesis: Bridging Deterministic and Generative Approaches](https://mlsanity.com/paper/2401.14718/) (2024)
- [Efficient Text-driven Motion Generation via Latent Consistency Training](https://mlsanity.com/paper/2405.02791/) (2024)
- [BiPO: Bidirectional Partial Occlusion Network for Text-to-Motion Synthesis](https://mlsanity.com/paper/2412.00112/) (2024)
- [Object-centric Video Prediction without Annotation](https://mlsanity.com/paper/2105.02799/) (2021)
- [Skeleton2Humanoid: Animating Simulated Characters for Physically-plausible Motion In-betweening](https://mlsanity.com/paper/2210.04294/) (2022)
- [Spatial-temporal Transformer-guided Diffusion based Data Augmentation for Efficient Skeleton-based Action Recognition](https://mlsanity.com/paper/2302.13434/) (2023)

## Abstract (arXiv)

Current deep learning results on video generation are limited while there are only a few first results on video prediction and no relevant significant results on video completion. This is due to the severe ill-posedness inherent in these three problems. In this paper, we focus on human action videos, and propose a general, two-stage deep framework to generate human action videos with no constraints or arbitrary number of constraints, which uniformly address the three problems: video generation given no input frames, video prediction given the first few frames, and video completion given the first and last frames. To make the problem tractable, in the first stage we train a deep generative model that generates a human pose sequence from random noise. In the second stage, a skeleton-to-image network is trained, which is used to generate a human action video given the complete human pose sequence generated in the first stage. By introducing the two-stage strategy, we sidestep the original ill-posed problems while producing for the first time high-quality video generation/prediction/completion results of much longer duration. We present quantitative and qualitative evaluation to show that our two-stage approach outperforms state-of-the-art methods in video generation, prediction and video completion. Our video result demonstration can be viewed at https://iamacewhite.github.io/supp/index.html