# Video Generation from Single Semantic Label Map

arXiv: https://arxiv.org/abs/1903.04480 · HTML: https://mlsanity.com/paper/1903.04480/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Mar 2019.

## Built on by (most cited first, 30 of 54)

- [Conditional Image-to-Video Generation with Latent Flow Diffusion Models](https://mlsanity.com/paper/2303.13744/) (2023)
- [Latent Image Animator: Learning to Animate Images via Latent Space Navigation](https://mlsanity.com/paper/2203.09043/) (2022)
- [Make It Move: Controllable Image-to-Video Generation with Text Descriptions](https://mlsanity.com/paper/2112.02815/) (2021)
- [CCVS: Context-aware Controllable Video Synthesis](https://mlsanity.com/paper/2107.08037/) (2021)
- [A Dynamic Multi-Scale Voxel Flow Network for Video Prediction](https://mlsanity.com/paper/2303.09875/) (2023)
- [Controllable Animation of Fluid Elements in Still Images](https://mlsanity.com/paper/2112.03051/) (2021)
- [PastNet: Introducing Physical Inductive Biases for Spatio-temporal Video Prediction](https://mlsanity.com/paper/2305.11421/) (2023)
- [Moonshot: Towards Controllable Video Generation and Editing with Multimodal Conditions](https://mlsanity.com/paper/2401.01827/) (2024)
- [LEO: Generative Latent Image Animator for Human Video Synthesis](https://mlsanity.com/paper/2305.03989/) (2023)
- [Generating Representative Samples for Few-Shot Classification](https://mlsanity.com/paper/2205.02918/) (2022)
- [Understanding Object Dynamics for Interactive Image-to-Video Synthesis](https://mlsanity.com/paper/2106.11303/) (2021)
- [Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioning](https://mlsanity.com/paper/2203.02573/) (2022)
- [Free4D: Tuning-free 4D Scene Generation with Spatial-Temporal Consistency](https://mlsanity.com/paper/2503.20785/) (2025)
- [Optimizing Video Prediction via Video Frame Interpolation](https://mlsanity.com/paper/2206.13454/) (2022)
- [Learning Semantic-Aware Dynamics for Video Prediction](https://mlsanity.com/paper/2104.09762/) (2021)
- [Cross-View Exocentric to Egocentric Video Synthesis](https://mlsanity.com/paper/2107.03120/) (2021)
- [Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video Generation](https://mlsanity.com/paper/2501.03059/) (2025)
- [InMoDeGAN: Interpretable Motion Decomposition Generative Adversarial Network for Video Generation](https://mlsanity.com/paper/2101.03049/) (2021)
- [Earthfarseer: Versatile Spatio-Temporal Dynamical Systems Modeling in One Model](https://mlsanity.com/paper/2312.08403/) (2023)
- [Optical-Flow Guided Prompt Optimization for Coherent Video Generation](https://mlsanity.com/paper/2411.15540/) (2024)
- [Simulating Fluids in Real-World Still Images](https://mlsanity.com/paper/2204.11335/) (2022)
- [Blowing in the Wind: CycleNet for Human Cinemagraphs from Still Images](https://mlsanity.com/paper/2303.08639/) (2023)
- [Dance Any Beat: Blending Beats with Visuals in Dance Video Generation](https://mlsanity.com/paper/2405.09266/) (2024)
- [MSPred: Video Prediction at Multiple Spatio-Temporal Scales with Hierarchical Recurrent Networks](https://mlsanity.com/paper/2203.09303/) (2022)
- [Towards Smooth Video Composition](https://mlsanity.com/paper/2212.07413/) (2022)
- [Click to Move: Controlling Video Generation with Sparse Motion](https://mlsanity.com/paper/2108.08815/) (2021)
- [VISAGE: Video Synthesis using Action Graphs for Surgery](https://mlsanity.com/paper/2410.17751/) (2024)
- [Accelerating Video Object Segmentation with Compressed Video](https://mlsanity.com/paper/2107.12192/) (2021)
- [Learning Fine-Grained Motion Embedding for Landscape Animation](https://mlsanity.com/paper/2109.02216/) (2021)
- [Layered Controllable Video Generation](https://mlsanity.com/paper/2111.12747/) (2021)

## Abstract (arXiv)

This paper proposes the novel task of video generation conditioned on a SINGLE semantic label map, which provides a good balance between flexibility and quality in the generation process. Different from typical end-to-end approaches, which model both scene content and dynamics in a single step, we propose to decompose this difficult task into two sub-problems. As current image generation methods do better than video generation in terms of detail, we synthesize high quality content by only generating the first frame. Then we animate the scene based on its semantic meaning to obtain the temporally coherent video, giving us excellent results overall. We employ a cVAE for predicting optical flow as a beneficial intermediate step to generate a video sequence conditioned on the initial single frame. A semantic label map is integrated into the flow prediction module to achieve major improvements in the image-to-video generation process. Extensive experiments on the Cityscapes dataset show that our method outperforms all competing methods.