# Probabilistic Video Generation using Holistic Attribute Control

arXiv: https://arxiv.org/abs/1803.08085 · HTML: https://mlsanity.com/paper/1803.08085/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Mar 2018.

## Built on by (most cited first, 24 of 24)

- [Latent Video Diffusion Models for High-Fidelity Long Video Generation](https://mlsanity.com/paper/2211.13221/) (2022)
- [Conditional Image-to-Video Generation with Latent Flow Diffusion Models](https://mlsanity.com/paper/2303.13744/) (2023)
- [UniMLVG: Unified Framework for Multi-view Long Video Generation with Comprehensive Control Capabilities for Autonomous Driving](https://mlsanity.com/paper/2412.04842/) (2024)
- [Disentangled Recurrent Wasserstein Autoencoder](https://mlsanity.com/paper/2101.07496/) (2021)
- [Learning Compositional Representation for 4D Captures with Neural ODE](https://mlsanity.com/paper/2103.08271/) (2021)
- [Learning Semantic-Aware Dynamics for Video Prediction](https://mlsanity.com/paper/2104.09762/) (2021)
- [Action2video: Generating Videos of Human 3D Actions](https://mlsanity.com/paper/2111.06925/) (2021)
- [I4VGen: Image as Free Stepping Stone for Text-to-Video Generation](https://mlsanity.com/paper/2406.02230/) (2024)
- [Click to Move: Controlling Video Generation with Sparse Motion](https://mlsanity.com/paper/2108.08815/) (2021)
- [M3L: Language-based Video Editing via Multi-Modal Multi-Level Transformers](https://mlsanity.com/paper/2104.01122/) (2021)
- [Layered Controllable Video Generation](https://mlsanity.com/paper/2111.12747/) (2021)
- [VDSM: Unsupervised Video Disentanglement with State-Space Modeling and Deep Mixtures of Experts](https://mlsanity.com/paper/2103.07292/) (2021)
- [Unsupervised Video Prediction from a Single Frame by Estimating 3D Dynamic Scene Structure](https://mlsanity.com/paper/2106.09051/) (2021)
- [Hamiltonian latent operators for content and motion disentanglement in image sequences](https://mlsanity.com/paper/2112.01641/) (2021)
- [ArrowGAN : Learning to Generate Videos by Learning Arrow of Time](https://mlsanity.com/paper/2101.03710/) (2021)
- [Online Unsupervised Learning of Visual Representations and Categories](https://mlsanity.com/paper/2109.05675/) (2021)
- [Video to Video Generative Adversarial Network for Few-shot Learning Based on Policy Gradient](https://mlsanity.com/paper/2410.20657/) (2024)
- [Geometry-Aware Single-Image 4D Synthesis via Dense Trajectory Generation](https://mlsanity.com/paper/2512.05044/) (2025)
- [Wildfire Forecasting with Satellite Images and Deep Generative Model](https://mlsanity.com/paper/2208.09411/) (2022)
- [Disentangling Content and Motion for Text-Based Neural Video Manipulation](https://mlsanity.com/paper/2211.02980/) (2022)
- [Learn Proportional Derivative Controllable Latent Space from Pixels](https://mlsanity.com/paper/2110.08239/) (2021)
- [SNED: Superposition Network Architecture Search for Efficient Video Diffusion Model](https://mlsanity.com/paper/2406.00195/) (2024)
- [Inference-based GAN Video Generation](https://mlsanity.com/paper/2512.21776/) (2025)
- [TriDrive: Joint Driver, Vehicle, and Road Modeling for Forecasting and Driver Monitoring](https://mlsanity.com/paper/2609.33000/) (2026)

## Abstract (arXiv)

Videos express highly structured spatio-temporal patterns of visual data. A video can be thought of as being governed by two factors: (i) temporally invariant (e.g., person identity), or slowly varying (e.g., activity), attribute-induced appearance, encoding the persistent content of each frame, and (ii) an inter-frame motion or scene dynamics (e.g., encoding evolution of the person ex-ecuting the action). Based on this intuition, we propose a generative framework for video generation and future prediction. The proposed framework generates a video (short clip) by decoding samples sequentially drawn from a latent space distribution into full video frames. Variational Autoencoders (VAEs) are used as a means of encoding/decoding frames into/from the latent space and RNN as a wayto model the dynamics in the latent space. We improve the video generation consistency through temporally-conditional sampling and quality by structuring the latent space with attribute controls; ensuring that attributes can be both inferred and conditioned on during learning/generation. As a result, given attributes and/orthe first frame, our model is able to generate diverse but highly consistent sets ofvideo sequences, accounting for the inherent uncertainty in the prediction task. Experimental results on Chair CAD, Weizmann Human Action, and MIT-Flickr datasets, along with detailed comparison to the state-of-the-art, verify effectiveness of the framework.