# Conditional COT-GAN for Video Prediction with Kernel Smoothing

arXiv: https://arxiv.org/abs/2106.05658 · HTML: https://mlsanity.com/paper/2106.05658/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 47 references that are in the map.
- First posted on arXiv 10 Jun 2021.

## Builds on (most cited first, 30 of 31)

- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [MoCoGAN: Decomposing Motion and Content for Video Generation](https://mlsanity.com/paper/1707.04993/) (2017)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [Generating Videos with Scene Dynamics](https://mlsanity.com/paper/1609.02612/) (2016)
- [Learning Generative Models with Sinkhorn Divergences](https://mlsanity.com/paper/1706.00292/) (2017)
- [Deep multi-scale video prediction beyond mean square error](https://mlsanity.com/paper/1511.05440/) (2015)
- [MMD GAN: Towards Deeper Understanding of Moment Matching Network](https://mlsanity.com/paper/1705.08584/) (2017)
- [Stochastic Video Generation with a Learned Prior](https://mlsanity.com/paper/1802.07687/) (2018)
- [Stochastic Variational Video Prediction](https://mlsanity.com/paper/1710.11252/) (2017)
- [Adversarial Audio Synthesis](https://mlsanity.com/paper/1802.04208/) (2018)
- [Temporal Generative Adversarial Nets with Singular Value Clipping](https://mlsanity.com/paper/1611.06624/) (2016)
- [Scaling Autoregressive Video Models](https://mlsanity.com/paper/1906.02634/) (2019)
- [Stochastic Adversarial Video Prediction](https://mlsanity.com/paper/1804.01523/) (2018)
- [Decomposing Motion and Content for Natural Video Sequence Prediction](https://mlsanity.com/paper/1706.08033/) (2017)
- [Video Pixel Networks](https://mlsanity.com/paper/1610.00527/) (2016)
- [Improved Conditional VRNNs for Video Prediction](https://mlsanity.com/paper/1904.12165/) (2019)
- [Parallel Multiscale Autoregressive Density Estimation](https://mlsanity.com/paper/1703.03664/) (2017)
- [COT-GAN: Generating Sequential Data via Causal Optimal Transport](https://mlsanity.com/paper/2006.08571/) (2020)
- [Variational Temporal Abstraction](https://mlsanity.com/paper/1910.00775/) (2019)
- [FutureGAN: Anticipating the Future Frames of Video Sequences using Spatio-Temporal 3d Convolutions in Progressively Growing GANs](https://mlsanity.com/paper/1810.01325/) (2018)
- [Spread Divergence](https://mlsanity.com/paper/1811.08968/) (2018)
- [Causal transport in discrete time and applications](https://mlsanity.com/paper/1606.04062/) (2016)
- [Causal optimal transport and its links to enlargement of filtrations and continuous-time stochastic optimization](https://mlsanity.com/paper/1611.02610/) (2016)
- [Extended Mean Field Control Problems: stochastic maximum principle and transport perspective](https://mlsanity.com/paper/1802.05754/) (2018)

## Built on by (most cited first, 3 of 3)

- [The Weisfeiler-Lehman Distance: Reinterpretation and Connection with GNNs](https://mlsanity.com/paper/2302.00713/) (2023)
- [Learning conditional distributions on continuous spaces](https://mlsanity.com/paper/2406.09375/) (2024)
- [Distributionally robust risk evaluation with a causality constraint and structural information](https://mlsanity.com/paper/2203.10571/) (2022)

## Abstract (arXiv)

Causal Optimal Transport (COT) results from imposing a temporal causality constraint on classic optimal transport problems, which naturally generates a new concept of distances between distributions on path spaces. The first application of the COT theory for sequential learning was given in Xu et al. (2020), where COT-GAN was introduced as an adversarial algorithm to train implicit generative models optimized for producing sequential data. Relying on (Xu et al., 2020), the contribution of the present paper is twofold. First, we develop a conditional version of COT-GAN suitable for sequence prediction. This means that the dataset is now used in order to learn how a sequence will evolve given the observation of its past evolution. Second, we improve on the convergence results by working with modifications of the empirical measures via kernel smoothing due to (Pflug and Pichler (2016)). The resulting kernel conditional COT-GAN algorithm is illustrated with an application for video prediction.