mlsanityData to 14 Oct 2026

Diverse Video Generation using a Gaussian Process Trigger

arXiv 2107.04619 ↗First posted 9 Jul 2021cs.CV cs.AI cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

Generating future frames given a few context (or past) frames is a challenging task. It requires modeling the temporal coherence of videos and multi-modality in terms of diversity in the potential future states. Current variational approaches for video generation tend to marginalize over multi-modal future outcomes. Instead, we propose to explicitly model the multi-modality in the future outcomes and leverage it to sample diverse futures. Our approach, Diverse Video Generator, uses a Gaussian Process (GP) to learn priors on future states given the past and maintains a probability distribution over possible futures given a particular sample. In addition, we leverage the changes in this distribution over time to control the sampling of diverse future states by estimating the end of ongoing sequences. That is, we use the variance of GP over the output function space to trigger a change in an action sequence. We achieve state-of-the-art results on diverse future frame generation in terms of reconstruction quality and diversity of the generated sequences.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 36 of its 53 references that are in the map. First posted on arXiv 9 Jul 2021.

Builds on

36 of 53 references.

References · 36

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  3. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  4. 2013Auto-Encoding Variational Bayescited by 4,053
  5. 2016Improved Techniques for Training GANscited by 2,970
  6. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
  7. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,234
  8. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 609
  9. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  10. 2015Unsupervised Learning of Video Representations using LSTMscited by 435
  11. 2016Generating Videos with Scene Dynamicscited by 376
  12. 2015Deep multi-scale video prediction beyond mean square errorcited by 300
24 more
  1. 2017Toward Multimodal Image-to-Image Translationcited by 296
  2. 2018Stochastic Video Generation with a Learned Priorcited by 245
  3. 2016Generating Images with Perceptual Similarity Metrics based on Deep Networkscited by 222
  4. 2016Temporal Generative Adversarial Nets with Singular Value Clippingcited by 198
  5. 2015Kernel Interpolation for Scalable Structured Gaussian Processes (KISS-GP)cited by 177
  6. 2018Stochastic Adversarial Video Predictioncited by 173
  7. 2017Decomposing Motion and Content for Natural Video Sequence Predictioncited by 169
  8. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 139
  9. 2019Improved Conditional VRNNs for Video Predictioncited by 104
  10. 2017Learning to Generate Long-term Future via Hierarchical Predictioncited by 88
  11. 2017The Pose Knows: Video Forecasting by Generating Pose Futurescited by 88
  12. 2019High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networkscited by 88
  13. 2019VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generationcited by 74
  14. 2018Accurate and Diverse Sampling of Sequences based on a "Best of Many" Sample Objectivecited by 56
  15. 2017Deep Video Generation, Prediction and Completion of Human Action Sequencescited by 52
  16. 2018Hierarchical Long-term Video Prediction without Supervisioncited by 50
  17. 2016Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networkscited by 46
  18. 2017Folded Recurrent Neural Networks for Future Video Predictioncited by 44
  19. 2018Disentangling Propagation and Generation for Video Predictioncited by 44
  20. 2015Thoughts on Massively Scalable Gaussian Processescited by 26
  21. 2016Learning Scalable Deep Kernels with Recurrent Structurecited by 22
  22. 2014Distributed Variational Inference in Sparse Gaussian Process Regression and Latent Variable Modelscited by 19
  23. 2016Video Ladder Networkscited by 4
  24. 2014Gaussian Process Models with Parallelization and GPU accelerationcited by 3

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 7; 2024: 6; 2025: 1.

17 papers.

Sort:

By area:

  1. 2023Diff2Lip: Audio Conditioned Diffusion Models for Lip-Synchronizationcited by 49
  2. 2024Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Modelcited by 35
  3. 2023LaMD: Latent Motion Diffusion for Image-Conditional Video Generationcited by 22
  4. 2022Methods for Recovering Conditional Independence Graphs: A Surveycited by 7
  5. 2022uGLAD: Sparse graph recovery by optimizing deep unrolled networkscited by 6
  6. 2023Video Dynamics Prior: An Internal Learning Approach for Robust Video Enhancementscited by 5
  7. 2022LOPR: Latent Occupancy PRediction using Generative Modelscited by 4
  8. 2023Neural Graph Revealerscited by 3
  9. 2023Federated Learning with Neural Graphical Modelscited by 2
  10. 2025Towards Efficient Real-Time Video Motion Transfer via Generative Time Series Modelingcited by 2
  11. 2024Continuous Video Process: Modeling Videos as Continuous Multi-Dimensional Processes for Video Predictioncited by 1
  12. 2024Video Decomposition Prior: A Methodology to Decompose Videos into Layerscited by 1
5 more
  1. 2024Efficient Continuous Video Flow Model for Video Predictioncited by 1
  2. 2023Are uGLAD? Time will tell!cited by 0
  3. 2023Knowledge Propagation over Conditional Independence Graphscited by 0
  4. 2024Generative Kaleidoscopic Networkscited by 0
  5. 2024Obstacle-aware Gaussian Process Regressioncited by 0