mlsanityData to 7 Oct 2026

Clockwork Variational Autoencoders

arXiv 2102.09532 ↗First posted 18 Feb 2021cs.CV cs.AI cs.LGcited by 40

Not on the map. Why?

Abstract (arXiv)

Deep learning has enabled algorithms to generate realistic images. However, accurately predicting long video sequences requires understanding long-term dependencies and remains an open challenge. While existing video prediction models succeed at generating sharp images, they tend to fail at accurately predicting far into the future. We introduce the Clockwork VAE (CW-VAE), a video prediction model that leverages a hierarchy of latent sequences, where higher levels tick at slower intervals. We demonstrate the benefits of both hierarchical latents and temporal abstraction on 4 diverse video prediction datasets with sequences of up to 1000 frames, where CW-VAE outperforms top video prediction models. Additionally, we propose a Minecraft benchmark for long-term video prediction. We conduct several experiments to gain insights into CW-VAE and confirm that slower levels learn to represent objects that change more slowly in the video, and faster levels learn to represent faster objects.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). Builds on 47 of its 61 references that are in the map. First posted on arXiv 18 Feb 2021.

Builds on

47 of 61 references.

References · 47

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  5. 2013Auto-Encoding Variational Bayescited by 3,980
  6. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  7. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  8. 2018World Modelscited by 1,444
  9. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  10. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  11. 2020Mastering Atari with Discrete World Modelscited by 990
  12. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
35 more
  1. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  2. 2014Neural Turing Machinescited by 630
  3. 2015Unsupervised Learning of Video Representations using LSTMscited by 434
  4. 2016Generating Videos with Scene Dynamicscited by 376
  5. 2015A Recurrent Latent Variable Model for Sequential Datacited by 362
  6. 2016Ladder Variational Autoencoderscited by 315
  7. 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 261
  8. 2018Stochastic Video Generation with a Learned Priorcited by 244
  9. 2017Stochastic Variational Video Predictioncited by 234
  10. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 210
  11. 2016Deep Predictive Coding Networks for Video Prediction and Unsupervised Learningcited by 203
  12. 2019Scaling Autoregressive Video Modelscited by 189
  13. 2016A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialoguescited by 174
  14. 2018Stochastic Adversarial Video Predictioncited by 173
  15. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
  16. 2015Deep Kalman Filterscited by 163
  17. 2016Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Datacited by 154
  18. 2016Video Pixel Networkscited by 108
  19. 2020Stochastic Latent Residual Video Predictioncited by 106
  20. 2019Improved Conditional VRNNs for Video Predictioncited by 104
  21. 2016Hierarchical Multiscale Recurrent Neural Networkscited by 102
  22. 2019VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generationcited by 74
  23. 2014A Clockwork RNNcited by 70
  24. 2020Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictorscited by 60
  25. 2016Clockwork Convnets for Video Semantic Segmentationcited by 55
  26. 2018Hierarchical Long-term Video Prediction without Supervisioncited by 50
  27. 2018Learning and Querying Fast Generative Models for Reinforcement Learningcited by 48
  28. 2018Temporal Difference Variational Auto-Encodercited by 48
  29. 2018Time-Agnostic Prediction: Predicting Predictable Video Framescited by 37
  30. 2019Variational Temporal Abstractioncited by 37
  31. 2017Learning Hierarchical Features from Generative Modelscited by 27
  32. 2019CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Networkcited by 25
  33. 2017Fast-Slow Recurrent Neural Networkscited by 12
  34. 2018Approximate Bayesian inference in spatial environmentscited by 8
  35. 2017Generative Temporal Models with Memorycited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 15; 2023: 10; 2024: 3; 2025: 4; 2026: 3.

40 papers.

Sort:

By area:

  1. 2022Flexible Diffusion Modeling of Long Videoscited by 294
  2. 2022Diffusion Probabilistic Modeling for Video Generationcited by 237
  3. 2023Video Language Planningcited by 160
  4. 2022Diffusion Models for Video Prediction and Infillingcited by 134
  5. 2021Discovering and Achieving Goals via World Modelscited by 131
  6. 2025Long-Context Autoregressive Video Modeling with Next-Frame Predictioncited by 119
  7. 2024Simple Hierarchical Planning with Diffusioncited by 71
  8. 2024Human-inspired Episodic Memory for Infinite Context LLMscited by 62
  9. 2022Temporally Consistent Transformers for Video Generationcited by 43
  10. 2023Facing Off World Model Backbones: RNNs, Transformers, and S4cited by 31
  11. 2022The Free Energy Principle for Perception and Action: A Deep Learning Perspectivecited by 24
  12. 2024SSM Meets Video Diffusion Models: Efficient Long-Term Video Generation with Structured State Spacescited by 13
28 more
  1. 2022MSPred: Video Prediction at Multiple Spatio-Temporal Scales with Hierarchical Recurrent Networkscited by 12
  2. 2023Revisiting Structured Variational Autoencoderscited by 11
  3. 2021Variational Predictive Routing with Nested Subjective Timescalescited by 9
  4. 2023On the Identifiability of Switching Dynamical Systemscited by 9
  5. 2021Disentangled Generative Models for Robust Prediction of System Dynamicscited by 8
  6. 2022Learning Robust Dynamics through Variational Sparse Gatingcited by 8
  7. 2022Multi-Horizon Representations with Hierarchical Forward Models for Reinforcement Learningcited by 7
  8. 2023Towards End-to-End Generative Modeling of Long Videos with Memory-Efficient Bidirectional Transformerscited by 7
  9. 2025Towards Identifiability of Hierarchical Temporal Causal Representation Learningcited by 7
  10. 2021Lossless compression with state space models using bits back codingcited by 6
  11. 2022Towards Robust Unsupervised Disentanglement of Sequential Data -- A Case Study Using Music Audiocited by 6
  12. 2022Long-horizon video prediction using a dynamic latent hierarchycited by 4
  13. 2023Weakly-Supervised Action Localization by Hierarchically-structured Latent Attention Modelingcited by 4
  14. 2026Causality in Video Diffusers is Separable from Denoisingcited by 4
  15. 2023The Acquisition of Physical Knowledge in Generative Neural Networkscited by 3
  16. 2023Physics-enhanced Gaussian Process Variational Autoencodercited by 2
  17. 2025Generative Video Bi-flowcited by 2
  18. 2021Unsupervised dynamic modeling of medical image transformationcited by 1
  19. 2022Bayesian sense of time in biological and artificial brainscited by 1
  20. 2022Benchmarking Generative Latent Variable Models for Speechcited by 1
  21. 2023Discovering Predictable Latent Factors for Time Series Forecastingcited by 1
  22. 2026Contextual Intelligence The Next Leap for Reinforcement Learningcited by 1
  23. 2022Stochastic Gradient Descent Captures How Children Learn About Physicscited by 0
  24. 2022Multi-Timescale Modeling of Human Behaviorcited by 0
  25. 2022Latent Discretization for Continuous-time Sequence Compressioncited by 0
  26. 2023Learning Recurrent Models with Temporally Local Rulescited by 0
  27. 2025Real-World Robot Control by Deep Active Inference With a Temporally Hierarchical World Modelcited by 0
  28. 2026SUNTA: Hierarchical Video Prediction with Surprise-based Chunkingcited by 0